Introduction

8th Wall Web enables developers to create augmented reality brand experiences that live on the mobile web.

Built entirely using standards-compliant JavaScript and WebGL, 8th Wall Web is a complete implementation of 8th Wall’s Simultaneous Localization and Mapping (SLAM) engine, hyper-optimized for real-time AR on mobile browsers. Features include 6-Degrees of Freedom Tracking, Surface Estimation, Lighting, World Points and Hit Tests.

8th Wall Web is easily integrated into 3D JavaScript frameworks such as:

Tutorial API Reference Need Help?

Videos:

What's New

8th Wall Web Release 12.1 is now available! This release provides a number of updates and enhancements:

Release 12.1:

  • Fixes and Enhancements:

    • Increased camera resolution on newer iOS devices.
    • Increased AFrame fps on high-res Android devices.
    • Fixed three.js r103+ raycasting issues.
    • Added support for iPadOS.
    • Fixed memory issue when loading many image targets repeatedly.
    • Minor performance enhances and bug fixes.

Click Here for to see a full list of changes.

Requirements

Web Browser Requirements

Mobile browsers require the following functionality to support 8th Wall Web experiences:

  • WebGL (canvas.getContext('webgl') || canvas.getContext('webgl2'))
  • getUserMedia (navigator.mediaDevices.getUserMedia)
  • deviceorientation (window.DeviceOrientationEvent - only needed if SLAM is enabled)
  • Web-Assembly/WASM (window.WebAssembly)

NOTE: 8th Wall Web experiences must be viewed via https. This is required by browsers for camera access.

This translates to the following for iOS and Android devices:

  • iOS:

    • OS: iOS 11 or newer
    • Supported Browsers:

      • Safari

        • Note: getUserMedia only returns video input devices directly in Safari, not in UIWebView or WKWebView.
      • SFSafariViewController: iOS 13 added getUserMedia support to SFSafariViewController. 8th Wall Web now works within iOS 13 apps that use SFSafariViewController web views.
  • Android:

    • Supported Browsers:

      • Chrome
      • Chromium-based browsers (Samsung Browser, Microsoft Edge, etc.)
      • Firefox
      • Android WebViews

Supported Frameworks

8th Wall Web is easily integrated into 3D JavaScript frameworks such as:

Supported Features

Platform Lighting AR Background Camera Motion Horizontal Surfaces Vertical Surfaces Image Detection & Tracking World Points Hit Tests
8th Wall Web Yes Yes 6 DoF (Scale Free) Yes, Instant Planar No Yes Yes Yes

Tutorial

This guide provides all of the steps required to get you up and running with a sample A-Frame or three.js project.

Download a Sample Project

Sample projects can be found on 8th Wall Web's public GitHub repo at https://github.com/8thwall/web

Create a Console Account

New Users:

  1. Create a free account at https://www.8thwall.com/sign-up
  2. Select the "Web Developer" Workspace Type (Note: you can create other workspaces later)

SignUpWebWorkspace

Existing Users:

  1. Login to https://www.8thwall.com and select your Web Developer workspace.

SelectWorkspace

If you don't have a Web Developer Workspace, go to https://www.8thwall.com/workspaces and create one by clicking the "+ Create a New Workspace" button.

Create App Key

  1. If you have multiple workspaces associated with your console user, select your Web Developer workspace.

ConsoleSelectWorkspace

  1. Click "Dashboard" in the left hand navigation

  2. Click "+ Create a new Web App +":

ConsoleCreateWebApp

  1. Enter a name for your Web App. The application name should be descriptive but short and only include letters, spaces, "." and "_". It should be unique. (e.g. “WebARTest”):

ConsoleCreateWebApp2

  1. Click Create

  2. After creation, your App Key information will be displayed. Copy the AppKey string.

Add App Key to Web Project

Add your App Key to the A-Frame or three.js project. If you are using a sample project from 8th Wall, edit index.html and replace the "X"'s with your app key.

  • Example (A-Frame):

    <script async src="//apps.8thwall.com/xrweb?appKey=XXXXXXXXXX"></script>

  • Example (three.js):

    <script defer src="https://apps.8thwall.com/xrweb?appKey=XXXXXXXXXX"></script>

Authorize Your Device

Authorizing a device (aka enabling Developer Mode) allows it to view 8th Wall Web applications under development. Authorizing a device installs a Dev Token (cookie) into its web browser, allowing it to view any app key within the current workspace.

There is no limit to the number of devices that can be authorized, but each device needs to be authorized individually. Views of your web application from an authorized device count toward your monthly usage total.

  1. Login to the console at https://www.8thwall.com/login

  2. Select the Workspace that contains your App Key.

  3. Click Device Authorization to expand the device authorization pane.

ConsoleDeviceAuthorization

  1. Select 8th Wall Web version to use during development. To use the latest stable version of 8th Wall Web, select release. To use a pre-release version, select beta.

ConsoleDeveloperModeChannel

From Desktop: If you are logged into the console on your laptop/desktop, Scan the QR code from the device you wish to authorize.

Note: A QR code can only be scanned once. After scanning, you will receive confirmation that your device has been authorized. The console will then generate a new QR code that can be scanned to authorize another device.

Before:

ConsoleDevTokenQR

After:

Confirmation (Console) Confirmation (On Device)
ConsoleQRConfirmation MobileQRConfirmation

From Mobile: If you are logged into the 8th Wall Console directly on the mobile device you wish to authorize, simply click Authorize browser. Doing so enables Developer Mode on this device's browser, authorizing it to view any app key within the current workspace.

Before:

DeveloperModeMobile

After:

DeveloperModeMobileAuthorized

Serve Application

From Web Server

If you have a development web server, upload the sample project and connect from a mobile device that has a dev token installed. You'll need to connect via HTTPS as browsers require HTTPS certificates to access the camera on your phone through a browser.

Locally From Mac

Serving web app locally from your computer can be tricky as browsers require HTTPS certificates to access the camera on your phone through a browser. As a convenience, 8th Wall has created a public GitHub repo (https://github.com/8thwall/web) where you can find a "serve" script that will run a local https webserver on your development computer. You can also download sample 8th Wall Web projects to help you get started.

  1. Install Node.js and npm

If you don't already have Node.js and npm installed, get it here: https://www.npmjs.com/get-npm

  1. Open a terminal window (Terminal.app, iTerm2, etc):
# cd <directory_where_you_saved_sample_project_files>
# cd serve
# npm install
# cd ..
# ./serve/bin/serve -d <sample_project_location>

Example:

./serve/bin/serve -n -d gettingstarted/xraframe/ -p 7777

ServeLocally

IMPORTANT: To connect to this local webserver, make sure to copy the entire "Listening" URL into your browser, including both the "https://" at the beginning and port number at the end.

NOTE: If the serve script states it's listening on 127.0.0.1:<port> (which is the loopback device aka "localhost") your mobile phone won't be able to connect to that IP address directly. Please re-run the serve script with the -i flag to specify the network interface the serve script should listen on.

Example - specify network interface:

./serve/bin/serve -d gettingstarted/xraframe/ -p 7777 -i en0

If you have issues connecting to the local webserver running on your computer, please refer to the troubleshooting section

Locally From Windows

Serving web app locally from your computer can be tricky as browsers require HTTPS certificates to access the camera on your phone through a browser. As a convenience, 8th Wall has created a public GitHub repo (https://github.com/8thwall/web) where you can find a "serve" script that will run a local https webserver on your development computer. You can also download sample 8th Wall Web projects to help you get started.

  1. Install Node.js and npm

If you don't already have Node.js and npm installed, get it here: https://www.npmjs.com/get-npm

  1. Open a Command Prompt (cmd.exe)

Note: Run the following command using a standard Command Prompt window (cmd.exe). The script will generate errors if run from PowerShell.

# cd <directory_where_you_saved_sample_project_files>
# cd serve
# npm install
# cd ..
# serve\bin\serve.bat -d <sample_project_location>

Example:

serve\bin\serve.bat -n -d gettingstarted\xraframe -p 7777

ServeLocallyWindows

IMPORTANT: To connect to this local webserver, make sure to copy the entire "Listening" URL into your browser, including both the "https://" at the beginning and port number at the end.

NOTE: If the serve script states it's listening on 127.0.0.1:<port> (which is the loopback device aka "localhost") your mobile phone won't be able to connect to that IP address directly. Please re-run the serve script with the -i flag to specify the network interface the serve script should listen on.

Example - specify network interface:

serve\bin\serve.bat -d gettingstarted\xraframe -p 7777 -i WiFi

If you have issues connecting to the local webserver running on your computer, please refer to the troubleshooting section

View App on Phone

iOS

  1. The “serve” command run in the previous step will display the IP and Port to connect to
  2. Open Safari on iOS 11+, and connect to the “Listening” URL. Note: Safari will complain about the SSL certificates, but you can safely proceed.

IMPORTANT: Make sure to copy the entire "Listening" URL into your browser, including both the "https://" at the beginning and port number at the end.

Example: https://192.168.1.50:8080

  1. Click "visit this website": iOSConnect1
  2. Click "Show Details": iOSConnect2
  3. Click "Visit Website": iOSConnect3
  4. Finally, click "Allow" to grant camera permissions and start viewing the sample AR experience: iOSConnect4

Android

  1. The “serve” command run in the previous step will display the IP and Port to connect to
  2. Open Chrome, a Chrome-variant (e.g. Samsung browser) or Firefox

IMPORTANT: Make sure to copy the entire "Listening" URL into your browser, including both the "https://" at the beginning and port number at the end.

Example: https://192.168.1.50:8080

  1. Chrome Example: The browser will complain that the cert is invalid, simply click "Advanced" to proceed: AndroidConnect1
  2. Click "PROCEED TO ... (UNSAFE)": AndroidConnect2

Console Overview

The 8th Wall Console is a web based portal that allows you to:

  • Generate and manage app keys for both native and Web applications
  • Create new Web applications
  • Create new web-based AR Camera applications
  • View app analytics
  • Manage workspaces
  • Access downloads and documentation
  • Manage your plan and billing

To access the 8th Wall Console, login at https://www.8thwall.com/login

New Users:

  1. Create a free account at https://www.8thwall.com/sign-up

SignUpWebWorkspace

Existing Users:

  1. Go to https://www.8thwall.com/login and login with your email and password.

  2. If you have multiple workspaces associated with your console user, select your Web Developer workspace.

ConsoleSelectWorkspace

Console Dashboard

The 8th Wall Web Developer Dashboard is your hub for creating and managing your Web Applications as well as viewing usage and analytics.

ConsoleDashboard

App Key Management

An App Key is required to run web apps using 8th Wall Web. This section decribes how to create, manage and use Web App Keys for 8th Wall Web.

Create Key

  1. Select your Web Developer workspace.

ConsoleSelectWorkspace

  1. Click "Dashboard" in the left navigation.

  2. Click "Create a new Web App +":

ConsoleCreateWebApp

  1. Enter a name for your Web App. The application name should be descriptive but short and only include letters, spaces, "." and "_". It should be unique. (e.g. “WebARTest”):

ConsoleCreateWebApp2

  1. Click Create.

  2. After creation, your App Key information will be displayed. Copy the AppKey string.

Copy Key

  1. Select your Web Developer workspace.

  2. Click Dashboard in the left navigation.

  3. Find the desired app key in the list of "My Web Applications" and click the Edit button to view the Application Information page:

ConsoleAppKeyCopy

  1. On the Application Information page copy the AppKey string.

  2. Add the App Key into your A-Frame or three.js project. If you are using a sample project from 8th Wall, edit index.html and replace the "X"'s with your app key.

  • Example (A-Frame):

    <script async src="//apps.8thwall.com/xrweb?appKey=XXXXXXXXXX"></script>

  • Example (three.js):

    <script defer src="https://apps.8thwall.com/xrweb?appKey=XXXXXXXXXX"></script>

Disable Key

Note: Disabling an App Key will cause any web applications using it to stop working. You can always re-enable it later.

  1. Click Dashboard in the left navigation.

  2. Find the desired app key in the list of "My Web Applications" and click on the name to expand.

  3. Click the Status slider to the OFF position.

  4. Click OK to confirm.

ConsoleAppKeyDisable

Enable Key

To re-enable an Web Application after it has been disabled:

  1. Click Dashboard in the left navigation

  2. Find the desired app key in the list of "My Web Applications" and click on the name to expand

  3. Click the Status slider to the ON position

  4. Click OK to confirm.

ConsoleAppKeyEnable

Delete Key

Note: Deleting an App Key will cause any web apps using it to stop working. You cannot undo this operation.

  1. Click Dashboard in the left navigation.

  2. Find the desired app key in the list of "My Web Applications" and click the Delete button.

  3. Type "DELETE" to confirm and then click OK.

ConsoleAppKeyDelete

Allowed Origins

If you have upgraded to an Agency or Business plan, you can host your Web Application publicly on your own web server (and viewed without device authorization). In order to do so, you will need to specify a list of approved URLs that can host this App Key.

  1. Click Dashboard in the left navigation

  2. Find the desired app key in the list of "My Web Applications" and click the Edit button to view the Application Information page:

ConsoleAppKeyCopy

  1. In the "Allowed Origins" section of the Application Information page, enter one or more origins. Click the "+" to add multiple. An origin may not contain a wildcard, a path, or a port.

ConsoleAppKeyDisable

XR Engine Version

NOTE: This is only available to workspaces on Agency or Business plans.

For each public Web Application, you can specify the version of the XR engine used when serving public web clients.

If you select a Channel (release or beta), public clients will always be served the most recent version of 8th Wall Web from that channel. If you freeze the version, you will need to manually unfreeze to receive the latest features and improvements of the engine.

In general, 8th Wall recommends using the official release channel for production web apps.

If you would like to test your web app against a pre-release version of 8th Wall Web, which may contain new features and/or bug fixes that haven't gone through full QA yet, you can switch to the beta channel:

ConsoleAppKeyChannel

  1. Click Dashboard in the left navigation

  2. Find the desired app key in the list of "My Web Applications" and click the Edit button to view the Application Information page:

To Freeze to a specific version, select the desired Channel (release or beta) and click the Freeze button

To Re-join a Channel and stay up-to-date, select the desired Channel radio button. This will unfreeze the XR Engine version associated with your App Key.

To Rollback to the previous selection, click the Rollback button.

Device Authorization

Authorizing a device (aka enabling Developer Mode) allows it to view 8th Wall Web applications under development. Authorizing a device installs a Dev Token (cookie) into its web browser, allowing it to view any app key within the current workspace.

There is no limit to the number of devices that can be authorized, but each device needs to be authorized individually. Views of your web application from an authorized device count toward your monthly usage total.

Note: If you upgrade to an Agency or Business plan, you gain the ability to host your Web Application publicly on internet (and view them without device authorization / dev tokens).

How to authorize a device:

  1. Login to the console at https://www.8thwall.com/login

  2. Select the Workspace that contains your App Key.

  3. Click Device Authorization to expand the device authorization pane.

ConsoleDeveloperMode

  1. Select 8th Wall Web version to use during development. To use the latest stable version of 8th Wall Web, select release. To use a pre-release version, select beta.

ConsoleDeveloperModeChannel

  1. Authorize your device:

From Desktop: If you are logged into the console on your laptop/desktop, Scan the QR code from the device you wish to authorize.

Note: A QR code can only be scanned once. After scanning, you will receive confirmation that your device has been authorized. The console will then generate a new QR code that can be scanned to authorize another device.

Before:

ConsoleDevTokenQR

After:

Confirmation (Console) Confirmation (On Device)
ConsoleQRConfirmation MobileQRConfirmation

From Mobile: If you are logged into the 8th Wall Console directly on the mobile device you wish to authorize, simply click Authorize browser. Doing so enables Developer Mode on this device's browser, authorizing it to view any app key within the current workspace.

Before:

DeveloperModeMobile

After:

DeveloperModeMobileAuthorized

Image Targets

8th Wall Web can detect and track 2D image files, allowing you to bring static content to life.

Not only can your designated image target trigger a web AR experience, but your content has the ability to track directly to it.

Image tracking works in tandem with our SLAM engine, giving you the flexibility to design experiences that interact with image targets as well as the physical space around you.

You may designate up to 5 "Autoloaded" image targets per App Key. An Autoloaded image target is enabled immediately as the page loads. This is useful for apps that use 5 or fewer image targets such as product packaging, a movie poster or business card.

All of your uploaded image targets are available to load at any time by calling XR.XrController.configure(). This lets you manage hundreds of image targets per App Key making possible use cases like geo-fenced image target hunts, AR books, guided art museum tours and much more. If your project utilizes SLAM most of the time but image targets some of the time, you can improve performance by only loading image targets when you need them. You can even read uploaded target names from URL parameters stored in different QR Codes, allowing you to have different targets initially load in the same web app depending on which QR Codes the user scans to enter the experience.

To enable Image Targets for an App Key:

  1. Click Dashboard in the left navigation.

  2. Find the desired app key in the list of "My Projects" and click the Add an image target button.

  3. Drag your image (.jpg or .png) to the "Upload Image" panel, or click "Upload Image" and use your file browser to select your image

  4. On the Image Target Upload screen, set orientation and use the zoom slider to select region of interest.

  5. Give your image target a name by editing the field at the top left of the window.

  6. Click Upload.

  7. After upload, test your image. On the “Image Target Editor” screen, scan the QR code to open our “Target Tester” web app, then scan your image.

  8. Optional: Add metadata to your image, in either Text or JSON format.

  9. Click Load automatically if you want this image target to be enabled automatically as the page loads. You may have up to 5 autoloaded image targets per app key.

  10. Click Save

Image Target Requirements

  • File Types: .jpg or .png
  • Aspect Ratio: 3:4
  • Dimensions:

    • Minimum: 480 x 640 pixels
    • Maximum length or width: 2048 pixels.

      • Note: If you upload something larger, the image is resized down to a max length/width of 2048, maintaining aspect ratio.)

Image Target Quantities

You may track up to 5 image targets simultaneously while SLAM is running. If you disable SLAM tracking by setting "disableWorldTracking: true" and request your image targets programmatically, you may track up to 10 simultaneously.

  • Active images per App Key (SLAM enabled): 5
  • Active images per App Key (SLAM disabled): 10
  • Uploaded images per App Key: Up to 1,000

Changing Active Image Targets

The set of active image targets can be modified at runtime by calling XR.XrController.configure()

Note: All currently active image targets will be replaced with the ones specified in this list.

Example - Change active image target set

XR.XrController.configure({imageTargets: ['image-target1', 'image-target2', 'image-target3']})

Optimizing Image Target Tracking

To ensure the highest quality image target tracking experience, be sure to follow these guidelines when selecting an image target.

DO have:

  • a lot of varied detail
  • high contrast

DON'T have:

  • repetitive patterns
  • excessive dead space
  • low resolution images

Color: Image target detection cannot distinguish between colors, so don't rely on it as a key differentiator between targets.

For best results, use flat surfaces for image target tracking.

Consider the reflectivity of your image target's physical material. Glossy surfaces and screen reflections can lower tracking quality. Use matte materials in diffuse lighting conditions for optimal tracking quality.

Note: Detection happens fastest in the center of the screen.

Good Markers Bad Markers

Image Target Events

8th Wall Web emits Events / Observables for various events in the image target lifecycle (e.g. imageloading, imagescaning, imagefound, imageupdated, imagelost) Please see the API reference for instructions on handling these events in your Web Application:

Example Projects

https://github.com/8thwall/web/tree/master/examples/aframe/artgallery

https://github.com/8thwall/web/tree/master/examples/aframe/flyer

Teams

Each Workspace has a team containing one or more Users, each with different permissions. Users can belong to multiple Workspace teams.

Add other members to your team to allow them to use the same App Keys you have created. You can collaboratively work on the same set of apps and see their usage.

Invite Users

  1. Select your Web Developer workspace.
  2. Click Team in the left navigation
  3. Enter the email address(es) for the users you want to invite. Enter multiple emails separated by commas.
  4. Click Invite users

ManagePlans

User Roles

Team members can have one of three roles:

  • OWNER
  • ADMIN
  • DEV

Capabilities for each role:

Capability OWNER ADMIN DEV
View Dashboard / Usage X X X
Web Apps - Create X X X
Web Apps - Edit X X X
Web Apps - Delete X X X
Web Apps - Enable/Disable X X X
Development Tokens X X X
Teams - View Users X X X
Teams - Invite Users X X
Teams - Remove Users X X
Teams - Manage User Roles X X
Workspaces - Create X X X
Workspaces - Edit X
Workspaces - Manage Plans X
View Quick Start X X X
View Release Notes X X X
Edit Profile X X X

Workspaces

A Workspace is a logical grouping of Users and Applications. Grouping Applications under the same Workspace allows you to view consolidated usage and billing. Workspaces can contain one or more Users, each with different permissions. Users can belong to multiple Workspaces.

There are 3 types of Workspaces:

  • Web Developer: Create and manage App Keys for 8th Wall Web - 8th Wall's Web SDK for creating rich AR experiences that run directly in a mobile web browser using JavaScript and WebGL.
  • XR Developer: Create and manage App Keys for 8th Wall XR for Unity® - 8th Wall's cross-platform Unity SDK for building mobile AR apps that can be deployed on iOS and Android devices.
  • AR Camera: Instantly create AR Cameras for the mobile web by uploading 3D models. No coding required!

Initial Workspace

When signing up for a new 8th Wall account, a free Web Developer workspace will be created. You can create additional workspaces later.

  1. Create a free account at <a href='https://www.8thwall.com/sign-up>https://conswwwole.8thwall.com/sign-up

SignUpWebWorkspace

Managing Workspaces

The Workspaces page in the console allows you to view and manage the Workspaces you are a member of.

Use one of the following methods to access the Workspaces page:

  1. Click WORKSPACES at the top of the left navigation.

AccountSwitcherManageAll

  1. Or click on your name at the top-right of the page and select Workspaces

ConsoleWorkspaceMenu

  1. Or go directly to https://www.8thwall.com/workspaces to view and manage your workspaces.

Create Workspace

NOTE: You can have a maximum of one FREE Web Developer workspace and one FREE AR Camera workspace.

  1. Go to https://www.8thwall.com/workspaces
  2. Click "Create a New Workspace"
  3. Select Workspace Type
  4. Click Create

ConsoleWorkspaceCreate

Switching Workspaces

  1. The workspace switcher menu can be found at the top of the left navigation. Click to select your desired Workspace:

AccountSwitcherManageAll

Manage Plans

  1. Click Manage Plans in the left navigation
  2. Click Upgrade to select an appropriate plan and enter your billing information to upgrade to a paid plan.

Please refer to https://www.8thwall.com/pricing for detailed information on plans and pricing.

For enterprise licensing options, please contact the 8th Wall team by filling out the form at https://www.8thwall.com/licensing

ManagePlans

API Overview

This section of the documentation contains details of 8th Wall Web's Javascript API.

XR

Description

Entry point for 8th Wall Web

Functions

Function Description
addCameraPipelineModule Adds a module to the camera pipeline that will receive event callbacks for each stage in the camera pipeline.
addCameraPipelineModules Add multiple camera pipeline modules. This is a convenience method that calls addCameraPipelineModule in order on each element of the input array.
isPaused Indicates whether or not the XR session is paused.
pause Pause the current XR session. While paused, the camera feed is stopped and device motion is not tracked.
resume Resume the current XR session.
run Open the camera and start running the camera run loop.
runPreRender Executes all lifecycle updates that should happen before rendering.
runPostRender Executes all lifecycle updates that should happen after rendering.
stop Stop the current XR session. While stopped, the camera feed is closed and device motion is not tracked.
version Get the 8th Wall Web engine version.

Modules

Module Description
AFrame Entry point for A-Frame integration with 8th Wall Web.
Babylonjs Entry point for Babylon.js integration with 8th Wall Web.
CameraPixelArray Provides a camera pipeline module that gives access to camera data as a grayscale or color uint8 array.
CanvasScreenshot Provides a camera pipeline module that can generate screenshots of the current scene.
FullWindowCanvas Provides a camera pipeline module that keeps the canvas specified in XR.run() covering the whole window.
GlTextureRenderer Provides a camera pipeline module that draws the camera feed to a canvas as well as extra utilities for GL drawing operations.
Sumerian Entry point for Sumerian integration with 8th Wall Web.
Threejs Provides a camera pipeline module that drives three.js camera to do virtual overlays.
XrController XrController provides 6DoF camera tracking and interfaces for configuring tracking.
XrDevice Provides information about device compatibility and characteristics.

XR.addCameraPipelineModule()

XR.addCameraPipelineModule()

Description

8th Wall camera applications are built using a camera pipeline module framework. For a full description on camera pipeline modules, see CameraPipelineModule.

Applications install modules which then control the behavior of the application at runtime. A module object must have a .name string which is unique within the application, and then should provide one or more of the camera lifecycle methods which will be executed at the appropriate point in the run loop.

During the main runtime of an application, each camera frame goes through the following cycle:

onStart -> onCameraStatusChange (requesting -> hasStream -> hasVideo) -> onProcessGpu -> onProcessCpu -> onUpdate -> onRender

Camera modules should implement one or more of the following camera lifecycle methods:

Function Description
onAppResourcesLoaded Called when we have received the resources attached to an app from the server.
onCameraStatusChange Called when a change occurs during the camera permissions request.
onCanvasSizeChange Called when the canvas changes size.
onDeviceOrientationChange Called when the device changes landscape/portrait orientation.
onException Called when an error occurs in XR. Called with the error object.
onPaused Called when XR.pause() is called.
onProcessCpu Called to read results of GPU processing and return usable data.
onProcessGpu Called to start GPU processing.
onRender Called after onUpdate. This is the time for the rendering engine to issue any WebGL drawing commands. If an application is providing its own run loop and is relying on XR.runPreRender() and XR.runPostRender(, this method is not called and all rendering must be coordinated by the external run loop.
onResume Called when XR.resume() is called.
onStart Called when XR starts. First callback after XR.run() is called.
onUpdate Called to update the scene before render. Data returned by modules in onProcessGpu and onProcessCpu will be present as processGpu.modulename and processCpu.modulename where the name is given by module.name = "modulename".
onVideoSizeChange Called when the canvas changes size.

Note: Camera modules that implement onProcessGpu or onProcessCpu can provide data to subsequent stages of the pipeline. This is done by the module's name.

Example1 - A camera pipeline module for managing camera permissions:

XR.addCameraPipelineModule({
  name = 'camerastartupmodule',
  onCameraStatusChange = ({status}) {
    if (status == 'requesting') {
      myApplication.showCameraPermissionsPrompt()
    } else if (status == 'hasStream') {
      myApplication.dismissCameraPermissionsPrompt()
    } else if (status == 'hasVideo') {
      myApplication.startMainApplictation()
    } else if (status == 'failed') {
      myApplication.promptUserToChangeBrowserSettings()
    }
  },
})

Example2 - a QR code scanning application could be built like this:

// Install a module which gets the camera feed as a UInt8Array.
XR.addCameraPipelineModule(
  XR.CameraPixelArray.pipelineModule({luminance: true, width: 240, height: 320}))

// Install a module that draws the camera feed to the canvas.
XR.addCameraPipelineModule(XR.GlTextureRenderer.pipelineModule())

// Create our custom application logic for scanning and displaying QR codes.
XR.addCameraPipelineModule({
  name = 'qrscan',
  onProcessCpu = ({onProcessGpuResult}) => {
    // CameraPixelArray.pipelineModule() returned these in onProcessGpu.
    const { pixels, rows, cols, rowBytes } = onProcesGpuResult.camerapixelarray
    const { wasFound, url, corners } = findQrCode(pixels, rows, cols, rowBytes)
    return { wasFound, url, corners }
  },
  onUpdate = ({onProcessCpuResult}) => {
    // These were returned by this module ('qrscan') in onProcessCpu
    const {wasFound, url, corners } = onProcessCpuResult.qrscan
    if (wasFound) {
      showUrlAndCorners(url, corners)
    }
  },
})

XR.addCameraPipelineModules()

XR.addCameraPipelineModules([ modules ])

Description

Add multiple camera pipeline modules. This is a convenience method that calls addCameraPipelineModule in order on each element of the input array.

Parameters

Parameter Description
modules An array of camera pipeline modules.

Example

const onxrloaded = () => {
  XR.addCameraPipelineModules([  // Add camera pipeline modules.
    // Existing pipeline modules.
    XR.FullWindowCanvas.pipelineModule(),   // Modifies the canvas to fill the window.
    XR.GlTextureRenderer.pipelineModule(),  // Draws the camera feed.
  ])

  // Request camera permissions and run the camera.
  XR.run({canvas: document.getElementById('camerafeed')})
}

// Wait until the XR javascript has loaded before making XR calls.
window.onload = () => {window.XR ? onxrloaded() : window.addEventListener('xrloaded', onxrloaded)}

XR.isPaused()

bool XR.isPaused()

Parameters

None

Description

Indicates whether or not the XR session is paused.

Example

// Call XR.pause() / XR.resume() when the button is pressed.
document.getElementById('pause').addEventListener(
  'click',
  () => {
    if (!XR.isPaused()) {
      XR.pause()
    } else {
      XR.resume()
    }
  },
  true)

XR.pause()

XR.pause()

Parameters

None

Description

Pause the current XR session. While paused, device motion is not tracked.

Example

// Call XR.pause() / XR.resume() when the button is pressed.
document.getElementById('pause').addEventListener(
  'click',
  () => {
    if (!XR.isPaused()) {
      XR.pause()
    } else {
      XR.resume()
    }
  },
  true)

XR.resume()

XR.resume()

Parameters

None

Description

Resume the current XR session after it has been paused.

Example

// Call XR.pause() / XR.resume() when the button is pressed.
document.getElementById('pause').addEventListener(
  'click',
  () => {
    if (!XR.isPaused()) {
      XR.pause()
    } else {
      XR.resume()
    }
  },
  true)

XR.run()

XR.run(canvas, webgl2: true, ownRunLoop: true)

Parameters

Parameters

Property Type Default Description
canvas HTMLCanvasElement The HTML Canvas that the camera feed will be drawn to.
webgl2 [Optional] bool true If true, use WebGL2 if available, otherwise fallback to WebGL1. If false, always use WebGL1.
ownRunLoop [Optional] bool true If true, XR should use it's own run loop. If false, you will provide your own run loop and be responsible for calling runPreRender and runPostRender yourself [Advanced Users only]
cameraConfig: {direction} [Optional] object {direction: XR.XrConfig.camera().BACK} Desired camera to use. Supported values for direction are XR.XrConfig.camera().BACK or XR.XrConfig.camera().FRONT

Notes:

  • cameraConfig: World tracking (SLAM) is only supported on the back camera. If you are using the front camera, you must disable world tracking by calling XR.XrController.configure({disableWorldTracking: true}) first.

Description

Open the camera and start running the camera run loop.

Example

// Open the camera and start running the camera run loop
// In index.html: <canvas id="camerafeed"></canvas>
XR.run({canvas: document.getElementById('camerafeed')})

Example - Using Front camera (image tracking only)

// Disable world tracking (SLAM). This is required to use the front camera.
XR.XrController.configure({disableWorldTracking: true})
// Open the camera and start running the camera run loop
// In index.html: <canvas id="camerafeed"></canvas>
XR.run({canvas: document.getElementById('camerafeed'), cameraConfig: {direction: XR.XrConfig.camera().FRONT}})

XR.runPreRender()

XR.runPreRender( timestamp )

Description

Executes all lifecycle updates that should happen before rendering.

IMPORTANT: Make sure that onStart has been called before calling runPreRender()/runPostRender().

Parameters

Parameter Description
timestamp The current time, in milliseconds.

Example

// Implement A-Frame components tick() method
function tick() {
  // Check device compatibility and run any necessary view geometry updates and draw the camera feed.
  ...
  // Run XR lifecycle methods
  XR.runPreRender(Date.now())
  }

XR.runPostRender()

XR.runPostRender()

Description

Executes all lifecycle updates that should happen after rendering.

IMPORTANT: Make sure that onStart has been called before calling runPreRender()/runPostRender().

Parameters

None

Example

// Implement A-Frame components tock() method
  function tock() {
  // Check whether XR is initialized
  ...
  // Run XR lifecycle methods
  XR.runPostRender()
}

XR.stop()

XR.stop()

Parameters

None

Description

While stopped, the camera feed is closed and device motion is not tracked. Must call XR.run() to restart after the engine is stopped.

Example

XR.stop()

XR.version()

string XR.version()

Parameters

None

Description

Get the 8th Wall Web engine version.

Example

console.log(XR.version())

XR.AFrame

A-Frame (https://aframe.io) is a web framework designed for building virtual reality experiences. By adding 8th Wall Web to your A-Frame project, you can now easily build augmented reality experiences for the web.

Adding 8th Wall Web to A-Frame

8th Wall Web can be added to your A-Frame project in a few easy steps:

  1. Include a slightly modified version of A-Frame (referred to as "8-Frame") which fixes some polish concerns:

<script src="//cdn.8thwall.com/web/aframe/8frame-0.8.2.min.js"></script>

  1. Add the following script tag to the HEAD of your page. Replace the X's with your app key:

<script async src="//apps.8thwall.com/xrweb?appKey=XXXXX"></script>

  1. Add an "xrweb" component to your a-scene tag:

<a-scene xrweb>

xrweb Attributes

Component Type Default Description
disableWorldTracking bool false If true, turn off SLAM tracking for efficiency.
cameraDirection string back Desired camera to use. Choose from: back or front

Notes:

  • cameraDirection: World tracking (SLAM) is only supported on the back camera. If you are using the front camera, you must disable world tracking by setting disableWorldTracking: true.

Functions

Function Description
xrwebComponent Creates an A-Frame component which can be registered with AFRAME.registerComponent(). Generally won't need to be called directly.

Example - SLAM enabled (default)

  <a-scene xrweb>

Example - SLAM disabled (image tracking only)

  <a-scene xrweb="disableWorldTracking: true">

Example - Front camera (image tracking only)

  <a-scene xrweb="disableWorldTracking: true; cameraDirection: front">

XR.AFrame.xrwebComponent()

XR.AFrame.xrwebComponent()

Parameters

None

Description

Creates an A-Frame component which can be registered with AFRAME.registerComponent(). This, however, generally won't need to be called directly. On 8th Wall Web script load, this component will be registered automatically if it is detected that A-Frame has loaded (i.e if window.AFRAME exists).

Example

window.AFRAME.registerComponent('xrweb', XR.AFrame.xrwebComponent())

AFrame Events

This section describes the events emitted by the "xrweb" A-Frame component.

You can listen for these events in your web application call a function to handle the event.

Events Emitted

Event Emitted Description
camerastatuschange This event is emitted when the status of the camera changes. See onCameraStatusChange from XR.addCameraPipelineModule for more information on the possible status.
realityerror This event is emitted when an error has occured when initializing 8th Wall Web. This is the recommended time at which any error messages should be displayed. The XrDevice API can help with determining what type of error messaging should be displayed.
realityready This event is emitted when 8th Wall Web has initialized and at least one frame has been successfully processed. This is the recommended time at which any loading elements should be hidden.
screenshoterror This event is emitted in response to the screenshotrequest resulting in an error.
screenshotready This event is emitted in response to the screenshotrequest event being being completed successfully. The JPEG compressed image of the AFrame canvas will be provided.
xrimageloading This event is emitted when detection image loading begins.
xrimagescanning This event is emitted when all detection images have been loaded and scanning has begun.
xrimagefound This event is emitted when an image target is first found.
xrimageupdated This event is emitted when an image target changes position, rotation or scale.
xrimagelost This event is emitted when an image target is no longer being tracked.

camerastatuschange

Description

This event is emitted when the status of the camera changes. See onCameraStatusChange from XR.addCameraPipelineModule for more information on the possible status.

Example:

var handleCameraStatusChange = function handleCameraStatusChange(event) {
  console.log('status change', event.detail.status);

  switch (event.detail.status) {
    case 'requesting':
      // Do something
      break;

    case 'hasStream':
      // Do something
      break;

    case 'failed':
      event.target.emit('realityerror');
      break;
  }
};
let scene = this.el.sceneEl
scene.addEventListener('camerastatuschange', handleCameraStatusChange)

realityerror

Description

This event is emitted when an error has occured when initializing 8th Wall Web. This is the recommended time at which any error messages should be displayed. The XrDevice API can help with determining what type of error messaging should be displayed.

Example:

let scene = this.el.sceneEl
  scene.addEventListener('realityerror', (event) => {
    if (XR.XrDevice.isDeviceBrowserCompatible()) {
      // Browser is compatible. Print the exception for more information.
      console.log(event.detail.error)
      return
    }

    // Browser is not compatible. Check the reasons why it may not be.
    for (let reason of XR.XrDevice.incompatibleReasons()) {
      // Handle each XR.XrDevice.IncompatibilityReasons
    }
  })

realityready

Description

This event is emitted when 8th Wall Web has initialized.

Example:

let scene = this.el.sceneEl
scene.addEventListener('realityready', () => {
  // Hide loading UI
})

screenshoterror

Description

This event is emitted in response to the screenshotrequest resulting in an error.

Example:

let scene = this.el.sceneEl
scene.addEventListener('screenshoterror', (event) => {
  console.log(event.detail)
  // Handle screenshot error.
})

screenshotready

Description

This event is emitted in response to the screenshotrequest event being being completed successfully. The JPEG compressed image of the AFrame canvas will be provided.

Example:

let scene = this.el.sceneEl
scene.addEventListener('screenshotready', (event) => {
  // screenshotPreview is an <img> HTML element
  const image = document.getElementById('screenshotPreview')
  image.src = 'data:image/jpeg;base64,' + event.detail
})

xrimageloading

Description

This event is emitted when detection image loading begins.

imageloading.detail : { imageTargets: {name, metadata, properties} }

Example:

const componentMap = {}

const addComponents = ({detail}) => {
  detail.imageTargets.forEach(({name, metadata, properties}) => {
    // ...
  })
}

this.el.sceneEl.addEventListener('xrimageloading', addComponents)

xrimagescanning

Description

This event is emitted when all detection images have been loaded and scanning has begun.

imagescanning.detail : { imageTargets: {name, metadata, properties} }

xrimagefound

Description

This event is emitted when an image target is first found.

imagefound.detail : { name, position, rotation, scale, scaledWidth, scaledHeight }

Property Description
name The image's name.
position: {x, y, z} The 3d position of the located image.
rotation {w, x, y, z} The 3d local orientation of the located image.
scale A scale factor that should be applied to object attached to this image.
scaledWidth The width of the image in the scene, when multiplied by scale.
scaledHeight The height of the image in the scene, when multiplied by scale.

Example:

AFRAME.registerComponent('my-named-image-target', {
  schema: {
    name: { type: 'string' }
  },
  init: function () {
    const object3D = this.el.object3D
    const name = this.data.name
    object3D.visible = false

    const showImage = ({detail}) => {
      if (name != detail.name) {
        return
      }
      object3D.position.copy(detail.position)
      object3D.quaternion.copy(detail.rotation)
      object3D.scale.set(detail.scale, detail.scale, detail.scale)
      object3D.visible = true
    }

    const hideImage = ({detail}) => {
      if (name != detail.name) {
        return
      }
      object3D.visible = false
    }

    this.el.sceneEl.addEventListener('xrimagefound', showImage)
    this.el.sceneEl.addEventListener('xrimageupdated', showImage)
    this.el.sceneEl.addEventListener('xrimagelost', hideImage)
  }
})

xrimageupdated

Description

This event is emitted when an image target changes position, rotation or scale.

imageupdated.detail : { name, position, rotation, scale, scaledWidth, scaledHeight }

Property Description
name The image's name.
position: {x, y, z} The 3d position of the located image.
rotation: {w, x, y, z} The 3d local orientation of the located image.
scale A scale factor that should be applied to object attached to this image.
scaledWidth The width of the image in the scene, when multiplied by scale.
scaledHeight The height of the image in the scene, when multiplied by scale.

Example:

AFRAME.registerComponent('my-named-image-target', {
  schema: {
    name: { type: 'string' }
  },
  init: function () {
    const object3D = this.el.object3D
    const name = this.data.name
    object3D.visible = false

    const showImage = ({detail}) => {
      if (name != detail.name) {
        return
      }
      object3D.position.copy(detail.position)
      object3D.quaternion.copy(detail.rotation)
      object3D.scale.set(detail.scale, detail.scale, detail.scale)
      object3D.visible = true
    }

    const hideImage = ({detail}) => {
      if (name != detail.name) {
        return
      }
      object3D.visible = false
    }

    this.el.sceneEl.addEventListener('xrimagefound', showImage)
    this.el.sceneEl.addEventListener('xrimageupdated', showImage)
    this.el.sceneEl.addEventListener('xrimagelost', hideImage)
  }
})

xrimagelost

Description

This event is emitted when an image target is no longer being tracked.

imagelost.detail : { name, position, rotation, scale, scaledWidth, scaledHeight }

Property Description
name The image's name.
position: {x, y, z} The 3d position of the located image.
rotation: {w, x, y, z} The 3d local orientation of the located image.
scale A scale factor that should be applied to object attached to this image.
scaledWidth The width of the image in the scene, when multiplied by scale.
scaledHeight The height of the image in the scene, when multiplied by scale.

Example:

AFRAME.registerComponent('my-named-image-target', {
  schema: {
    name: { type: 'string' }
  },
  init: function () {
    const object3D = this.el.object3D
    const name = this.data.name
    object3D.visible = false

    const showImage = ({detail}) => {
      if (name != detail.name) {
        return
      }
      object3D.position.copy(detail.position)
      object3D.quaternion.copy(detail.rotation)
      object3D.scale.set(detail.scale, detail.scale, detail.scale)
      object3D.visible = true
    }

    const hideImage = ({detail}) => {
      if (name != detail.name) {
        return
      }
      object3D.visible = false
    }

    this.el.sceneEl.addEventListener('xrimagefound', showImage)
    this.el.sceneEl.addEventListener('xrimageupdated', showImage)
    this.el.sceneEl.addEventListener('xrimagelost', hideImage)
  }
})

AFrame Event Listeners

This section describes the events that are listened for by the "xrweb" A-Frame component

You can emit these events in your web application to perform various actions:

Event Listener Description
hidecamerafeed Hides the camera feed. Tracking does not stop.
recenter Recenters the camera feed to its origin. If a new origin is provided as an argument, the camera's origin will be reset to that, then it will recenter.
screenshotrequest Emits a request to the engine to capture a screenshot of the AFrame canvas. The engine will emit a screenshotready event with the JPEG compressed image or screenshoterror if an error has occured.
showcamerafeed Shows the camera feed.
stopxr Stop the current XR session. While stopped, the camera feed is stopped and device motion is not tracked.

hidecamerafeed

scene.emit('hidecamerafeed')

Parameters

None

Description

Hides the camera feed. Tracking does not stop.

Example

let scene = this.el.sceneEl
scene.emit('hidecamerafeed')

recenter

scene.emit('recenter', {origin, facing})

Parameters

Parameter Description
origin: {x, y, z} [Optional] The location of the new origin.
facing: {w, x, y, z} [Optional] A quaternion representing direction the camera should face at the origin.

Description

Recenters the camera feed to its origin. If a new origin is provided as an argument, the camera's origin will be reset to that, then it will recenter.

If origin and facing are not provided, camera is reset to origin previously specified by a call to recenter or the last call to updateCameraProjectionMatrix(). Note: with A-Frame, updateCameraProjectionMatrix() initially gets called based on initial camera position in the scene.

Example

let scene = this.el.sceneEl
scene.emit('recenter')

// OR

let scene = this.el.sceneEl
scene.emit('recenter', {
  origin: {x: 1, y: 4, z: 0},
  facing: {w: 0.9856, x:0, y:0.169, z:0}
})

screenshotrequest

scene.emit('screenshotrequest')

Parameters

None

Description

Emits a request to the engine to capture a screenshot of the AFrame canvas. The engine will emit a screenshotready event with the JPEG compressed image or screenshoterror if an error has occured.

Example

const scene = this.el.sceneEl
const photoButton = document.getElementById('photoButton')

// Emit screenshotrequest when user taps
photoButton.addEventListener('click', () => {
  image.src = ""
  scene.emit('screenshotrequest')
})

scene.addEventListener('screenshotready', event => {
  image.src = 'data:image/jpeg;base64,' + event.detail
})

scene.addEventListener('screenshoterror', event => {
  console.log("error")
})

showcamerafeed

scene.emit('showcamerafeed')

Parameters

None

Description

Shows the camera feed.

Example

let scene = this.el.sceneEl
scene.emit('showcamerafeed')

stopxr

scene.emit('stopxr')

Parameters

None

Description

Stop the current XR session. While stopped, the camera feed is stopped and device motion is not tracked.

Example

let scene = this.el.sceneEl
scene.emit('stopxr')

XR.Babylonjs

Babylon.js (https://www.babylonjs.com/) is a complete JavaScript framework for building 3D games and experiences with HTML5 and WebGL. Combined with 8th Wall Web, you can create powerful Web AR experiences.

Tutorial Video:

Description

Provides an integration that interfaces with the BabylonJS environment and lifecyle to drive the Babylon.js camera to do virtual overlays.

Functions

Function Description
xrCameraBehavior Get a behavior that can be attached to a Babylon camera.

XR.Babylonjs.xrCameraBehavior()

XR.Babylonjs.xrCameraBehavior()

Description

Get a behavior that can be attached to a Babylon camera like so: camera.addBehavior(XR.Babylonjs.xrCameraBehavior())

Parameters

None

Returns

A Babylon JS behavior that connects the XR engine to the Babylon camera and starts the camera feed and tracking.

Example

let surface, engine, scene, camera

const startScene = () => {
  const canvas = document.getElementById('renderCanvas')

  engine = new BABYLON.Engine(canvas, true, { stencil: true, preserveDrawingBuffer: true })
  engine.enableOfflineSupport = false

  scene = new BABYLON.Scene(engine)
  camera = new BABYLON.FreeCamera('camera', new BABYLON.Vector3(0, 3, 0), scene)

  initXrScene({ scene, camera }) // Add objects to the scene and set starting camera position.

  // Connect the camera to the XR engine and show camera feed
  camera.addBehavior(XR.Babylonjs.xrCameraBehavior())

  engine.runRenderLoop(() => {
    scene.render()
  })

  window.addEventListener('resize', () => {
    engine.resize()
  })
}

BabylonJS Observables

Image Target Observables

onXrImageLoadingObservable: Fires when detection image loading begins.

onXrImageLoadingObservable : { imageTargets: {name, metadata, properties} }

onXrImageScanningObservable: Fires when all detection images have been loaded and scanning has begun.

onXrImageScanningObservable : { imageTargets: {name, metadata, properties} }

onXrImageFoundObservable: Fires when an image target is first found.

onXrImageFoundObservable : { name, position, rotation, scale, scaledWidth, scaledHeight }

onXrImageUpdatedObservable: Fires when an image target changes position, rotation or scale.

onXrImageUpdatedObservable : { name, position, rotation, scale, scaledWidth, scaledHeight }

onXrImageLostObservable: Fires when an image target is no longer being tracked.

onXrImageLostObservable : { name, position, rotation, scale, scaledWidth, scaledHeight }

Example

scene.onXrImageUpdatedObservable.add(e => {
  target.position.copyFrom(e.position)
  target.rotationQuaternion.copyFrom(e.rotation)
  target.scaling.set(e.scale, e.scale, e.scale)
})

CameraPipelineModule

8th Wall camera applications are built using a camera pipeline module framework. Applications install modules which then control the behavior of the application at runtime.

Refer to XR.addCameraPipelineModule() for details on adding camera pipeline modules to your application.

A camera pipeline module object must have a .name string which is unique within the application. It should implement one or more of the following camera lifecycle methods. These methods will be executed at the appropriate point in the run loop.

During the main runtime of an application, each camera frame goes through the following cycle:

onStart -> onCameraStatusChange (requesting -> hasStream -> hasVideo) -> onProcessGpu -> onProcessCpu -> onUpdate -> onRender

Camera modules should implement one or more of the following camera lifecycle methods:

Function Description
onAppResourcesLoaded Called when we have received the resources attached to an app from the server.
onCameraStatusChange Called when a change occurs during the camera permissions request.
onCanvasSizeChange Called when the canvas changes size.
onDeviceOrientationChange Called when the device changes landscape/portrait orientation.
onException Called when an error occurs in XR. Called with the error object.
onPaused Called when XR.pause() is called.
onProcessCpu Called to read results of GPU processing and return usable data.
onProcessGpu Called to start GPU processing.
onRender Called after onUpdate. This is the time for the rendering engine to issue any WebGL drawing commands. If an application is providing its own run loop and is relying on XR.runPreRender() and XR.runPostRender(), this method is not called and all rendering must be coordinated by the external run loop.
onResume Called when XR.resume() is called.
onStart Called when XR starts. First callback after XR.run() is called.
onUpdate Called to update the scene before render. Data returned by modules in onProcessGpu and onProcessCpu will be present as processGpu.modulename and processCpu.modulename where the name is given by module.name = "modulename".
onVideoSizeChange Called when the canvas changes size.

Note: Camera modules that implement onProcessGpu or onProcessCpu can provide data to subsequent stages of the pipeline. This is done by the module's name.

onAppResourcesLoaded()

onAppResourcesLoaded: ({ framework, imageTargets, version })

Description

Called when we have received the resources attached to an app from the server.

Parameters

Parameter Description
framework The framework bindings for this module provided at the beginning of a frame.
imageTargets An array of image targets with the fields {imagePath, metadata, name}
version The engine version, e.g. 10.1.7.520

Example

XR.addCameraPipelineModule({
  name = 'myPipelineModule',
  onAppResourcesLoaded = ({ framework, version, imageTargets }) => {
    // ...
  },
})

onCameraStatusChange()

onCameraStatusChange: ({ status, stream, video })

Description

Called when a change occurs during the camera permissions request.

Called with the status, and, if applicable, a reference to the newly available data. The typical status flow will be:

requesting -> hasStream -> hasVideo.

Parameters

Parameter Description
status One of [ 'requesting', 'hasStream', 'hasVideo', 'failed' ]
stream: [Optional] The MediaStream associated with the camera feed, if status is hasStream.
video: [Optional] The video DOM element displaying the stream, if status is hasVideo.

The status parameter has the following states:

State Description
requesting In 'requesting', the browser is opening the camera, and if applicable, checking the user permissons. In this state, it is appropriate to display a prompt to the user to accept camera permissions.
hasStream Once the user permissions are granted and the camera is successfully opened, the status switches to 'hasStream' and any user prompts regarding permissions can be dismissed.
hasVideo Once camera frame data starts to be available for processing, the status switches to 'hasVideo', and the camera feed can begin displaying.
failed If the camera feed fails to open, the status is 'failed'. In this case it's possible that the user has denied permissions, and so helping them to re-enable permissions is advisable.

Example

XR.addCameraPipelineModule({
  name = 'camerastartupmodule',
  onCameraStatusChange = ({status}) => {
    if (status == 'requesting') {
      myApplication.showCameraPermissionsPrompt()
    } else if (status == 'hasStream') {
      myApplication.dismissCameraPermissionsPrompt()
    } else if (status == 'hasVideo') {
      myApplication.startMainApplictation()
    } else if (status == 'failed') {
      myApplication.promptUserToChangeBrowserSettings()
    }
  },
})

onCanvasSizeChange()

onCanvasSizeChange: ({ GLctx, videoWidth, videoHeight, canvasWidth, canvasHeight })

Description

Called when the canvas changes size. Called with dimensions of video and canvas.

Parameters

Parameter Description
GLctx The WebGLRenderingContext or WebGL2RenderingContext.
videoWidth The width of the camera feed, in pixels.
videoHeight The height of the camera feed, in pixels.
canvasWidth The width of the GLctx canvas, in pixels.
canvasHeight The height of the GLctx canvas, in pixels.

Example

XR.addCameraPipelineModule({
  name: 'mycamerapipelinemodule',
  onCanvasSizeChange: ({ GLctx, videoWidth, videoHeight, canvasWidth, canvasHeight }) => {
    myHandleResize({ GLctx, videoWidth, videoHeight, canvasWidth, canvasHeight })
  },
})

onDeviceOrientationChange()

onDeviceOrientationChange: ({ GLctx, videoWidth, videoHeight, orientation })

Description

Called when the device changes landscape/portrait orientation.

Parameters

Parameter Description
GLctx The WebGLRenderingContext or WebGL2RenderingContext.
videoWidth The width of the camera feed, in pixels.
videoHeight The height of the camera feed, in pixels.
orientation The rotation of the ui from portrait, in degrees (-90, 0, 90, 180).

Example

XR.addCameraPipelineModule({
  name: 'mycamerapipelinemodule',
  onDeviceOrientationChange: ({ GLctx, videoWidth, videoHeight, orientation }) => {
    // handleResize({ GLctx, videoWidth, videoHeight, orientation })
  },
})

onException()

onException: (error)

Description

Called when an error occurs in XR. Called with the error object.

Parameters

Parameter Description
error The error object that was thrown

Example

XR.addCameraPipelineModule({
  name: 'mycamerapipelinemodule',
    onException : (error) => {
      console.error('XR threw an exception', error)
  },
})

onPaused()

onPaused: ()

Description

Called when XR.pause() is called.

Parameters

None

Example

XR.addCameraPipelineModule({
  name: 'mycamerapipelinemodule',
  onPaused: () => {
    console.log('pausing application')
  },
})

onProcessGpu()

onProcessGpu: ({ framework, frameStartResult })

Description

Called to start GPU processing

Parameters

Parameter Description
framework { dispatchEvent(eventName, detail) } : Emits a named event (imagefound, imageupdated, imagelost) with the supplied detail.
frameStartResult { cameraTexture, GLctx, textureWidth, textureHeight, orientation, videoTime, repeatFrame }

The frameStartResult parameter has the following properties:

Property Description
cameraTexture The WebGLTexture containing camera feed data.
GLctx The WebGLRenderingContext or WebGL2RenderingContext.
textureWidth The width (in pixels) of the camera feed texture.
textureHeight The height (in pixels) of the camera feed texture.
orientation The rotation of the ui from portrait, in degrees (-90, 0, 90, 180).
videoTime The timestamp of this video frame.
repeatFrame True if the camera feed has not updated since the last call.

Returns

Any data that you wish to provide to onProcessCpu and onUpdate should be returned. It will be provided to those methods as processGpuResult.modulename

Example

XR.addCameraPipelineModule({
  name: 'mycamerapipelinemodule',
  onProcessGpu: ({frameStartResult}) => {
    const {cameraTexture, GLctx, textureWidth, textureHeight} = frameStartResult

    if(!cameraTexture.name){
      console.error("[index] Camera texture does not have a name")
    }

    const restoreParams = XR.GlTextureRenderer.getGLctxParameters(GLctx, [GLctx.TEXTURE0])
    // Do relevant GPU processing here
    ...
    XR.GlTextureRenderer.setGLctxParameters(GLctx, restoreParams)

    // These fields will be provided to onProcessCpu and onUpdate
    return {gpuDataA, gpuDataB}
  },
})

onProcessCpu()

onProcessCpu: ({ framework, frameStartResult, processGpuResult })

Description

Called to read results of GPU processing and return usable data. Called with { frameStartResult, processGpuResult }. Data returned by modules in onProcessGpu will be present as processGpu.modulename where the name is given by module.name = "modulename".

Parameter Description
framework The framework bindings for this module provided at the beginning of a frame.
frameStartResult The data that was provided at the beginning of a frame.
processGpuResult Data returned by all installed modules during onProcessGpu.

Returns

Any data that you wish to provide to onUpdate should be returned. It will be provided to that method as processCpuResult.modulename

Example

XR.addCameraPipelineModule({
  name: 'mycamerapipelinemodule',
  onProcessCpu: ({ frameStartResult, processGpuResult }) => {
    const GLctx = frameStartResult.GLctx
    const { cameraTexture } = frameStartResult
    const { camerapixelarray, mycamerapipelinemodule } = processGpuResult

    // Do something interesting with mycamerapipelinemodule.gpuDataA and mycamerapipelinemodule.gpuDataB
    ...
    
    // These fields will be provided to onUpdate
    return {cpuDataA, cpuDataB}
  },
})

onRender()

onRender: ()

Description

Called after onUpdate. This is the time for the rendering engine to issue any WebGL drawing commands. If an application is providing its own run loop and is relying on XR.runPreRender() and XR.runPostRender(), this method is not called and all rendering must be coordinated by the external run loop.

Parameters

None

Example

XR.addCameraPipelineModule({
  name: 'mycamerapipelinemodule',
  onRender: () => {
    // This is already done by XR.Threejs.pipelineModule() but is provided here as an illustration.
    XR.Threejs.xrScene().renderer.render()
  },
})

onResume()

onResume: ()

Description

Called when XR.resume() is called.

Parameters

None

Example

XR.addCameraPipelineModule({
  name: 'mycamerapipelinemodule',
  onResume: () => {
    console.log('resuming application')
  },
})

onStart()

onStart: ({ canvas, GLctx, isWebgl2, orientation, videoWidth, videoHeight, canvasWidth, canvasHeight })

Description

Called when XR starts. First callback after XR.run() is called.

Parameters

Parameter Description
canvas The canvas that backs GPU processing and user display.
GLctx The WebGLRenderingContext or WebGL2RenderingContext.
isWebgl2 True if GLCtx is a WebGL2RenderingContext.
orientation The rotation of the ui from portrait, in degrees (-90, 0, 90, 180).
videoWidth The height of the camera feed, in pixels.
videoHeight The height of the camera feed, in pixels.
canvasWidth The width of the GLctx canvas, in pixels.
canvasHeight The height of the GLctx canvas, in pixels.

Example

XR.addCameraPipelineModule({
  name: 'mycamerapipelinemodule',
  onStart: ({canvasWidth, canvasHeight}) => {
    // Get the 3js scene. This was created by XR.Threejs.pipelineModule().onStart(). The
    // reason we can access it here now is because 'mycamerapipelinemodule' was installed after
    // XR.Threejs.pipelineModule().
    const {scene, camera} = XR.Threejs.xrScene()

    // Add some objects to the scene and set the starting camera position.
    myInitXrScene({scene, camera})

    // Sync the xr controller's 6DoF position and camera paremeters with our scene.
    XR.XrController.updateCameraProjectionMatrix({
      origin: camera.position,
      facing: camera.quaternion,
    })
  },
})

onUpdate()

onUpdate: ({ framework, frameStartResult, processGpuResult, processCpuResult })

Description

Called to update the scene before render. Called with { frameStartResult, processGpuResult, processCpuResult }. Data returned by modules in onProcessGpu and onProcessCpu will be present as processGpu.modulename and processCpu.modulename where the name is given by module.name = "modulename".

Parameters

Parameter Description
framework The framework bindings for this module provided at the beginning of a frame.
frameStartResult The data that was provided at the beginning of a frame.
processGpuResult Data returned by all installed modules during onProcessGpu.
processCpuResult Data returned by all installed modules during onProcessCpu.

Example

XR.addCameraPipelineModule({
  name: 'mycamerapipelinemodule',
  onUpdate: ({ frameStartResult, processGpuResult, processCpuResult }) => {
    if (!processCpuResult.reality) {
      return
    }
    const {rotation, position, intrinsics} = processCpuResult.reality
    const {cpuDataA, cpuDataB} = processCpuResult.mycamerapipelinemodule
    // ...
  },
})

onVideoSizeChange()

onVideoSizeChange: ({ GLctx, videoWidth, videoHeight, canvasWidth, canvasHeight, orientation })

Description

Called when the canvas changes size. Called with dimensions of video and canvas as well as device orientation.

Parameters

Parameters Description
GLctx The WebGLRenderingContext or WebGL2RenderingContext.
videoWidth The width of the camera feed, in pixels.
videoHeight The height of the camera feed, in pixels.
canvasWidth The width of the GLctx canvas, in pixels.
canvasHeight The height of the GLctx canvas, in pixels.
orientation The rotation of the ui from portrait, in degrees (-90, 0, 90, 180).

Example

XR.addCameraPipelineModule({
  name: 'mycamerapipelinemodule',
  onVideoSizeChange: ({ GLctx, videoWidth, videoHeight, canvasWidth, canvasHeight }) => {
    myHandleResize({ GLctx, videoWidth, videoHeight, canvasWidth, canvasHeight })
  },
})

XR.CameraPixelArray

Description

Provides a camera pipeline module that gives access to camera data as a grayscale or color uint8 array.

Functions

Function Description
pipelineModule A pipeline module that provides the camera texture as an array of RGBA or grayscale pixel values that can be used for CPU image processing.

XR.CameraPixelArray.pipelineModule()

XR.CameraPixelArray.pipelineModule({ luminance, maxDimension, width, height })

Description

A pipeline module that provides the camera texture as an array of RGBA or grayscale pixel values that can be used for CPU image processing.

Parameters

Parameter Default Description
luminance [Optional] false If true, output grayscale instead of RGBA
maxDimension: [Optional] The size in pixels of the longest dimension of the output image. The shorter dimension will be scaled relative to the size of the camera input so that the image is resized without cropping or distortion.
width [Optional] The width of the camera feed texture. Width of the output image. Ignored if maxDimension is specified.
height [Optional] The height of the camera feed texture. Height of the output image. Ignored if maxDimension is specified.

Returns

Return value is an object made available to onProcessCpu and onUpdate as:

processGpuResult.camerapixelarray: {rows, cols, rowBytes, pixels}

Property Description
rows Height in pixels of the output image.
cols Width in pixels of the output image.
rowBytes Number of bytes per row of the output image.
pixels A UInt8Array of pixel data.
srcTex A texture containing the source image for the returned pixels.

Example

XR.addCameraPipelineModule(XR.CameraPixelArray.pipelineModule({ luminance: true }))
XR.addCameraPipelineModule({
  name: 'mycamerapipelinemodule',
  onProcessCpu: ({ processGpuResult }) => {
    const { camerapixelarray } = processGpuResult
    if (!camerapixelarray || !camerapixelarray.pixels) {
      return
    }
    const { rows, cols, rowBytes, pixels } = camerapixelarray

    ...
  },

XR.CanvasScreenshot

Description

Provides a camera pipeline module that can generate screenshots of the current scene.

Functions

Function Description
configure Configures the expected result of canvas screenshots.
pipelineModule Creates a camera pipeline module that, when installed, receives callbacks on when the camera has started and when the canvas size has changed.
setForegroundCanvas Sets a foreground canvas to be displayed on top of the camera canvas. This must be the same dimensions as the camera canvas.
takeScreenshot Returns a Promise that when resolved, provides a buffer containing the JPEG compressed image. When rejected, an error message is provided.

XR.CanvasScreenshot.configure()

XR.CanvasScreenshot.configure({ maxDimension, jpgCompression })

Description

Configures the expected result of canvas screenshots.

Parameters

Parameter Description
maxDimension: [Optional] The value of the largest expected dimension.
jpgCompression: [Optional] 1-100 value representing the JPEG compression quality. 100 is little to no loss, and 1 is a very low quality image.

Note: maxDimension defaults to 1280 unless overridden. jpgCompression defaults to 75 unless overridden.

Example

XR.CanvasScreenshot.configure({ maxDimension: 640, jpgCompression: 50 })

XR.CanvasScreenshot.pipelineModule()

XR.CanvasScreenshot.pipelineModule()

Description

Creates a camera pipeline module that, when installed, receives callbacks on when the camera has started and when the canvas size has changed.

Parameters

None

Returns

A CanvasScreenshot pipeline module that can be added via XR.addCameraPipelineModule().

Example

XR.addCameraPipelineModule(XR.CanvasScreenshot.pipelineModule())

XR.CanvasScreenshot.setForegroundCanvas()

XR.CanvasScreenshot.setForegroundCanvas(canvas)

Description

Sets a foreground canvas to be displayed on top of the camera canvas. This must be the same dimensions as the camera canvas.

Only required if you use separate canvases for camera feed vs virtual objects.

Parameters

Parameter Description
canvas The canvas to use as a foreground in the screenshot

Example

const myOtherCanvas = document.getElementById('canvas2')
XR.CanvasScreenshot.setForegroundCanvas(myOtherCanvas)

XR.CanvasScreenshot.takeScreenshot()

XR.CanvasScreenshot.takeScreenshot()

Description

Returns a Promise that when resolved, provides a buffer containing the JPEG compressed image. When rejected, an error message is provided.

Parameters

None

Example

XR.addCameraPipelineModule(XR.canvasScreenshot().cameraPipelineModule())
XR.canvasScreenshot().takeScreenshot().then(
  data => {
    // myImage is an <img> HTML element
    const image = document.getElementById('myImage')
    image.src = 'data:image/jpeg;base64,' + data
  },
  error => {
    console.log(error)
    // Handle screenshot error.
  })
})

XR.FullWindowCanvas

Description

Provides a camera pipeline module that keeps the canvas specified in XR.run() covering the whole window.

Functions

Function Description
pipelineModule Creates a pipeline module that keeps the canvas specified in XR.run() covering the whole window.

XR.FullWindowCanvas.pipelineModule()

XR.FullWindowCanvas.pipelineModule()

Description

Creates a camera pipeline module that, when installed, keeps the canvas specified in XR.run() covering the whole window.

Parameters

None

Returns

A FullWindowCanvas pipeline module module that keeps the canvas specified in XR.run() covering the whole window.

Example

XR.addCameraPipelineModule(XR.FullWindowCanvas.pipelineModule())

XR.GlTextureRenderer

Description

Provides a camera pipeline module that draws the camera feed to a canvas as well as extra utilities for GL drawing operations.

Functions

Function Description
configure Configures the pipeline module that draws the camera feed to the canvas.
create Creates an object for rendering from a texture to a canvas or another texture.
fillTextureViewport Convenience method for getting a Viewport struct that fills a texture or canvas from a source without distortion. This is passed to the render method of the object created by GlTextureRenderer.create()
getGLctxParameters Gets the current set of WebGL bindings so that they can be restored later.
pipelineModule Creates a pipeline module that draws the camera feed to the canvas.
setGLctxParameters Restores the WebGL bindings that were saved with getGLctxParameters.
setTextureProvider Sets a provider that passes the texture to draw.

XR.GlTextureRenderer.configure()

XR.GlTextureRenderer.configure({ vertexSource, fragmentSource, toTexture, flipY })

Description

Configures the pipeline module that draws the camera feed to the canvas.

Parameters

Parameter Description
vertexSource [Optional] The vertex shader source to use for rendering.
fragmentSource [Optional] The fragment shader source to use for rendering.
toTexture [Optional] A WebGlTexture to draw to. If no texture is provided, drawing will be to the canvas.
flipY [Optional] If true, flip the rendering upside-down.

Example

const purpleShader = 
  // Purple.
  ` precision mediump float;
    varying vec2 texUv;
    uniform sampler2D sampler;
    void main() {
      vec4 c = texture2D(sampler, texUv);
      float y = dot(c.rgb, vec3(0.299, 0.587, 0.114));
      vec3 p = vec3(.463, .067, .712);
      vec3 p1 = vec3(1.0, 1.0, 1.0) - p;
      vec3 rgb = y < .25 ? (y * 4.0) * p : ((y - .25) * 1.333) * p1 + p;
      gl_FragColor = vec4(rgb, c.a);
    }`

XR.GlTextureRenderer.configure({fragmentSource: purpleShader})

XR.GlTextureRenderer.create()

XR.GlTextureRenderer.create({ GLctx, vertexSource, fragmentSource, toTexture, flipY })

Description

Creates an object for rendering from a texture to a canvas or another texture.

Parameters

Parameter Description
GLctx The WebGlRenderingContext (or WebGl2RenderingContext) to use for rendering. If no toTexture is specified, content will be drawn to this context's canvas.
vertexSource [Optional] The vertex shader source to use for rendering.
fragmentSource [Optional] The fragment shader source to use for rendering.
toTexture [Optional] A WebGlTexture to draw to. If no texture is provided, drawing will be to the canvas.
flipY [Optional] If true, flip the rendering upside-down.

Returns

Returns an object that has a render function:

Property Description
render({ renderTexture, viewport }) A function that renders the renderTexture to the specified viewport. Depending on if toTexture is supplied, the viewport is either on the canvas that created GLctx, or it's relative to the render texture provided.

The render function has the following parameters:

Parameter Description
renderTexture A WebGlTexture (source) to draw.
viewport The region of the canvas or output texture to draw to; this can be constructed manually, or using GlTextureRenderer.fillTextureViewport().

The viewport is specified by { width, height, offsetX, offsetY } :

Property Description
width The width (in pixels) to draw.
height The height (in pixels) to draw.
offsetX [Optional] The minimum x-coordinate (in pixels) to draw to.
offsetY [Optional] The minimum y-coordinate (in pixels) to draw to.

XR.GlTextureRenderer.fillTextureViewport()

XR.GlTextureRenderer.fillTextureViewport(srcWidth, srcHeight, destWidth, destHeight)

Description

Convenience method for getting a Viewport struct that fills a texture or canvas from a source without distortion. This is passed to the render method of the object created by GlTextureRenderer.create()

Parameters

Parameter Description
srcWidth The width of the texture you are rendering.
srcHeight The height of the texture you are rendering.
destWidth The width of the render target.
destHeight The height of the render target.

Returns

An object: { width, height, offsetX, offsetY }

Property Description
width The width (in pixels) to draw.
height The height (in pixels) to draw.
offsetX The minimum x-coordinate (in pixels) to draw to.
offsetY The minimum y-coordinate (in pixels) to draw to.

XR.GlTextureRenderer.getGLctxParameters()

XR.GlTextureRenderer.getGLctxParameters(GLctx, textureUnit)

Description

Gets the current set of WebGL bindings so that they can be restored later.

Parameters

Parameter Description
GLctx The WebGLRenderingContext or WebGL2RenderingContext to get bindings from.
textureunits The texture units to preserve state for, e.g. [GLctx.TEXTURE0]

Returns

A struct to pass to setGLctxParameters.

Example

const restoreParams = XR.GlTextureRenderer.getGLctxParameters(GLctx, [GLctx.TEXTURE0])
// Alter context parameters as needed
...
XR.GlTextureRenderer.setGLctxParameters(GLctx, restoreParams)
// Context parameters are restored to their previous state

XR.GlTextureRenderer.pipelineModule()

XR.GlTextureRenderer.pipelineModule({ vertexSource, fragmentSource, toTexture, flipY })

Description

Creates a pipeline module that draws the camera feed to the canvas.

Parameters

Parameter Description
vertexSource [Optional] The vertex shader source to use for rendering.
fragmentSource [Optional] The fragment shader source to use for rendering.
toTexture [Optional] A WebGlTexture to draw to. If no texture is provided, drawing will be to the canvas.
flipY [Optional] If true, flip the rendering upside-down.

Returns

Return value is an object made available to onProcessCpu and onUpdate as:

processGpuResult.gltexturerenderer.viewport: { width, height, offsetX, offsetY }

Property Description
width The width (in pixels) to draw.
height The height (in pixels) to draw.
offsetX The minimum x-coordinate (in pixels) to draw to.
offsetY The minimum y-coordinate (in pixels) to draw to.

Example

XR.addCameraPipelineModule(XR.GlTextureRenderer.pipelineModule())
XR.addCameraPipelineModule({
  name: 'mycamerapipelinemodule',
  onProcessCpu: ({ processGpuResult }) => {
    const { viewport } = processGpuResult.gltexturerenderer.viewport
    if (!viewport) {
      return
    }
    const { width, height, offsetX, offsetY } = viewport

    // ...
  },

XR.GlTextureRenderer.setGLctxParameters()

XR.GlTextureRenderer.setGLctxParameters(GLctx, restoreParams)

Description

Restores the WebGL bindings that were saved with getGLctxParameters.

Parameters

Parameter Description
GLctx The WebGLRenderingContext or WebGL2RenderingContext to restore bindings on.
restoreParams The output of getGLctxParameters.

Example

const restoreParams = XR.GlTextureRenderer.getGLctxParameters(GLctx, [GLctx.TEXTURE0])
// Alter context parameters as needed
...
XR.GlTextureRenderer.setGLctxParameters(GLctx, restoreParams)
// Context parameters are restored to their previous state

XR.GlTextureRenderer.setTextureProvider()

XR.GlTextureRenderer.setTextureProvider(({ frameStartResult, processGpuResult, processCpuResult }) => {} )

Description

Sets a provider that passes the texture to draw. This should be a function that take the same inputs as cameraPipelineModule.onUpdate.

Parameters

setTextureProvider() takes a function with the following parameters:

Parameter Description
frameStartResult The data that was provided at the beginning of a frame.
processGpuResult Data returned by all installed modules during onProcessGpu.
processCpuResult Data returned by all installed modules during onProcessCpu.

Example

XR.GlTextureRenderer.setTextureProvider(
  ({processGpuResult}) => {
    return processGpuResult.camerapixelarray ? processGpuResult.camerapixelarray.srcTex : null
  })

XR.PlayCanvas

PlayCanvas (https://www.playcanvas.com/) is an open-source 3D game engine/interactive 3D application engine alongside a proprietary cloud-hosted creation platform that allows for simultaneous editing from multiple computers via a browser-based interface.

Description

Provides an integration that interfaces with the PlayCanvas environment and lifecyle to drive the PlayCanvas camera to do virtual overlays.

Functions

Function Description
runXr Opens the camera and starts running XR in a playcanvas scene.

Getting Started with PlayCanvas

To get started go to https://playcanvas.com/the8thwall and fork one of our sample projects:

Add your App Key

Go to Settings -> External Scripts

The following two scripts should be added added:

https://cdn.8thwall.com/web/xrextras/xrextras.js

https://apps.8thwall.com/xrweb?appKey=XXXXXX

(Note: replace the X's with your own unique App Key obtained from the 8th Wall Console.

Enable "Transparent Canvas"

Go to Settings -> Rendering

Make sure that "Transparent Canvas" is checked

Add XRController

The 8th Wall sample PlayCanvas projects are populated with an XRController game object. If you are starting with a blank project, download xrcontroller.js from https://www.github.com/8thwall/web/tree/master/gettingstarted/playcanvas/scripts/ and attach to an Entity in your scene.

Options:

Option Description
disableWorldTracking If true, turn off SLAM tracking for efficiency.
shadowmaterial Material which you want to use as a transparent shadow receiver (e.g. for ground shadows). Typically this material will be used on a "ground" plane entity positioned at (0,0,0)

XR.PlayCanvas.runXr()

XR.PlayCanvas.runXr( {pcCamera, pcApp}, [extraModules] )

Description

Opens the camera and starts running XR in a playcanvas scene.

Parameters

Parameter Description
pcCamera The playcanvas scene camera to drive with AR.
pcApp The playcanvas app, typically this.app.
extraModules [Optional] An optional array of extra pipeline modules to install.

Example

var xrcontroller = pc.createScript('xrcontroller')

// Optionally, world tracking can be disabled to increase efficiency when tracking image targets.
xrcontroller.attributes.add('disableWorldTracking', {type: 'boolean'})

xrcontroller.prototype.initialize = function() {
  const disableWorldTracking = this.disableWorldTracking

  // After XR has fully loaded, open the camera feed and start displaying AR.
  const runOnLoad = ({pcCamera, pcApp}, extramodules) => () => {
    XR8.xrController().configure({disableWorldTracking})
    XR8.PlayCanvas.runXr({pcCamera, pcApp}, extramodules)
  }

  // Find the camera in the playcanvas scene, and tie it to the motion of the user's phone in the
  // world.
  const pcCamera = XRExtras.PlayCanvas.findOneCamera(this.entity)

  // While XR is still loading, show some helpful things.
  // Almost There: Detects whether the user's environment can support web ar, and if it doesn't,
  //     shows hints for how to view the experience.
  // Loading: shows prompts for camera permission and hides the scene until it's ready for display.
  // Runtime Error: If something unexpected goes wrong, display an error screen.
  XRExtras.Loading.showLoading({onxrloaded: runOnLoad({pcCamera, pcApp: this.app}, [
    // Optional modules that developers may wish to customize or theme.
    XRExtras.AlmostThere.pipelineModule(),       // Detects unsupported browsers and gives hints.
    XRExtras.Loading.pipelineModule(),           // Manages the loading screen on startup.
    XRExtras.RuntimeError.pipelineModule(),      // Shows an error image on runtime error.
  ])})
}

PlayCanvas Events

This section describes the events fired by 8th Wall Web in a PlayCanvas environment.

You can listen for these events in your web application.

Events Emitted

Event Emitted Description
xr:camerastatuschange This event is emitted when the status of the camera changes. See onCameraStatusChange from XR.addCameraPipelineModule for more information on the possible status.
xr:realityerror This event is emitted when an error has occured when initializing 8th Wall Web. This is the recommended time at which any error messages should be displayed. The XrDevice API can help with determining what type of error messaging should be displayed.
xr:realityready This event is emitted when 8th Wall Web has initialized and at least one frame has been successfully processed. This is the recommended time at which any loading elements should be hidden.
xr:screenshoterror This event is emitted in response to the screenshotrequest resulting in an error.
xr:screenshotready This event is emitted in response to the screenshotrequest event being being completed successfully. The JPEG compressed image of the AFrame canvas will be provided.
xr:imageloading This event is emitted when detection image loading begins.
xr:imagescanning This event is emitted when all detection images have been loaded and scanning has begun.
xr:imagefound This event is emitted when an image target is first found.
xr:imageupdated This event is emitted when an image target changes position, rotation or scale.
xr:imagelost This event is emitted when an image target is no longer being tracked.

xr:camerastatuschange

Description

This event is fired when the status of the camera changes. See onCameraStatusChange from XR.addCameraPipelineModule for more information on the possible status.

Example:

const handleCameraStatusChange = function handleCameraStatusChange(detail) {
  console.log('status change', detail.status);

  switch (detail.status) {
    case 'requesting':
      // Do something
      break;

    case 'hasStream':
      // Do something
      break;

    case 'failed':
      this.app.fire('xr:realityerror');
      break;
  }
}
this.app.on('xr:camerastatuschange', handleCameraStatusChange, this)

xr:realityerror

Description

This event is emitted when an error has occured when initializing 8th Wall Web. This is the recommended time at which any error messages should be displayed. The XrDevice API can help with determining what type of error messaging should be displayed.

Example:

this.app.on('xr:realityerror', ({error, isDeviceBrowserSupported, compatibility}) => {
  if (detail.isDeviceBrowserSupported) {
    // Browser is compatible. Print the exception for more information.
    console.log(error)
    return
  }

  // Browser is not compatible. Check the reasons why it may not be in `compatibility`
  console.log(compatibility)
}, this)

xr:realityready

Description

This event is fired when 8th Wall Web has initialized and at least one frame has been successfully processed.

Example:

this.app.on('xr:realityready', () => {
  // Hide loading UI
}, this)

xr:screenshoterror

Description

This event is emitted in response to the xr:screenshotrequest resulting in an error.

Example:

this.app.on('xr:screenshoterror', (detail) => {
  console.log(detail)
  // Handle screenshot error.
}, this)

xr:screenshotready

Description

This event is emitted in response to the xr:screenshotrequest event being being completed successfully. The JPEG compressed image of the PlayCanvas canvas will be provided.

Example:

this.app.on('xr:screenshotready', (event) => {
  // screenshotPreview is an <img> HTML element
  const image = document.getElementById('screenshotPreview')
  image.src = 'data:image/jpeg;base64,' + event.detail
}, this)

PlayCanvas Image Target Events

Image target events can be listened to as this.app.on(event, handler, this).

xr:imageloading: Fires when detection image loading begins.

xr:imageloading : { imageTargets: {name, metadata, properties} }

xr:imagescanning: Fires when all detection images have been loaded and scanning has begun.

xr:imagescanning : { imageTargets: {name, metadata, properties} }

xr:imagefound: Fires when an image target is first found.

xr:imagefound : { name, position, rotation, scale, scaledWidth, scaledHeight }

xr:imageupdated: Fires when an image target changes position, rotation or scale.

xr:imageupdated : { name, position, rotation, scale, scaledWidth, scaledHeight }

xr:imagelost: Fires when an image target is no longer being tracked.

xr:imagelost : { name, position, rotation, scale, scaledWidth, scaledHeight }

Example

const showImage = (detail) => {
  if (name != detail.name) { return }
  const {rotation, position, scale} = detail
  entity.setRotation(rotation.x, rotation.y, rotation.z, rotation.w)
  entity.setPosition(position.x, position.y, position.z)
  entity.setLocalScale(scale, scale, scale)
  entity.enabled = true
}

const hideImage = (detail) => {
  if (name != detail.name) { return }
  entity.enabled = false
}

this.app.on('xr:imagefound', showImage, {})
this.app.on('xr:imageupdated', showImage, {})
this.app.on('xr:imagelost', hideImage, {})

PlayCanvas Event Listeners

This section describes the events that are listened for by 8th Wall Web in a PlayCanvas environment.

You can fire these events in your web application to perform various actions:

Event Listener Description
xr:hidecamerafeed Hides the camera feed. Tracking does not stop.
xr:recenter Recenters the camera feed to its origin. If a new origin is provided as an argument, the camera's origin will be reset to that, then it will recenter.
xr:screenshotrequest Emits a request to the engine to capture a screenshot of the PlayCanvas canvas. The engine will emit a xr:screenshotready event with the JPEG compressed image or xr:screenshoterror if an error has occured.
xr:showcamerafeed Shows the camera feed.
xr:stopxr Stop the current XR session. While stopped, the camera feed is stopped and device motion is not tracked.

xr:hidecamerafeed

this.app.fire('xr:hidecamerafeed')

Parameters

None

Description

Hides the camera feed. Tracking does not stop.

Example

this.app.fire('xr:hidecamerafeed')

xr:recenter

this.app.fire('xr:recenter')

Description

Recenters the camera feed to its origin. If a new origin is provided as an argument, the camera's origin will be reset to that, then it will recenter.

Parameters

Parameter Description
origin: {x, y, z} [Optional] The location of the new origin.
facing: {w, x, y, z} [Optional] A quaternion representing direction the camera should face at the origin.

Example

/*jshint esversion: 6, asi: true, laxbreak: true*/

// taprecenter.js: Defines a playcanvas script that re-centers the AR scene when the screen is
// tapped.

var taprecenter = pc.createScript('taprecenter')

// Fire a 'recenter' event to move the camera back to its starting location in the scene.
taprecenter.prototype.initialize = function() {
  this.app.touch.on(pc.EVENT_TOUCHSTART,
    (event) => { if (event.touches.length !== 1) { return } this.app.fire('xr:recenter')})
}

xr:screenshotrequest

this.app.fire('xr:screenshotrequest')

Parameters

None

Description

Emits a request to the engine to capture a screenshot of the PlayCanvas canvas. The engine will emit a xr:screenshotready event with the JPEG compressed image or xr:screenshoterror if an error has occured.

Example

this.app.on('xr:screenshotready', (event) => {
  // screenshotPreview is an <img> HTML element
  const image = document.getElementById('screenshotPreview')
  image.src = 'data:image/jpeg;base64,' + event.detail
}, this)

this.app.on('xr:screenshoterror', (detail) => {
  console.log(detail)
  // Handle screenshot error.
}, this)

this.app.fire('xr:screenshotrequest')

xr:showcamerafeed

this.app.fire('xr:showcamerafeed')

Parameters

None

Description

Shows the camera feed.

Example

this.app.fire('xr:showcamerafeed')

xr:stopxr

this.app.fire('xr:stopxr')

Parameters

None

Description

Stop the current XR session. While stopped, the camera feed is stopped and device motion is not tracked.

Example

this.app.fire('xr:stopxr')

XR.Sumerian

Amazon Sumerian lets you create VR, AR, and 3D applications quickly and easily. For more information on Sumerian, please see https://aws.amazon.com/sumerian/

Adding 8th Wall Web to Sumerian

Please refer to the following URL for a getting stared guide on using 8th Wall Web with Amazon Sumerian:

https://github.com/8thwall/web/tree/master/gettingstarted/xrsumerian

Functions

Function Description
addXRWebSystem Adds a custom Sumerian System to the provided Sumerian world.

XR.Sumerian.addXRWebSystem()

XR.Sumerian.addXRWebSystem()

Description

Adds a custom Sumerian System to the provided Sumerian world. If the given world is already running (i.e. in a {World#STATE_RUNNING} state), this system will start itself. Otherwise, it will wait for the world to start before running. When starting, this system will attach to the camera in the scene, modify it's position, and render the camera feed to the background. The given Sumerian world must only contain one camera.

Parameters

Parameter Description
world The Sumerian world that corresponds to the loaded scene.

Example

window.XR.Sumerian.addXRWebSystem(world)

Sumerian Events

This section describes the events emitted when using 8th Wall Web with Amazon Sumerian

You can listen for these events in your web application call a function to handle the event.

Events Emitted

Event Emitted Description
camerastatuschange This event is emitted when the status of the camera changes. See onCameraStatusChange from XR.addCameraPipelineModule for more information on the possible status.
screenshoterror This event is emitted in response to the screenshotrequest resulting in an error.
screenshotready This event is emitted in response to the screenshotrequest event being being completed successfully. The JPEG compressed image will be provided.
xrerror This event is emitted when an error has occured when initializing 8th Wall Web.
xrready This event is emitted when 8th Wall Web has initialized and at least one frame has been successfully processed.
xrimageloading This event is emitted when detection image loading begins.
xrimagescanning This event is emitted when all detection images have been loaded and scanning has begun.
xrimagefound This event is emitted when an image target is first found.
xrimageupdated This event is emitted when an image target changes position, rotation or scale.
xrimagelost This event is emitted when an image target is no longer being tracked.

camerastatuschange (Sumerian)

Description

This event is emitted when the status of the camera changes. See onCameraStatusChange from XR.addCameraPipelineModule for more information on the possible status.

Example:

var handleCameraStatusChange = function handleCameraStatusChange(data) {
  console.log('status change', data.status);

  switch (data.status) {
    case 'requesting':
      // Do something
      break;

    case 'hasStream':
      // Do something
      break;

    case 'failed':
      // Do something
      break;
  }
};
window.sumerian.SystemBus.addListener('camerastatuschange', handleCameraStatusChange)

screenshoterror (Sumerian)

Description

This event is emitted in response to the screenshotrequest resulting in an error.

Example:

window.sumerian.SystemBus.addListener('screenshoterror', (data) => {
  console.log(event.detail)
  // Handle screenshot error.
})

screenshotready (Sumerian)

Description

This event is emitted in response to the screenshotrequest event being being completed successfully. The JPEG compressed image of the Sumerian canvas will be provided.

Example:

window.sumerian.SystemBus.addListener('screenshotready', (data) => {
    // screenshotPreview is an <img> HTML element
    const image = document.getElementById('screenshotPreview')
    image.src = 'data:image/jpeg;base64,' + data
  })

xrerror

Description

This event is emitted when an error has occured when initializing 8th Wall Web. This is the recommended time at which any error messages should be displayed. The XrDevice API can help with determining what type of error messaging should be displayed.

Example:

window.sumerian.SystemBus.addListener('xrerror', (data) => {
    if (XR.XrDevice.isDeviceBrowserCompatible) {
      // Browser is compatible. Print the exception for more information.
      console.log(data.error)
      return
    }

    // Browser is not compatible. Check the reasons why it may not be.
    for (let reason of XR.XrDevice.incompatibleReasons()) {
      // Handle each XR.XrDevice.IncompatibleReason
    }
  })

xrready

Description

This event is emitted when 8th Wall Web has initialized and at least one frame has been successfully processed. This is the recommended time at which any loading elements should be hidden.

Example:

window.sumerian.SystemBus.addListener('xrready', () => {
  // Hide loading UI
})

xrimageloading (Sumerian)

Description

This event is emitted when detection image loading begins.

imageloading.detail : { imageTargets: {name, metadata, properties} }

xrimagescanning (Sumerian)

Description

This event is emitted when all detection images have been loaded and scanning has begun.

imagescanning.detail : { imageTargets: {name, metadata, properties} }

xrimagefound (Sumerian)

Description

This event is emitted when an image target is first found.

imagefound.detail : { name, position, rotation, scale, scaledWidth, scaledHeight }

Property Description
name The image's name.
position: {x, y, z} The 3d position of the located image.
rotation {w, x, y, z} The 3d local orientation of the located image.
scale A scale factor that should be applied to object attached to this image.
scaledWidth The width of the image in the scene, when multiplied by scale.
scaledHeight The height of the image in the scene, when multiplied by scale.

xrimageupdated (Sumerian)

Description

This event is emitted when an image target changes position, rotation or scale.

imageupdated.detail : { name, position, rotation, scale, scaledWidth, scaledHeight }

Property Description
name The image's name.
position: {x, y, z} The 3d position of the located image.
rotation: {w, x, y, z} The 3d local orientation of the located image.
scale A scale factor that should be applied to object attached to this image.
scaledWidth The width of the image in the scene, when multiplied by scale.
scaledHeight The height of the image in the scene, when multiplied by scale.

xrimagelost (Sumerian)

Description

This event is emitted when an image target is no longer being tracked.

imagelost.detail : { name, position, rotation, scale, scaledWidth, scaledHeight }

Property Description
name The image's name.
position: {x, y, z} The 3d position of the located image.
rotation: {w, x, y, z} The 3d local orientation of the located image.
scale A scale factor that should be applied to object attached to this image.
scaledWidth The width of the image in the scene, when multiplied by scale.
scaledHeight The height of the image in the scene, when multiplied by scale.

Sumerian Event Listeners

This section describes the events that are listened for by the Sumerian module in 8th Wall Web.

You can emit these events in your web application to perform various actions:

Event Listener Description
recenter Recenters the camera feed to its origin. If a new origin is provided as an argument, the camera's origin will be reset to that, then it will recenter.
screenshotrequest Emits a request to the engine to capture a screenshot of the Sumerian canvas. The engine will emit a screenshotready event with the JPEG compressed image or screenshoterror if an error has occured.

recenter (Sumerian)

window.sumerian.SystemBus.emit('recenter', {origin, facing})

Description

Recenters the camera feed to its origin. If a new origin is provided as an argument, the camera's origin will be reset to that, then it will recenter.

Parameters

Parameter Description
origin: {x, y, z} [Optional] The location of the new origin.
facing: {w, x, y, z} [Optional] A quaternion representing direction the camera should face at the origin.

Example

window.sumerian.SystemBus.emit('recenter')

// OR

window.sumerian.SystemBus.emit('recenter', {
  origin: { x: 1, y: 4, z: 0 },
  facing: { w: 0.9856, x: 0, y: 0.169, z: 0 }
})

screenshotrequest (Sumerian)

window.sumerian.SystemBus.emit('screenshotrequest')

Parameters

None

Description

Emits a request to the engine to capture a screenshot of the Sumerian canvas. The engine will emit a screenshotready event with the JPEG compressed image or screenshoterror if an error has occured.

Example

const photoButton = document.getElementById('photoButton')

// Emit screenshotrequest when user taps
photoButton.addEventListener('click', () => {
  image.src = ""
  window.sumerian.SystemBus.emit('screenshotrequest')
})

window.sumerian.SystemBus.addListener('screenshotready', event => {
  image.src = 'data:image/jpeg;base64,' + event.detail
})

window.sumerian.SystemBus.addListener('screenshoterror', event => {
  console.log("error")
})

XR.Threejs

Description

Provides a camera pipeline module that drives three.js camera to do virtual overlays.

Functions

Function Description
pipelineModule A pipeline module that interfaces with the threejs environment and lifecyle.
xrScene Get a handle to the xr scene, camera and renderer.

XR.Threejs.pipelineModule()

XR.Threejs.pipelineModule()

Description

A pipeline module that interfaces with the threejs environment and lifecyle. The threejs scene can be queried using Threejs.xrScene() after Threejs.pipelineModule()'s onStart method is called. Setup can be done in another pipeline module's onStart method by referring to Threejs.xrScene() as long as XR.addCameraPipelineModule is called on the second module after calling XR.addCameraPipelineModule(Threejs.pipelineModule()).

  • onStart, a threejs renderer and scene are created and configured to draw over a camera feed.
  • onUpdate, the threejs camera is driven with the phone's motion.
  • onRender, the renderer's render() method is invoked.

Note that this module does not actually draw the camera feed to the canvas, GlTextureRenderer does that. To add a camera feed in the background, install the GlTextureRenderer.pipelineModule() before installing this module (so that it is rendered before the scene is drawn).

Parameters

None

Returns

A Threejs pipeline module that can be added via XR.addCameraPipelineModule().

Example

// Add XrController.pipelineModule(), which enables 6DoF camera motion estimation.
XR.addCameraPipelineModule(XR.XrController.pipelineModule())

// Add a GlTextureRenderer which draws the camera feed to the canvas.
XR.addCameraPipelineModule(XR.GlTextureRenderer.pipelineModule())

// Add Threejs.pipelineModule() which creates a threejs scene, camera, and renderer, and
// drives the scene camera based on 6DoF camera motion.
XR.addCameraPipelineModule(XR.Threejs.pipelineModule())

// Add custom logic to the camera loop. This is done with camera pipeline modules that provide
// logic for key lifecycle moments for processing each camera frame. In this case, we'll be
// adding onStart logic for scene initialization, and onUpdate logic for scene updates.
XR.addCameraPipelineModule({
  // Camera pipeline modules need a name. It can be whatever you want but must be unique
  // within your app.
  name: 'myawesomeapp',

  // onStart is called once when the camera feed begins. In this case, we need to wait for the
  // XR.Threejs scene to be ready before we can access it to add content.
  onStart: ({canvasWidth, canvasHeight}) => {
    // Get the 3js scene. This was created by XR.Threejs.pipelineModule().onStart(). The
    // reason we can access it here now is because 'myawesomeapp' was installed after
    // XR.Threejs.pipelineModule().
    const {scene, camera} = XR.Threejs.xrScene()

    // Add some objects to the scene and set the starting camera position.
    myInitXrScene({scene, camera})

    // Sync the xr controller's 6DoF position and camera paremeters with our scene.
    XR.XrController.updateCameraProjectionMatrix({
      origin: camera.position,
      facing: camera.quaternion,
    })
  },

  // onUpdate is called once per camera loop prior to render. Any 3js geometry scene would
  // typically happen here.
  onUpdate: () => {
    // Update the position of objects in the scene, etc.
    updateScene(XR.Threejs.xrScene())
  },
})

XR.Threejs.xrScene()

XR.Threejs.xrScene()

Description

Get a handle to the xr scene, camera and renderer.

Parameters

None

Returns

An object: { scene, camera, renderer }

Property Description
scene The Threejs scene.
camera The Threejs main camera.
renderer The Threejs renderer.

Example

const {scene, camera} = XR.Threejs.xrScene()

XR.XrController

Description

XrController provides 6DoF camera tracking and interfaces for configuring tracking.

Functions

Function Description
configure Configures what processing is performed by XrController (may have performance implications).
hitTest Estimate the 3D position of a point on the camera feed.
pipelineModule Creates a camera pipeline module that, when installed, receives callbacks on when the camera has started, camera proessing events, and other state changes. These are used to calculate the camera's position.
recenter Repositions the camera to the origin / facing direction specified by updateCameraProjectionMatrix and restart tracking.
updateCameraProjectionMatrix Reset the scene's display geometry and the camera's starting position in the scene. The display geometry is needed to properly overlay the position of objects in the virtual scene on top of their corresponding position in the camera image. The starting position specifies where the camera will be placed and facing at the start of a session.

XR.XrController.configure()

XrController.configure({ enableWorldPoints, enableLighting, disableWorldTracking, imageTargets: [] })

Description

Configures the processing performed by XrController (may have performance implications).

Parameters

Parameter Description
enableLighting [Optional] If true, lighting will be provided by XrController.pipelineModule() as processCpuResult.reality.lighting
enableWorldPoints [Optional] If true, worldPoints will be provided by XrController.pipelineModule() as processCpuResult.reality.worldPoints.
disableWorldTracking [Optional] If true, turn off SLAM tracking for efficiency. This needs to be done BEFORE XR.Run() is called.
imageTargets [Optional] List of names of the image target to detect. Can be modified at runtime. Note: All currently active image targets will be replaced with the ones specified in this list.

IMPORTANT: disableWorldTracking: true needs to be set BEFORE both XR.XrController.pipelineModule() and XR.Run() are called.

Example

XR.XrController.configure({ enableLighting: true, enableWorldPoints: true, disableWorldTracking: false })

Example - Disable world tracking

// Disable world tracking (SLAM)
XR.XrController.configure({disableWorldTracking: true})
// Open the camera and start running the camera run loop
// In index.html: <canvas id="camerafeed"></canvas>
XR.run({canvas: document.getElementById('camerafeed')})

Example - Change active image target set

XR.XrController.configure({imageTargets: ['image-target1', 'image-target2', 'image-target3']})

XR.XrController.hitTest()

XrController.hitTest(X, Y, includedTypes = [])

Description

Estimate the 3D position of a point on the camera feed. X and Y are specified as numbers between 0 and 1, where (0, 0) is the upper left corner and (1, 1) is the lower right corner of the camera feed as rendered in the camera that was specified by updateCameraProjectionMatrix. Mutltiple 3d position esitmates may be returned for a single hit test based on the source of data being used to estimate the position. The data source that was used to estimate the position is indicated by the hitTest.type.

Parameters

Parameter Description
X Value between 0 and 1 that represents the horizontal position on camera feed from left to right.
Y Value between 0 and 1 that represents the vertical position on camera feed from top to bottom.
includedTypes List of one or more of: 'FEATURE_POINT', 'ESTIMATED_SURFACE' or 'DETECTED_SURFACE'. Note: Currently only 'FEATURE_POINT' is supported.

Returns

An array of estimated 3D positions from the hit test:

[{ type, position, rotation, distance }]

Property Description
type One of 'FEATURE_POINT', 'ESTIMATED_SURFACE', 'DETECTED_SURFACE', or 'UNSPECIFIED'
position: {x, y, z} The estimated 3D position of the queried point on the camera feed.
rotation: {x, y, z, w} The estimated 3D rotation of the queried point on the camera feed.
distance The estimated distance from the device of the queried point on the camera feed.

Example

const hitTestHandler = (e) => {
  const x = e.touches[0].clientX / window.innerWidth
  const y = e.touches[0].clientY / window.innerHeight
  const hitTestResults = XR.XrController.hitTest(x, y, ['FEATURE_POINT'])
}

XR.XrController.pipelineModule()

XR.XrController.pipelineModule()

Parameters

None

Description

Creates a camera pipeline module that, when installed, receives callbacks on when the camera has started, camera proessing events, and other state changes. These are used to calculate the camera's position.

Returns

Return value is an object made available to onUpdate as:

processCpuResult.reality: { rotation, position, intrinsics, trackingStatus, trackingReason, worldPoints, realityTexture, lighting }

Property Description
rotation: {w, x, y, z} The orientation (quaternion) of the camera in the scene.
position: {x, y, z} The position of the camera in the scene.
intrinsics A column-major 4x4 projection matrix that gives the scene camera the same field of view as the rendered camera feed.
trackingStatus One of 'UNSPECIFIED', 'NOT_AVAILABLE', 'LIMITED' or 'NORMAL'.
trackingReason One of 'UNSPECIFIED', 'INITIALIZING', 'RELOCALIZING', 'TOO_MUCH_MOTION' or 'NOT_ENOUGH_TEXTURE'.
worldPoints: [{id, confidence, position: {x, y, z}}] An array of detected points in the world at their location in the scene. Only filled if XrController is configured to return world points and trackingReason != INITIALIZING.
realityTexture The WebGLTexture containing camera feed data.
lighting: {exposure, temperature} Exposure and temperature of the lighting in your environment.

Dispatched Events

imageloading: Fires when detection image loading begins.

imageloading.detail : { imageTargets: {name, metadata, properties} }

imagescanning: Fires when all detection images have been loaded and scanning has begun.

imagescanning.detail : { imageTargets: {name, metadata, properties} }

imagefound: Fires when an image target is first found.

imagefound.detail : { name, position, rotation, scale, scaledWidth, scaledHeight }

Property Description
name The image's name.
position: {x, y, z} The 3d position of the located image.
rotation {w, x, y, z} The 3d local orientation of the located image.
scale A scale factor that should be applied to object attached to this image.
scaledWidth The width of the image in the scene, when multiplied by scale.
scaledHeight The height of the image in the scene, when multiplied by scale.

imageupdated: Fires when an image target changes position, rotation or scale.

imageupdated.detail : { name, position, rotation, scale, scaledWidth, scaledHeight }

Property Description
name The image's name.
position: {x, y, z} The 3d position of the located image.
rotation: {w, x, y, z} The 3d local orientation of the located image.
scale A scale factor that should be applied to object attached to this image.
scaledWidth The width of the image in the scene, when multiplied by scale.
scaledHeight The height of the image in the scene, when multiplied by scale.

imagelost: Fires when an image target is no longer being tracked.

imagelost.detail : { name, position, rotation, scale, scaledWidth, scaledHeight }

Property Description
name The image's name.
position: {x, y, z} The 3d position of the located image.
rotation: {w, x, y, z} The 3d local orientation of the located image.
scale A scale factor that should be applied to object attached to this image.
scaledWidth The width of the image in the scene, when multiplied by scale.
scaledHeight The height of the image in the scene, when multiplied by scale.

Example - adding pipeline module

XR.addCameraPipelineModule(XR.XrController.pipelineModule())

Example - dispatched events

const logEvent = ({name, detail}) => {
  console.log(`Handling event ${name}, got detail, ${JSON.stringify(detail)}`)
}

XR.addCameraPipelineModule({
  name: 'eventlogger',
  listeners: [
    {event: 'reality.imageloading', process: logEvent },
    {event: 'reality.imagescanning', process: logEvent },
    {event: 'reality.imagefound', process: logEvent},
    {event: 'reality.imageupdated', process: logEvent},
    {event: 'reality.imagelost', process: logEvent},
  ],
})

XR.XrController.recenter()

XR.XrController.recenter()

Parameters

None

Description

Repositions the camera to the origin / facing direction specified by updateCameraProjectionMatrix and restart tracking.

XR.XrController.updateCameraProjectionMatrix()

XR.XrController.updateCameraProjectionMatrix({ cam, origin, facing })

Description

Reset the scene's display geometry and the camera's starting position in the scene. The display geometry is needed to properly overlay the position of objects in the virtual scene on top of their corresponding position in the camera image. The starting position specifies where the camera will be placed and facing at the start of a session.

Parameters

Parameter Description
cam [Optional] { pixelRectWidth, pixelRectHeight, nearClipPlane, farClipPlane }
origin: { x, y, z } [Optional] The starting position of the camera in the scene.
facing: { w, x, y, z } [Optional] The starting direction (quaternion) of the camera in the scene.

cam has the following parameters:

Parameter Description
pixelRectWidth The width of the canvas that displays the camera feed.
pixelRectHeight The height of the canvas that displays the camera feed.
nearClipPlane The closest distance to the camera at which scene objects are visible.
farClipPlane The farthest distance to the camera at which scene objects are visible.

Example

XR.XrController.updateCameraOrigin({
  origin: { x: 1, y: 4, z: 0 },
  facing: { w: 0.9856, x: 0, y: 0.169, z: 0 }
})

XR.XrDevice

Description

Provides information about device compatibility and characteristics.

Properties

Property Type Description
IncompatibilityReasons Enum The possible reasons for why a device and browser may not be compatible with 8th Wall Web.

Functions

Function Description
deviceEstimate Returns an estimate of the user's device (e.g. make / model) based on user agent string and other factors. This information is only an estimate, and should not be assumed to be complete or reliable.
incompatibleReasons Returns an array of XrDevice.IncompatibilityReasons why the device the device and browser are not supported. This will only contain entries if XR.XrDevice.isDeviceBrowserCompatible() returns false.
incompatibleReasonDetails Returns extra details about the reasons why the device and browser are incompatible. This information should only be used as a hint to help with further error handling. These should not be assumed to be complete or reliable. This will only contain entries if XR.XrDevice.isDeviceBrowserCompatible() returns false.
isDeviceBrowserCompatible Returns an estimate of whether the user's device and browser is compatible with 8th Wall Web. If this returns false, XrDevice.incompatibleReasons() will return reasons about why the device and browser are not supported.

XR.XrDevice.IncompatibilityReasons

Enumeration

Description

The possible reasons for why a device and browser may not be compatible with 8th Wall Web.

Properties

Property Value Description
UNSPECIFIED 0 The incompatible reason is not specified.
UNSUPPORTED_OS 1 The estimated operating system is not supported.
UNSUPPORTED_BROWSER 2 The estimated browser is not supported.
MISSING_DEVICE_ORIENTATION 3 The browser does not support device orientation events.
MISSING_USER_MEDIA 4 The browser does not support user media acccess.
MISSING_WEB_ASSEMBLY 5 The browser does not support web assembly.

XR.XrDevice.deviceEstimate()

XR.XrDevice.deviceEstimate()

Description

Returns an estimate of the user's device (e.g. make / model) based on user agent string and other factors. This information is only an estimate, and should not be assumed to be complete or reliable.

Parameters

None

Returns

An object: { locale, os, osVersion, manufacturer, model }

Property Description
locale The user's locale.
os The device's operating system.
osVersion The device's operating system version.
manufacturer The device's manufacturer.
model The device's model.

XR.XrDevice.incompatibleReasons()

XR.XrDevice.incompatibleReasons()

Description

Returns an array of XR.XrDevice.IncompatibilityReasons why the device the device and browser are not supported. This will only contain entries if XR.XrDevice.isDeviceBrowserCompatible() returns false.

Parameters

None

Returns

Returns an array of XrDevice.IncompatibleReasons

Example

const reasons = XR.XrDevice.incompatibleReasons()
for (let reason of reasons) {
  switch (reason) {
    case XR.XrDevice.IncompabilityReasons.UNSUPPORTED_OS:
      // Handle unsupported os error messaging.
      break;
    case XR.XrDevice.IncompabilityReasons.UNSUPPORTED_BROWSER:
       // Handle unsupported browser
       break;
   ...
}

XR.XrDevice.incompatibleReasonDetails()

XR.XrDevice.incompatibleReasonDetails()

Description

Returns extra details about the reasons why the device and browser are incompatible. This information should only be used as a hint to help with further error handling. These should not be assumed to be complete or reliable. This will only contain entries if XrDevice.isDeviceBrowserCompatible() returns false.

Parameters

None

Returns

An object: { inAppBrowser, inAppBrowserType }

Property Description
inAppBrowser The name of the in-app browser detected (e.g. 'Twitter')
inAppBrowserType A string that helps describe how to handle the in-app browser.

XR.XrDevice.isDeviceBrowserCompatible()

XR.XrDevice.isDeviceBrowserCompatible()

Description

Returns an estimate of whether the user's device and browser is compatible with 8th Wall Web. If this returns false, XrDevice.incompatibleReasons() will return reasons about why the device and browser are not supported.

Parameters

None

Returns

True or false.

Converting Models to GLB format

If you are using 8th Wall Web with A-Frame, three.js or Babylon.js, we recommend using 3D models in GLB (glTF 2.0 binary) format in your Web AR experiences. We believe GLB is currently the best format for Web AR with its small file size, great performance and versatile feature support (PBR, animations, etc).

For more information about 3d model best practices and links to a number of GLB converters, please visit:

https://8thwall.com/glb.html

iOS: 8th Wall Web inside an iframe

Starting with iOS 9.2, Safari blocked deviceorientation and devicemotion event access from cross-origin iframes.

This prevents 8th Wall Web (if running inside the iframe) from receiving necessary deviceorientation and devicemotion data required for proper tracking. (See Web Browser Requirements. The result is that the orientation of your digital content will appear to be wrong, and the content will "jump" all over the place when you move the phone.

If you have access to the parent window, it's possible to add a script on the parent page that will send custom messages containing deviceorientation and devicemotion data to 8th Wall Web inside the iframe via JavaScript's postMessage() method. The postMessage() method safely enables cross-origin communication between Window objects; e.g., between a page and an iframe embedded within it. (See https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage)

For your convenience, the library is served at //cdn.8thwall.com/web/iframe/iframe.js

Parameter Description
id The id of the iframe to send deviceorientation/devicemotion data.

Example

  <!-- Send deviceorientation/devicemotion into a cross-domain iOS iframe -->
  <script src="//cdn.8thwall.com/web/iframe/iframe.js?id=my-iframe"></script>
  
  ...
  
  <body>
    <iframe
      id="my-iframe"
      style="border: 0; width: 100%; height: 100%"
      allow="camera;microphone;gyroscope;accelerometer;"
      src="https://www.other-domain.com/my-web-ar/">
    </iframe>
  </body>

Changelog

Release 12.1:

  • Fixes and Enhancements:

    • Increased camera resolution on newer iOS devices.
    • Increased AFrame fps on high-res Android devices.
    • Fixed three.js r103+ raycasting issues.
    • Added support for iPadOS.
    • Fixed memory issue when loading many image targets repeatedly.
    • Minor performance enhances and bug fixes.

Release 12:

  • New Features:

    • Increased image target upload limit to 1000 image targets per app.
    • New API for selecting active image targets at runtime.
    • Apps can now scan for up to 10 image targets simultaneously.
    • Front facing camera is supported in camera framework and image targets.
    • Engine support for PlayCanvas.
  • Fixes:

    • Improved experience for some Android phones with multiple cameras.
  • XRExtras:

    • Improved visual quality on Android Phones.
    • Support for iOS 13 device orientation permissions.
    • Better error handling for missing web assembly on some older versions of iOS.
    • Support for PlayCanvas.

Release 11.2:

  • New Features:

    • iOS 13 motion support.

Release 11.1:

  • Fixes and Enhancements:

    • Reduced memory usage.
    • Improved tracking performance.
    • Enhanced detection of browser capabilities.

Release 11:

  • New Features:

    • Added support for Image Targets.
    • Added support for BabylonJS.
    • Reduced JS binary size to 1MB.
    • Added support running 8th Wall Web inside a cross-origin iframe.
    • Minor API additions.

Release 10.1:

  • New Features:

    • Added support for A-Frame 0.9.0.
  • Fixes:

    • Fixed error when providing includedTypes to XrController.hitTest().
    • Reduced memory usage when tracking over extended distances.

Release 10:

Release 10 adds a revamped web developer console with streamlined developer-mode, access to allowed origins and QR codes. It adds 8th Wall Web support for XRExtras, an open-source package for error handling, loading visualizations, "almost there" flows, and more.

  • New Features:

    • Revamped web developer console.
    • XR Extras provides a convenient solution for:

      • Load screens and requesting camera permissions.
      • Redirecting users from unsupported devices or browsers ("almost there").
      • Runtime error handling.
      • Drawing a full screen camera feed in low-level frameworks like threejs.
    • Added public lighting, hit test interfaces to XrController.
    • Other minor API additions.
  • Fixes:

    • Improved app startup speed.
    • Fixed a framework issue where errors were not propagated on startup.
    • Fixed an issue that could occur with WebGL during initialization.
    • Use window.screen interface for device orientation if available.
    • Fixed a threejs issue that could occur when the canvas is resized.

Release 9.3:

  • New Features:

    • Minor API additions: XR.addCameraPipelineModules() and XR.FullWindowCanvas.pipelineModule()

Release 9.2:

Release 9.1:

  • New Features:

    • Added support for Amazon Sumerian in 8th Wall Web
    • Improved tracking stability and eliminated jitter

Release 9:

  • Initial release of 8th Wall Web!

Device Not Authorized

Issue: When trying to view my Web App, I receive a "Device Not Authorized" error message.

Safari specific:

The situation:

  • While viewing your project, you see 'Device not Authorized' alerts, but
  • apps.8thwall.com/token shows the correct authorization.

Why does this happen?

Safari has a feature called Intelligent Tracking Prevention that can block third party cookies (what we use to authorize your device while you're developing). When they get blocked, we can't verify your device.

Steps to fix:

  1. Close Safari
  2. Turn off Intelligent Tracking Prevention at Settings>Safari>Prevent Cross-Site Tracking
  3. Clear 8th Wall cookies at Settings>Safari>Advanced>Website Data>8thwall.com
  4. Reauthorize from console
  5. Check your project
  6. If not fixed: Clear all cookies at Settings>Safari>Clear History and Website Data
  7. Reauthorize from console

Otherwise

See Invalid App Key steps from #5 onwards for more troubleshooting.

Invalid App Key

Issue: When trying to view my Web App, I receive an "Invalid App Key" error message.

  1. Verify your app key was pasted properly into source code.
  2. Verify you are connecting to your web app via https. This is required for camera access.
  3. Verify you are using a supported browser, see Web Browser Requirements
  4. Verify your device has been properly authorized. On your phone, visit https://apps.8thwall.com/token to view device authorization status.
  5. If you are a member of multiple Web Developer workspaces, make sure that the App Key and Dev Token are from the same workspace.
  6. If your web browser is in Private Browsing or Incognito mode, please exit Private/Incognito mode, re-authorize your device, and try again.
  7. Clear website data & cookies from your web browser, re-authorize your device, and try again.

6DoF Camera Motion Not Working

Issue: As I move my phone, the camera position does not update.

Resolution: Check the position of the camera in your scene. The camera should NOT be at a height (Y) of zero. Set it to Non-Zero value. The Y position of the camera at start effectively determines the scale of virtual content on a surface (e.g. smaller y, bigger content)

Object Not Tracking Surface Properly

Issue: Content in my scene doesn't appear to be "sticking" to a surface properly

Resolution:

To place an object on a surface, the base of the object needs to be at a height of Y=0

Note: Setting the position at a height of Y=0 isn't necesarily sufficient.

For example, if the transform your model is at the center of the object, placing it at Y=0 will result in part of the object living below the surface. In this case you'll need to adjust the vertical position of the object so that the bottom of the object sits at Y=0.

It's often helpful to visualize object positioning relative to the surface by placing a semi-transparent plane at Y=0.

A-Frame example:

<a-plane position="0 0 0" rotation="-90 0 0" width="4" height="4" material="side: double; color: #FFFF00; transparent: true; opacity: 0.5" shadow></a-plane>

Three.js example:

  // Create a 1x1 Plane with a transparent yellow material
  var geometry = new THREE.PlaneGeometry( 1, 1, 1, 1 );   // THREE.PlaneGeometry (width, height, widthSegments, heightSegments)
  var material = new THREE.MeshBasicMaterial( {color: 0xffff00, transparent:true, opacity:0.5, side: THREE.DoubleSide} );
  var plane = new THREE.Mesh( geometry, material );
  // Rotate 90 degrees (in radians) along X so plane is parallel to ground 
  plane.rotateX(1.5708)
  plane.position.set(0, 0, 0)
  scene.add( plane );

Can't connect to "serve" script

Issue:

I'm using the "serve" script (from 8th Wall Web's public GitHub repo: https://github.com/8thwall/web) to run a local webserver on my laptop and it says it's listening on 127.0.0.1. My phone is unable to connect to the laptop using that IP address.

ServeLocalhost

"127.0.0.1" is the loopback address of your laptop (aka "localhost"), so other devices such as your phone won't be able to connect directly to that IP address. For some reason, the serve script has decided to listen on the loopback interface.

Resolution:

Please re-run the serve script with the -i flag and specify the network interface you wish to use.

Example (Mac):

./serve/bin/serve -d gettingstarted/xraframe/ -p 7777 -i en0

Example (Windows):

Note: Run the following command using a standard Command Prompt window (cmd.exe). The script will generate errors if run from PowerShell.

serve\bin\serve.bat -d gettingstarted\xraframe -p 7777 -i WiFi

If you are still unable to connect, please check the following:

  • Make sure that your computer and mobile device are both connected to the same WiFi network.
  • Disable the local firewall running on your computer.
  • To connect, either scan the QR code or make sure to copy the entire "Listening" URL into your browser, including both the "https://" at the beginning and port number at the end.

Help & Support

Need some help? 8th Wall is here to help you succeed. Contact us directly, or reach out to the community to get answers.

Ways to get help:

Slack Email Support Stack Overflow GitHub
Join our public Slack channel to discuss and ask questions with members of the 8th Wall community Email support@8thwall.com to get help directly from our support team Get help and discuss solutions with other 8th Wall Web users by using the 8thwall-web tag Download sample code and view step-by-step setup guides on our GitHub repo


























[1] Intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for 8th Wall’s products remain at the sole discretion of 8th Wall, Inc.