XR Components

Cocos CreatorXR empowers entities through the combination of components, and different functional systems manage entities based on their different characteristics. Therefore, all XR-related features in Cocos Creator are driven by encapsulated XR components.

The functional components in Cocos CreatorXR consist of five main parts:

• Device Mapping
• Interaction Component
• Interaction Constraint
• Virtual Movement
• XR UI

After enabling the xr-plugin extension, you can add XR-related functional components to objects in the scene by clicking the Add Component button in the Inspector. In the component list that appears, navigate to the XR category and select the desired XR component, and then select the corresponding component under that category.

Device Mapping Components

This component is mainly used to synchronize input/output information between physical devices in the real world and agent nodes in the virtual world. It ensures consistency between user interactions on XR devices and feedback in the virtual world.

The main components are as following:

TrackingOrigin

HMDCtrl

HMD (Head Mounted Display) controller is responsible for handling the head-mounted display devices. It can be considered as the general category for XR glasses devices that have the capability of rendering stereoscopic images. This component defines the parameters related to the rendering output for XR glasses.

PoseTracker

PoseTracker is a component for tracking the position and orientation of an object.

TargetEye

Specifies the target eye.

XRController

Abstract component for controllers.

ARCameraMgr

Properties for the mobile handheld device camera in AR.

Note: Starting from version 1.2.0, light estimation supports two modes: Basic and HDR

To enable the Basic mode, set the Envmap property of the Skybox in the scene to LDR.

To enable the HDR mode, set the Envmap property of the Skybox in the scene to HDR.

Note: Currently, the Basic mode is only supported on iOS platforms.

Interaction Components

An interaction operation requires coordination between two objects: the interaction subject and the interactable object. Correspondingly, the interaction components can be divided into two categories: Interactor and Interactable.

RayInteractor

RayInteractor is a component for ray-based interaction.

Note: The functionality of projectile curves and Bezier curves requires the extended version >= v1.1.0 and the Cocos Creator version >= 3.7.1.

DirectInteractor

DirectInteractor is a component for direct interaction.

XRGazeInteractor

GazeInteractor allows interaction with objects (UI) by gazing at them with the center point of the HMD when there are no controllers or other input devices available. After gazing for a certain period, the interaction behavior is triggered.

Note: GazeInteractor requires an extension version of >= v1.1.0 and Cocos Creator version of >=3.7.1.

ScreenTouchInteractor

ScreenTouchInteractor is a component for screen gesture interaction.

Note:

1. ScreenTouchInteractor requires an extension version of >=v1.1.0 and an editor version of >=3.7.1. The toggle functionality for each gesture action requires an extension version of >=v1.2.0.
2. Double-finger gesture input is not supported on the WebXR platform.

GrabInteractable

The GrabInteractable component allows an object to be grabbed and interacted with.

Teleportable

The Teleportable component allows an object to be teleported.

Selectable

The Selectable component allows an object to be selected and enables displacement, rotation, and scaling actions while selected.

Note: The selection interaction requires an extension version of >= v1.1.0 and Cocos Creator version of >= 3.7.1.

Placeable

The Placeable component allows an object to be placed in space, on an AR Plane, or an AR Mesh using a specified method.

Note: The placement interaction requires an extension version of >= v1.1.0 and Cocos Creator version of >= 3.7.1.

Events

During the development process, users often encounter logic that relies on signals from XR components to trigger certain behaviors. To allow users to focus on logic development without worrying about the notification relationship between the components, Cocos CreatorXR encapsulates a set of event signals in some interactive components that are commonly used for event dependency. Users only need to bind the desired objects and methods to specific signals to trigger the desired behavior.

Event Signals

Event signals are divided into the following three types:

• Hover：Triggered when the ray from the XR input device hovers over an object.
• Select：Triggered when the Select button mapped to the input device is pressed.
• Active：Triggered when the Active button mapped to the input device is pressed.

The interaction behaviors are described as follows:

These event signals can be used to handle input states in specific scenarios and can be selected in the following components.

Interactive Event Components

Interactive Components

The interactive components are as follows:

• InteractorEvents： Interactor event component.
• InteractableEvents：Interactable object event component.

InteractorEvents

Audio Events

Audio Events can trigger different types of events based on the event signals and play specific audio clips when triggered.

By enabling or disabling the event signals on the right side, you can select which event signals to use. Once enabled, you can choose different audio resources from the dropdown menu on the right side of the AudioClip property.

Haptic Events

Note: This event requires an extension version of >= v1.1.0 and the Cocos Creator version of >=3.7.1.

By selecting the radio button on the right side of the event signals, you can adjust the controller’s vibration feedback. Vibration feedback can provide users with a more realistic tactile experience.

Haptic Intensity：The intensity of the vibration [0,1]

Duration：The duration of the vibration.

Interactor Events

Enter any integer value in the input box on the right side to add elements to the Interactor Events array.

After adding an event, you can configure the event callback:

• Node：The node that receives the callback.
• Component：The component of the callback. Method: The callback method.
• CustomEventData：Custom data for the event, which will be passed as an argument to the callback method.

InteractableEvents

Interaction Constraint Components

For interactive objects that have the Selectable component attached, you can use interaction restriction components to limit their interaction effects.

MinMaxScaleConstrain

RotationAxisConstrain

Virtual Movement Components

In most VR projects, users navigate through virtual scenes from a first-person perspective. This type of movement behavior generally does not rely on real-world movement feedback, as tracking of positional data is limited by the physical space. Therefore, we need a component that can drive movement behaviors by accepting input signals from controllers, similar to how it’s done in traditional 3D games. This component is called the Locomotion Component.

LocomotionChecker

Movement checker.

Teleporter

Teleportation driver component.

SharpTurner

Instant turning driver.

ContinuousTurner

Continuous turning driver.

ContinuousMover

Translation movement driver.

XR UI

The traditional way of generating UI controls is to draw them on a canvas, which doesn’t have depth information (immutable position attributes). This results in the canvas being projected onto the screen, and interaction with the UI controls requires feedback from the screen.

However, XR devices have dual lenses and lack interactive capabilities, which doesn’t meet the requirements of XR projects. Therefore, we need to change the interaction method of UI controls to interact with them in 3D space using interactors. This involves detaching UI controls from the canvas, allowing them to exist independently in space with complete positional attributes and collision detection capabilities.

XR UI is an extension of 2D UI and can be created by adding the XRUI component. For information on how to use 2D/UI components, refer to the 2D Objects Overview.

Creating XR UI

To create XR UI, go to Hierarchy -> Create -> XR -> XRUI.

Compared to traditional UI controls, XR UI introduces additional components for collision detection and interaction. As shown in the image, the RaycastChecker and BoxCollider components enable XR input interaction with the UI.

Note: For UI in 3D space, the root node requires the RenderRoot2D component for proper rendering.

Converting Existing UI

If you have already created UI and want to convert it from a 2D canvas to XR UI, you can follow these steps.

Method 1

Right-click in the Hierarchy and create an empty node (name it UIRoot). Add the RenderRoot2D component to the node, which will automatically add the UITransform component.

Detach the existing 2D UI nodes from the Canvas and move them under the UIRoot node. Modify the position and layer properties of the buttons. Set the Layer property of the button and its child nodes to match the layer of the Camera (e.g., DEFAULT).

Add a material to the button and its child nodes. In the Assets window, navigate to xr-plugin -> xr -> res -> default_materials and select xr-3dui-sprite-material. Drag it onto the CustomMaterial property of the Sprite component.

Add the RaycastChecker component for ray interaction to the button. Click on the button node and, in the Inspector panel, click the “Add Component” button. Select XR -> UI -> RaycastChecker.

The RaycastChecker and BoxCollider components will appear in the Inspector panel. Ensure that the xy values of the BoxCollider’s size property match the Content Size value of the node’s UITransform. You can replace the BoxCollider with other 3D colliders as needed, as long as they fit the UI components.

Move the button node to the desired position in the scene and adjust the Rotation and Scale values to meet the design requirements.

Once all the UI components have been added, delete the old Canvas node.

With these steps, you have successfully converted your existing UI to XR UI.

Method 2

Starting from version v1.1.0, there is a new feature to convert 2D UI to XR UI with a single click.

Right-click on the traditional 2D UI in the scene. In the context menu, you will see the option “2DUI to XRUI”.

After successful conversion, the UI node will automatically add the following three components, and you can adjust their spatial properties in the cc.Node.

Note: Once UI is converted to XRUI, it cannot be converted back.

XR Keyboard

To add an EditBox as XR UI and attach a keyboard to it, create a child node named “KeyboardRoot” (you can choose any name) under the EditBox node and adjust the position information of KeyboardRoot as needed (you can temporarily place the XR Keyboard under the node for adjustment).

Create an XR Keyboard object: In the Asset Browser, navigate to xr-plugin -> xr -> res -> default_prefab and select XR Keyboard. Drag it into the scene.

Add the XRKeyboardInputField component to the EditBox node and bind it to SuspendTransform and XRKeyboard by dragging the nodes into the corresponding fields.

Raycast Material

When interacting with XR UI using raycasting, you need to assign the xr-default-line-material to the ray. You can find this material in Assets -> xr-plugin -> xr -> res -> default_materials.