Up to date

This page is up to date for Godot 4.0. If you still find outdated information, please open an issue.

The OpenXR passthrough

Passthrough is a technique where camera images are used to present the environment of the user as the background. This turns a VR headset into an AR headset, often referred to as Mixed Reality or MR.

Note

As passthrough is relatively new there isn’t a singular way this is implemented across platforms. There may be additions in the future so this is a work in progress.

Passthrough extension

OpenXR has a vendor extension for passthrough submitted by Meta. Currently this extension is only supported on Quest but may be adopted by other headsets in the future.

XRInterface has entry points for passthrough so different interfaces can implement this feature. For OpenXRInterface the meta passthrough extension is implemented here.

In code you can call is_passthrough_supported to check if this extension is available. If so you can simply enable passthrough by calling start_passthrough. You can call stop_passthrough to disable passthrough.

This will automatically set the main viewports transparent_bg property to true. It will also result in the camera image being displayed as the background. This will result in the background settings in the environment being ignored and alpha being applied.

Note

For privacy reasons no access is given to the camera image.

Warning

After passthrough is enabled it is possible to change settings that will break passthrough. Be sure not to change the transparent_bg setting or the environment blend mode. This will result in the camera image no longer being visible but you still incur the overhead.

Always use stop_passthrough if you wish to turn off passthrough.

Finally, for using passthrough on the Quest you must set the following export property:

Passthrough through AR

Some of the headsets recently adding OpenXR support have taken a different approach. They simply mimic being an AR device. The Lynx R1 is such a device but others may be doing the same.

The following thus applies to both passthrough devices that mimic AR, and actual AR devices.

If is_passthrough_supported returns false the next step is to call get_supported_environment_blend_modes. This will return a list of supported blend modes for submitting the main render image to OpenXR.

We need to check if XR_ENV_BLEND_MODE_ALPHA_BLEND is present in this list. If so we can tell OpenXR to expect an image that can be alpha blended with a background. To do this, we simply call set_environment_blend_mode(xr_interface.XR_ENV_BLEND_MODE_ALPHA_BLEND).

We must also set transparent_bg to true to ensure we submit the right image.

Putting it together

Putting the above together we can use the following code as a base:

func enable_passthrough() -> bool:
  var xr_interface: XRInterface = XRServer.primary_interface
  if xr_interface and xr_interface.is_passthrough_supported():
    return xr_interface.start_passthrough()
  else:
    var modes = xr_interface.get_supported_environment_blend_modes()
    if xr_interface.XR_ENV_BLEND_MODE_ALPHA_BLEND in modes:
      xr_interface.set_environment_blend_mode(xr_interface.XR_ENV_BLEND_MODE_ALPHA_BLEND)
      return true
    else:
      return false