KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #developer-tools #ai-agent #claude-code #codex #typescript #ai-agents #deepseek-harness #mcp #cli #rust

SlicerVirtualReality

★ 136 Open GitHub ↗

A Slicer extension that enables user to interact with a Slicer scene using virtual reality.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# SlicerVirtualReality

> [!NOTE]
> It is recommended to use a recent Slicer Preview Release, OpenXR interface, and Meta Quest 3 headset.
> Other configurations may work, but not regularly tested.

Extension for 3D Slicer enabling interaction with the 3D scene through virtual reality (VR) or augmented reality (AR).

The extension works with all OpenXR-compatible virtual reality and augmented reality headsets. The recommended headset is the **[Meta Quest 3](#setup-meta-quest)**, which works wirelessly via Air Link or with a USB cable. Other supported headsets include [HTC Vive](#setup-htc-vive), and [HoloLens 2](#setup-hololens-2). The legacy OpenVR backend is also available as a fallback for headsets that do not support OpenXR.

![SlicerVirtualReality Logo](https://raw.githubusercontent.com/KitwareMedical/SlicerVirtualReality/master/SlicerVirtualReality.png)

See the YouTube video below or the [Kitware blog post](https://blog.kitware.com/slicervirtualreality/) for some more background and application examples.

[![Demo: Pedicle screw insertion in virtual reality using Slicer](https://img.youtube.com/vi/F_UBoE4FaoY/0.jpg)](https://www.youtube.com/watch?v=F_UBoE4FaoY)

## Features

Key features:

- View all content of any 3D viewer in Slicer with a single click.
- Display volumes as 2D image slices or volume renderings, render surfaces, points, etc.
- Visualize any 4D datasets using various rendering techniques (including volume rendering) provided by the Sequences extension.
- Align the headset's view to match the viewpoint of the selected 3D view in Slicer.
- Fly around using the touchpad of the right controller: direction is specified by orientation of the controller; speed is determined by the position of the finger on the touchpad (touch at the top to fly forward, touch at the bottom to fly backward).
- Grab and reposition objects using either controller's grab button.
- Translate, rotate, and scale the entire scene using a two-handed gesture (see [How to use controllers](#controllers) for the exact buttons, which vary by headset).
- Advanced volume rendering performance tuning available in the Virtual Reality module to balance image quality and refresh rate.
- Make controller positions available as transforms in the Slicer scene. These transforms can be used in custom modules to reslice volumes (using Volume Reslice Driver module in SlicerIGT extension) or transform any nodes in the scene.

Continuous improvements are made to the feature set. Feedback and ideas for improvement can be submitted via the [issue tracker](https://github.com/KitwareMedical/SlicerVirtualReality/issues).

## Setup

The extension uses **OpenXR** as its primary backend, supporting both virtual reality (VR) and augmented reality (AR, via OpenXR Remoting). The legacy OpenVR backend is retained as a fallback.

**Platform support:** Currently, the extension only functions on Windows computers. Linux support is experimental and untested due to the lack of official OpenXR runtimes and [limited SteamVR support on Linux](https://github.com/ValveSoftware/SteamVR-for-Linux/blob/master/README.md). The extension is available on macOS but lacks backend support due to Apple's lack of OpenXR support for the Vision Pro.

|         | OpenXR (recommended) | OpenVR (legacy)    |
| ------- | -------------------- | ------------------ |
| Windows | :white_check_mark:   | :white_check_mark: |
| Linux   | :x:                  | :x:                |
| macOS   | :x:                  | :x:                |

If you wish to use Virtual Reality extension on Linux or macOS and you have virtual or augmented reality capable hardware working well on your computer through an OpenXR runtime then add a comment in the issue tracker ([macOS](https://github.com/KitwareMedical/SlicerVirtualReality/issues/3) / [Linux](https://github.com/KitwareMedical/SlicerVirtualReality/issues/57)).

**Configuring graphics:** On systems with both integrated display card and high-performance GPU (typically this is the case on laptops with NVidia GPUs), configure the graphics card application settings to use high-performance GPU for `SlicerApp-real.exe` (it is not necessary to use high-performance GPU for the launcher, `Slicer.exe`).

### Meta Quest 3 headset (recommended)

- Install [Meta Horizon Link app][OculusSetup-link] and set up your headset.
- In Meta Horizon Link, click on `Settings`, then in `General` section at `OpenXR Runtime` click `Set Meta Horizon Link as...` button. If this step is skipped then Slicer will not be able to connect to the headset (it will log the error: "Error [GENERAL |  | OpenXR-Loader] : RuntimeManifestFile::FindManifestFiles - failed to find active runtime file in registry")
- Install Slicer and the SlicerVirtualReality extension.
- To view content of the selected 3D view in your headset, click on the "Show scene in virtual reality" button <img src="https://github.com/KitwareMedical/SlicerVirtualReality/raw/master/VirtualReality/Resources/Icons/VirtualRealityHeadset.png" width="24"> on the Slicer toolbar.

OpenXR is used by default. If OpenXR does not work, the legacy OpenVR backend can be used instead:
- Install [Steam][Steam-link], [SteamVR][SteamVR-link] and start the SteamVR home application in your headset.
- In Slicer, click on the wrench icon :wrench: in the toolbar, then select the OpenVR backend and enable rendering.

[OculusSetup-link]: https://www.meta.com/help/quest/articles/headsets-and-accessories/oculus-rift-s/install-app-for-link/

_Supported headsets: **Quest 3** (recommended), Quest Pro, Quest 2, Quest_

_Supported XR modality: Virtual Reality (Augmented Reality via camera pass-through is coming soon)_

### HTC Vive headset

- Install [VIVESetup][VIVESetup-link] and set up your headset.
- Install Slicer and the SlicerVirtualReality extension.
- To view content of the selected 3D view in your headset, click on the "Show scene in virtual reality" button <img src="https://github.com/KitwareMedical/SlicerVirtualReality/raw/master/VirtualReality/Resources/Icons/VirtualRealityHeadset.png" width="24"> on the Slicer toolbar.

[VIVESetup-link]: https://www.vive.com/us/setup/pc-vr/

OpenXR is used by default. If OpenXR does not work, the legacy OpenVR backend can be used instead:
- Start the SteamVR home application in your headset.
- In Slicer, click on the wrench icon :wrench: in the toolbar, then select the OpenVR backend and enable rendering.

_Supported headsets: Pro 2, Pro, Vive_

_Supported XR modality: Virtual Reality_

### HoloLens 2 headset

HoloLens 2 is supported via [OpenXR Holographic Remoting](https://learn.microsoft.com/en-us/windows/mixed-reality/develop/native/holographic-remoting-overview), which streams the rendered scene wirelessly from the PC to the headset.

- On the HoloLens 2, install and start the [Holographic Remoting Player](https://learn.microsoft.com/en-us/windows/mixed-reality/develop/native/holographic-remoting-player) from the Microsoft Store. Note the IP address shown on the HoloLens display.
- Install Slicer and the SlicerVirtualReality extension on the PC.
- In Slicer, click on the wrench icon :wrench: in the toolbar, then:
  1. Select the **OpenXR** backend.
  2. Enable **Remoting** and enter the IP address shown on the HoloLens.
  3. Enable rendering.

> [!NOTE]
> - The PC and HoloLens 2 must be on the same network.
> - Rendering is performed on the PC and streamed to the headset, so a high-performance GPU on the PC is important.

_Supported XR modality: Augmented Reality_

## How to use controllers

| Action | Meta Quest | HTC Vive |
| --- | --- | --- |
| Fly forward | Right thumbstick forward | Touchpad forward |
| Fly backward | Right thumbstick backward | Touchpad backward |
| Grab/move object | Grip button (either controller) | Trigger |
| Transform/scale entire scene | X button (left) + A button (right), pressed together | Grip button (both controllers), pressed together |
| Select interaction mode | _Not available_[^1] | Menu button |

[^1]: On Meta Quest, grabbing/moving objects is always available via either grip button, so there is no separate "interaction mode" to select. The B button is not mapped to anything by default.

![Meta Quest 3 Controller](https://raw.githubusercontent.com/KitwareMedical/SlicerVirtualReality/master/ControllerMetaQuest3.jpg)

![HTC Vive Controller](https://raw.githubusercontent.com/KitwareMedical/SlicerVirtualReality/master/ControllerHtcVive.jpg)

### Fly

Move around in space. Equivalent to physically walking around.
Orient the controller in the intended direction of motion and use `Fly forward` or `Fly backward` action.

> [!NOTE]
> - Flying direction is specified by the orientation of the controller.
> - Speed is proportional to distance of the fingertip from the touchpad center.
> - Maximum speed is configurable in Virtual Reality module.

### Transform objects

Translate/rotate a selected object.

- On Meta Quest: hold down either controller's `Grip` button when the controller is inside a selectable object, move the controller to translate/rotate the object, then release the `Grip` button. No interaction mode needs to be selected first.
- On HTC Vive: make sure interaction mode is set to `Grab` (this is the default), then hold down the `Trigger` when the controller is inside a selectable object, move the controller to translate/rotate the object, then release the `Trigger`.

> [!NOTE]
> - When you grab and move object, a parent transform is automatically created for it (if it has not been under a transform already) and that transform is modified.
> - To move a group of objects together, assign the same parent transform to them. You can do that in _Data_ module's _Transform hierarchy_ tab by drag-and-dropping objects under the same transform (or by double-clicking in the _Applied Transform_ column in  _Data_ module's _Transform hierarchy_ tab and selecting a transform; or by selecting a transform in _Transforms_ module and applying it to all the nodes that must move together).
> - By default all objects are selectable. An object can be made non-selectable (thus non-movable) in Data module / Subject hierarchy tab, right-clicking on the node and unchecking "Toggle Selectable".

## How to use hand interaction

See [Action poses for hand interactions](https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#ext_hand_interaction-the-four-action-poses)

### Transform objects

Translate/rotate a selected object.

Controls: do the "grip" or "pinch" pose when the hand is inside a selectable object

## Other features

### Accessing VR transforms (controller, headset, generic trackers) in Slicer

Go to _Virtual Reality_ module and check the desired checkbox to update linear transform nodes with the various devices' positions.

> [!NOTE]
>
> * The magnification factor in advanced settings affects these transforms.
>
> * Generic trackers are only supported using the legacy OpenVR backend. See issue [#171](https://github.com/KitwareMedical/SlicerVirtualReality/issues/171).

## Frequently asked questions

### How to clip models

3D Slicer can clip models using slice planes. This feature can be used in virtual reality by moving the slice plane using a transform, with the help of _Volume reslice driver_ module.

The models, transforms, etc. only need to be set up once, because the scene can then be saved to file and next time it can be readily used.

#### Simple clipping

Move clipping planes continuously, as the controllers are moved.

- Go to _Models_ module, select the model to clip, and in _Clipping_ section enable _Clip selected model_. By default the _Red_ slice will be chosen for clipping.
- Go to _Virtual Reality_ module and enable making controller transform appear as transforms in the scene.
- Go to _Volume reslice driver_ module (in _SlicerIGT_ extension) to make the controller transform move the _Red_ slice.
- Moved the controller inside the model to clip it

#### Clipping with handles

Grab and move clipping planes using controllers (clipping plane remains in place).

- Go to _Models_ module, select the model to clip, and in _Clipping_ section enable _Clip selected model_. By default the _Red_ slice will be chosen for clipping.
- Add a small model, for example a box to the scene. It can be loaded from STL or created using _Create models_ module (in _SlicerIGT_ extension).
- Move the model using a controller. This creates a parent transform for this model.
- Go to _Volume reslice driver_ module (in _SlicerIGT_ extension) to make the model's parent transform move the _Red_ slice.
- Grab and move the model using the controller to move the clipping plane.

### Rendering is slow

There are several settings that help in increasing the performance of virtual reality rendering:

- If you are using a computer that has two graphics cards (for example, laptops often have an integrated Intel and a high-performance NVidia graphics card), make sure that Slicer is forced to use the high-performance card that the headset is connected to. Most laptops assign applications to the integrated card by default. When you need to select the application executable, choose `SlicerApp-real.exe` (and not the Slicer launcher application `Slicer.exe`).
- Optimize scene for virtual reality button (magic wand icon on toolbar): this switches volume rendering to use GPU, turns off backface culling for all existing models (to see surfaces even when going inside an object), turns off slice intersection visibility for all existing models and segmentations (to make slice view updates faster)
- Settings in virtual reality module panel related to performance (click on wrench icon :wrench: in toolbar):
  - Update rate: Volume rendering quality is set to produce the highest possible quality while keeping the desired frame per second
  - Motion sensitivity: It is very important to keep rendering smooth when moving. This setting detects head movement and significantly lowers volume rendering quality while it is happening. At value of 0 motion is never detected, at high values a little motion triggers the quality change
- Settings in Volume rendering module: open "Advanced" section / "Techniques" tab, try "Adaptive" setting with different "Interactive speed" values. Also try and "Normal" setting: it disables the automatic mechanism that tries to dynamically adjust rendering quality based on predicted rendering time (in some cases the prediction does not work well and results in sub-optimal image quality).
- Switch layout of desktop Slicer: Any additional 3D view to render decreases virtual reality rendering performance. By switching to a layout that contains no 3D view (e.g. Red slice only), this can be prevented

Some scenes are too complex to render fluently by mid-range graphics cards. If the scene includes volume rendering of high-resolution CT for example, then it may be necessary to upgrade to a high-end GPU.

### How to record virtual reality videos?

Use [OBS Studio](https://obsproject.com/) (free) to capture the VR headset view, application window, webcam, etc.:
- **OpenXR (recommended):** Most headset companion apps (Meta Horizon Link for Quest 3, SteamVR) provide a screen-mirror or casting window that OBS can capture directly. On Meta Quest 3, the mirror application can be found at `C:\Program Files\Oculus\Support\oculus-diagnostics\OculusMirror.exe` or similar location. The Quest app on your phone can also cast directly to a TV or browser.
- **OpenVR (SteamVR):** Enable screen mirroring in SteamVR and capture the mirror window with OBS.

Note that [OBS Studio](https://obsproject.com/) may crash if NVidia hardware-based compression is used. If this happens, choose the CPU-based video compression option for recording.

### How to ask questions, report problems, or suggest new features?

Visit [Slicer forum](https://discourse.slicer.org) and search for similar discussions. If you do not find related topics then create a new one. Add _virtual-reality_ tag to make sure people who monitor virtual reality related questions get a notification about your question.

If you are certain that you have found a software bug and no similar issue has been reported in the [issue tracker](https://github.com/KitwareMedical/SlicerVirtualReality/issues) then please submit a new issue.

Please do not use "VR" acronym (you can spell out "virtual reality" instead), because "VR" may mean "volume rendering" just as well as "virtual reality" - you can even do volume rendering in virtual reality in Slicer - and so it becomes confusing very quickly.

## For developers

Information for developers is available in the [Developer Guide](DeveloperGuide.md).

## Contributors

Contributors include:
- Kitware: Jean-Christophe Fillion-Robin, Jean-Baptiste Vimort, Lucas Gandel, Sankhesh Jhaveri
- PerkLab (Queen's University): Csaba Pinter, Andras Lasso
- VASST Lab (Robarts Research Institute): Adam Rankin

## How to cite

Pinter, C., Lasso, A., Choueib, S., Asselin, M., Fillion-Robin, J. C., Vimort, J. B., Martin, K., Jolley, M. A. & Fichtinger, G. (2020). SlicerVR for Medical Intervention Training and Planning in Immersive Virtual Reality. IEEE Transactions on Medical Robotics and Bionics, vol. 2, no. 2, pp. 108-117, May 2020, doi: 10.1109/TMRB.2020.2983199

## License

It is covered by the Apache License, Version 2.0:

http://www.apache.org/licenses/LICENSE-2.0

SlicerVirtualReality

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README