eye_control_example

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

eye_control_example

★ 10 Open GitHub ↗

3D eye control example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Eye control example

## Used Rerun types

[`EyeControls3D`](https://ref.rerun.io/docs/python/0.27.3/common/blueprint_archetypes/#rerun.blueprint.archetypes.EyeControls3D),
[`Transform3D`](https://www.rerun.io/docs/reference/types/archetypes/transform3d), [`Points3D`](https://www.rerun.io/docs/reference/types/archetypes/points3d), [`Pinhole`](https://www.rerun.io/docs/reference/types/archetypes/pinhole), [`EncodedImage`](https://www.rerun.io/docs/reference/types/archetypes/encoded_image), [`Image`](https://www.rerun.io/docs/reference/types/archetypes/image), [`LineStrips3D`](https://www.rerun.io/docs/reference/types/archetypes/line_strips3d), [`Scalars`](https://www.rerun.io/docs/reference/types/archetypes/scalars), [`TextDocument`](https://www.rerun.io/docs/reference/types/archetypes/text_document)

## Background

This example demonstrates how to programmatically configure and control the 3D view camera using the Rerun Blueprint API. By defining camera states in Python, you can precisely tailor your workspace to highlight the most relevant aspects of your data.

In this example, we define several specialized perspectives:

* Top-down overview: a global scene perspective for general spatial awareness.
* Comparative close-up: A focused view designed to analyze trajectory deviations between different localization methods.
* 3rd-person follow:  dynamic camera that tracks the ego vehicle as it moves through the environment.

Finally, we demonstrate how to control the camera at runtime, enabling the creation of cinematic visualizations or automated data storytelling for presentations and datasets.

## Useful resources

Below you will find a collection of useful Rerun resources for this example:

* [Blueprints](https://rerun.io/docs/concepts/blueprints)
* [Building blueprints programmatically](https://rerun.io/docs/howto/build-a-blueprint-programmatically)

## Run the code

To run this example, make sure you have the [Pixi](https://pixi.sh/latest/#installation) package manager installed.

### KTH RPL (indoor handheld dataset)

```sh
pixi run kth_rpl
```

You can type:

```sh
pixi run kth_rpl -h
```

to see all available commands. For example, you can set the voxel size used for downsampling, where the dataset is located, and for how long to sleep in-between frames.

### NTU VIRAL (drone dataset)

```sh
pixi run ntu_viral
```

<!-- You can pass the different scenes as an argument, for example:

```sh
pixi run ntu_viral eee_02
``` -->

Visit [NTU VIRAL dataset](https://ntu-aris.github.io/ntu_viral_dataset/) to see the scene names available.

## Explaination

### KTH RPL

Within `eye_control/kth_rpl.py`, the `get_blueprint` function serves as the orchestrator for our scene layout. This is where we define the specific 3D viewpoints using the [`Spatial3DView`](https://rerun.io/docs/reference/types/views/spatial3d_view) and [`EyeControls3D`](https://ref.rerun.io/docs/python/0.27.3/common/blueprint_archetypes/#rerun.blueprint.archetypes.EyeControls3D) classes.

#### Dynamic camera movement: inside view

The inside view is designed to be dynamic. Rather than a static observation point, we calculate the camera's orientation and look-at target programmatically.

```python
qw = orientation[0]
qx = orientation[1]
qy = orientation[2]
qz = orientation[3]

look_dir = np.array([
    1 - 2 * (qy * qy + qz * qz),
    2 * (qx * qy + qw * qz),
    2 * (qx * qz - qw * qy)
], dtype=np.float32)

look_target = position + look_dir
position -= look_dir

eye_control = rrb.EyeControls3D(
    position=position,
    look_target=look_target,
    eye_up=(0.0, 0.0, 1.0),
    spin_speed=0.0,
    kind=rrb.Eye3DKind.FirstPerson,
    speed=10.0,
)

inside_view = rrb.Spatial3DView(
    origin="/",
    name="Inside View",
    eye_controls=eye_control,
    ...
)
```

By updating the `position` and `look_target` parameters at runtime, you can create "cinematic" traversals through a dataset. In this example, the camera begins in a third-person perspective and then transitions into a continuous loop along the trajectory.

> Tip: You can extend this logic by using splines or interpolation to create even smoother, multi-point camera paths for high-quality video exports.

#### Static perspectives: top-down and outside views

For the top-down and outside views, instead of calculating frames dynamically, we set the `position` and `look_target` values such that they best showcase the relevant spatial data.

```python
top_down_eye_control = rrb.EyeControls3D(
    position=(0.02071, 0.17345, 5.0),
    look_target=(0.01474, 0.25178, 0.00327),
    eye_up=(0.0, 0.0, 1.0),
    spin_speed=0.0,
    kind=rrb.Eye3DKind.FirstPerson,
    speed=10.0,
)

outside_eye_control = rrb.EyeControls3D(
    position=(-3.1584, -0.06133, 0.41640),
    look_target=(-2.2398, -0.00435, 0.02432),
    eye_up=(0.0, 0.0, 1.0),
    spin_speed=0.0,
    kind=rrb.Eye3DKind.FirstPerson,
    speed=10.0,
)

top_down_view = rrb.Spatial3DView(
    origin="/",
    name="Top View",
    eye_controls=top_down_eye_control,
    ...
)

outside_view = rrb.Spatial3DView(
    origin="/",
    name="Outside View",
    eye_controls=outside_eye_control,
    ...
)
```

These views are essential for maintaining context. While the inside view follows the action, these views remain fixed, providing a consistent global frame of reference for the entire scene.

### NTU VIRAL

The NTU VIRAL example uses the same structure as the KTH RPL dataset. The code is located in `eye_control/ntu_viral.py`, specifically within the `get_blueprint` function. To ensure optimal visibility for each unique environment, we define scene-specific camera `positions` and `look_at` targets.

#### Follow view

The follow view tracks the drone as it traverses the scene. By setting the origin to the `BODY_FRAME`, the camera becomes relative to the drone's coordinate system.

In this setup, the [`EyeControls3D`](https://ref.rerun.io/docs/python/0.27.3/common/blueprint_archetypes/#rerun.blueprint.archetypes.EyeControls3D) position is locked 8 meters behind and 3 meters above the drone.

> Note: The z-coordinate is set to -3 because the eye_up vector is defined as negative in the Z-axis.

```python
follow_eye_control = rrb.EyeControls3D(
    position=(-8, 0, -3),
    look_target=(0.0, 0.0, 0),
    eye_up=(0.0, 0.0, -1.0),
    spin_speed=0.0,
    kind=rrb.Eye3DKind.FirstPerson,
    speed=20.0,
)

follow_view = rrb.Spatial3DView(
    name="Follow View",
    eye_controls=follow_eye_control,
    ...
)
```

#### Top-down view

The top-down view provides a global overview of the environment. Unlike static datasets, the NTU VIRAL scenes vary significantly in scale and layout. Consequently, the camera parameters are pulled from the `TOP_DOWN_VIEW_POINT` dictionary based on the active scene.

```python
top_down_eye_control = rrb.EyeControls3D(
    position=TOP_DOWN_VIEW_POINT[scene][0],
    look_target=TOP_DOWN_VIEW_POINT[scene][1],
    eye_up=(0.0, 0.0, 1.0),
    spin_speed=0.0,
    kind=rrb.Eye3DKind.FirstPerson,
    speed=20.0,
)

top_down_view = rrb.Spatial3DView(
    name="Top Down View",
    eye_controls=top_down_eye_control,
    ...
)
```

#### Close-up view

The close-up view is designed for detailed analysis. It focuses on specific segments of the flight path, making it easier to compare the precision and drift of various localization methods. These points of interest are also scene-specific and defined in the `CLOSE_UP_VIEW_POINT` configuration.

```python
close_up_eye_control = rrb.EyeControls3D(
    position=CLOSE_UP_VIEW_POINT[scene][0],
    look_target=CLOSE_UP_VIEW_POINT[scene][1],
    eye_up=(0.0, 0.0, 1.0),
    spin_speed=0.0,
    kind=rrb.Eye3DKind.FirstPerson,
    speed=20.0,
)

close_up_view = rrb.Spatial3DView(
    name="Close-Up View",
    eye_controls=close_up_eye_control,
    ...
)
```

## Acknowledgements

[NTU VIRAL: A Visual-Inertial-Ranging-Lidar Dataset for Autonomous Aerial Vehicle](https://ntu-aris.github.io/ntu_viral_dataset/)

```bibtex
@article{nguyen2022ntu,
  title     = {NTU VIRAL: A Visual-Inertial-Ranging-Lidar Dataset, From an Aerial Vehicle Viewpoint},
  author    = {Nguyen, Thien-Minh and Yuan, Shenghai and Cao, Muqing and Lyu, Yang and Nguyen, Thien Hoang and Xie, Lihua},
  journal   = {The International Journal of Robotics Research},
  volume    = {41},
  number    = {3},
  pages     = {270--280},
  year      = {2022},
  publisher = {SAGE Publications Sage UK: London, England}
}
```

eye_control_example

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README