unity-sentis-robustvideomatting-example

KO | EN

gitlite — search

#python #hacktoberfest #react #ai #typescript #llm #go #deep-learning #linux #macos #java #windows #mcp #openai

unity-sentis-robustvideomatting-example

(★ 16)

A simple Unity example demonstrating real-time video background removal using Robust Video Matting (ONNX) with Unity Sentis.

unity-sentis-robustvideomatting-example Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Sentis-RobustVideoMatting-Unity

A Unity implementation of Robust Video Matting using Unity Sentis to run ONNX models. This project was created because there were no existing examples of running RobustVideoMatting ONNX models in Unity.

## Demo

![Result](https://raw.githubusercontent.com/realmorm/unity-sentis-robustvideomatting-example/main/result.gif)

## Features

- Real-time video matting with Unity Sentis
- Support for both MobileNetV3 and ResNet50 models
- GPU-accelerated inference
- Recurrent state feedback (`r1o~r4o` → `r1i~r4i`) for temporal consistency
- Resolution-independent downsample ratio (`TargetShortSide` mode)
- Easy integration with VideoPlayer component

## Downsample Ratio

RVM takes a scalar `downsample_ratio` that decides the internal resolution the network
actually runs at. `VideoMatting.cs` offers two ways to pick it:

| Mode | Behavior |
|------|----------|
| `PixelCountBuckets` | Original approach. Fixed value per total-pixel-count bucket. |
| `TargetShortSide` (default) | Solves for the ratio so the internal **short side** hits `targetShortSide`. |

`PixelCountBuckets` only looks at `width * height`, so two clips with the same pixel count
but different aspect ratios — a 16:9 landscape vs. a tall portrait frame — get the same
ratio while ending up at very different internal resolutions, which shows up as a real
performance gap on portrait sources. `TargetShortSide` keeps the internal resolution stable
regardless of orientation or non-standard resolutions.

The ratio is a single scalar, so simply matching the short side would leave the long side
misaligned and the model dies in the U-Net skip connections (size mismatch in `Concat`).
The search therefore only accepts candidates where the long side lands on an exact integer
**and** both sides are multiples of `downsampleAlignment` (16 by default). If no candidate
qualifies it logs a warning and falls back to `PixelCountBuckets`.

At 1920x1080 the default `targetShortSide = 432` resolves to ratio `0.4` — identical to the
old bucket value — so standard resolutions behave exactly as before.

Enable `logDownsampleRatio` to print the applied ratio and internal resolution
(logged only when the value changes).

## Webcam Support

While webcam support is not implemented in this project, it can be easily achieved by following the same approach as `VideoPlayerController.cs`. Simply feed the webcam texture to VideoMatting in a similar manner. The sample code is minimal and straightforward, making it easy to adapt for your own implementation.

## Performance

Tested on **Mac Mini M4 Pro (24GB RAM)** with **1920x1080 video**
- Unity Version: 6000.0.37f1
- Sentis Version: 2.1.3

| Model | Performance |
|-------|-------------|
| rvm_mobilenetv3_fp32.onnx | 70~ FPS |
| rvm_resnet50_fp32.onnx | 40~ FPS |

### Notes on the inference loop

- **No CPU readback.** The tensors go straight to render textures via `TextureConverter`.
  Anything that pulls tensor data back to the CPU (`ReadbackAndCloneAsync`, `DownloadToArray`,
  the `Tensor<T>` indexer on GPU-resident data) stalls the pipeline every frame.
- **One reusable `CommandBuffer`.** The static `TextureConverter.ToTensor` /
  `RenderToTexture` helpers each allocate a throwaway `CommandBuffer` internally and never
  dispose it. Recording into a buffer you own avoids three allocations per frame and lets the
  two output blits share a single `Graphics.ExecuteCommandBuffer`.
- **`enableRandomWrite` on the result textures.** `RenderToTexture` writes through a UAV. If the
  render texture was created without it, Sentis quietly calls `Release()` + `Create()` on it the
  first time. Creating it correctly up front avoids the churn.
- **No depth buffer on the result textures.** `ForegroundRT` / `AlphaRT` are pure compute targets;
  a 24-bit depth+stencil surface on each is wasted VRAM.

### Further ideas (unmeasured)

- `Worker.ScheduleIterable()` spreads the per-layer CPU dispatch cost across frames. It trades
  latency for a flatter frame time, which helps if `Schedule()` shows up as a main-thread spike.
- `AlphaRT` only needs one channel — the shader samples `.r`. An `R8` render texture would cut its
  bandwidth 4x, but UAV (`LoadStore`) support for `R8` is not universal, so gate it behind
  `SystemInfo.IsFormatSupported`.
- `CompositionWithThreshold.shader` branches on `_UseBackground` and `_AlphaSmoothness` as uniforms.
  Turning them into `shader_feature` keywords removes the branch from a full-screen fragment pass.

## Setup

1. Download ONNX models from the [official RobustVideoMatting repository](https://github.com/PeterL1n/RobustVideoMatting)
2. Place the downloaded model file in your Unity project
3. Assign the model to the `ModelAsset` field in `VideoMatting.cs`
4. Configure your video input and run the scene

## Model Downloads

Get the ONNX models from:
- [rvm_mobilenetv3_fp32.onnx](https://github.com/PeterL1n/RobustVideoMatting/releases/download/v1.0.0/rvm_mobilenetv3_fp32.onnx)
- [rvm_mobilenetv3_fp16.onnx](https://github.com/PeterL1n/RobustVideoMatting/releases/download/v1.0.0/rvm_mobilenetv3_fp16.onnx)
- [rvm_resnet50_fp32.onnx](https://github.com/PeterL1n/RobustVideoMatting/releases/download/v1.0.0/rvm_resnet50_fp32.onnx)
- [rvm_resnet50_fp16.onnx](https://github.com/PeterL1n/RobustVideoMatting/releases/download/v1.0.0/rvm_resnet50_fp16.onnx)

## License

This project follows the same license as the original RobustVideoMatting project.

unity-sentis-robustvideomatting-example

unity-sentis-robustvideomatting-example Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

unity-sentis-robustvideomatting-example

unity-sentis-robustvideomatting-example Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README