Sharp_Depth_Injection

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

Sharp_Depth_Injection

★ 10 Open GitHub ↗

External-depth-prior extension for Apple's ml-sharp Gaussian Splat predictor. Houdini cam_zdepth + Nano Banana RGB -> metric Gaussian Splat. Previz pipeline.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Sharp Depth Injection

External-depth-prior extension to Apple's [ml-sharp](https://github.com/apple/ml-sharp) Gaussian Splat predictor. Feeds a metric depth map (e.g. a Houdini `cam_zdepth` AOV, or any camera-space Z depth) into SHARP at inference time so the resulting splat has the correct **metric scale** and plausible parallax under modest camera moves.

Typical pipeline it was built for:

1. **3D scene** (Houdini / Blender / etc.) — beauty render + a camera-space depth AOV (Karma `cam_zdepth` / `hitPz`)
2. **Image model** (an img2img pass, e.g. Nano Banana 2 / Gemini Flash Image) — a photoreal RGB from the beauty
3. **SHARP (this extension)** — input = RGB + metric depth → metric Gaussian Splat

Use case: film / previz — the quality bar is that camera moves of a few degrees of orbit or a metre of dolly look parallactically plausible. The subclass + runtime hot-swap is the only modification path; **Apple's `ml-sharp` is never edited.**

## Layout

The extension is its own package alongside an **unmodified** ml-sharp checkout (a sibling directory):

```
<parent>/
├── ml-sharp/                  ← apple/ml-sharp @ cdb4ddc6, untouched
└── Sharp_Depth_Injection/     ← this repo
    └── sharp_ext/
        ├── external_depth.py  ← ExternalDepthGaussianComposer subclass
        ├── swap_composer.py   ← runtime hot-swap into a built predictor
        └── predict_with_depth.py
```

## Install

```bash
# 1. Clone Apple's ml-sharp at the pinned commit, as a sibling of this repo
git clone https://github.com/apple/ml-sharp.git ../ml-sharp
git -C ../ml-sharp checkout cdb4ddc6

# 2. Create a Python 3.13 venv inside this repo, and activate it
python -m venv .venv
#   Windows:  .venv\Scripts\Activate.ps1      macOS/Linux:  source .venv/bin/activate

# 3. Install ml-sharp's deps + ml-sharp itself + this extension
pip install -r ../ml-sharp/requirements.txt
pip install -e ../ml-sharp        # so `import sharp` resolves
pip install -e .                  # so `import sharp_ext` resolves
```

**CUDA torch:** ml-sharp's requirements pull a **CPU-only** torch by default (silently runs on CPU). Force the CUDA build for your CUDA version, e.g. `pip install --force-reinstall torch torchvision --index-url https://download.pytorch.org/whl/cu128`, then verify `python -c "import torch; print(torch.cuda.is_available())"` prints `True`.

The SHARP checkpoint downloads automatically on first inference to `~/.cache/torch/hub/checkpoints/`.

## Verification

Set `SHARP_TEST_IMAGE` to any local RGB (falls back to `../ml-sharp/data/teaser.jpg` if present). From the activated venv at the repo root:

```bash
python tests/test_decoder_stride.py     # decoder grid resolution (stride 2 -> 768 grid)
python tests/test_self_consistency.py   # feed SHARP's own depth back, expect near-identity
```

## Usage

Depth must be **camera-space Z, in metres, positive forward** (not ray distance, not world-Z). Sky / no-hit pixels that read `0` are remapped to a finite far distance by `load_depth_exr`, so SHARP doesn't plant fake near-camera Gaussians where the sky is.

```python
import imageio.v3 as iio
from sharp_ext import load_depth_exr, predict_image_with_depth
from tests._common import build_predictor, pick_device

device = pick_device()
predictor = build_predictor(device)

image = iio.imread("beauty.png")          # H×W×3 uint8
depth = load_depth_exr("depth.exr")       # H×W float32, metres, +Z forward

# f_px = focal_mm / horizontal_aperture_mm * image_width
gaussians = predict_image_with_depth(
    predictor, image, f_px=877.71,
    external_depth=depth, blend_alpha=0.4,
)
```

See [`SINGLE_FRAME_INTEGRATION.md`](SINGLE_FRAME_INTEGRATION.md) for a self-contained brief to embed the single-frame path in another app.

## HTTP service

A FastAPI service wraps this library for `image (+ metric depth EXR) → .ply` over HTTP — see
[`service/README.md`](service/README.md) and [`service/API.md`](service/API.md). It runs on `:8765`
(one model held in VRAM) and exposes three `depth_method`s: `sharp`, `exr_pixel` (per-pixel blend),
and `exr_grade` (remap SHARP's *own* predicted depth to the EXR's distribution). **For splats prefer
`exr_grade`** (`grade_source=region`, `grade_curve=polynomial`, `grade_min_slope=1.0`): it scales
SHARP's coherent geometry instead of injecting depth per pixel, which avoids the flying / smeared
gaussians `exr_pixel` produces at silhouettes.

## The ~3.4× NDC scale offset

The released checkpoint applies a fixed internal NDC scaling, so an injected metric depth `D` lands at world-Z ≈ 3.4·D (both injection methods, uniform). A `metric_rescale` + `far_cap` fix was tried and **reverted** (`far_cap` flattened far geometry onto a wall). Correct the scale **downstream** in your consumer with a single uniform factor — divide positions **and** Gaussian scales by the measured factor (~3.4) — rather than in the shared service. Plain `sharp` (no injection) is unaffected. See [CLAUDE.md](CLAUDE.md) for the full note.

## Status

Used in production: the extension + FastAPI service (`service/serve.py`) run as an auto-start service (`:8765`, one CUDA model held resident) driven over HTTP by downstream apps. Three depth methods ship (`sharp` / `exr_pixel` / `exr_grade`); the working splat recipe is `exr_grade` / `region` / `polynomial` / `min_slope 1.0`. Self-consistency + decoder-stride tests pass.

Sharp_Depth_Injection

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README