CasaMaestro

KO | EN

gitlite — search

#javascript #python #react #typescript #llm #go #android #linux #nodejs #awesome #cli #kubernetes #vue #pytorch

(★ 14)

[ECCV 2026] CasaMaestro: Multi-View Panoramas for House-Scale 3D Reconstruction

CasaMaestro 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">CasaMaestro: Multi-View Panoramas for House-Scale 3D Reconstruction</h1>

<div align="center">
 
 <a href="https://george-attano.github.io">Yuzhou Ji</a>1*&nbsp;&nbsp;
 Xiaotian Yang1*&nbsp;&nbsp;
 <a href="https://zhipengzhang.cn/">Zhipeng Zhang</a>1†
 
 
 1AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University
 
 
 * Equal Contribution &nbsp;&nbsp;&nbsp;
 † Corresponding Author
 
</div>

![CasaMaestro teaser](https://raw.githubusercontent.com/AutoLab-SAI-SJTU/CasaMaestro/feat/CasaMeastro/assets/images/Traza.png)

## 📣 News

- [July 2] Code release

- [July 2] Model release

## ✨ Overview

We present CasaMaestro, a feedforward model that can take only twenty to fifty sparse multi-view indoor panoramas as input and 
directly predicts metric depth along with camera poses, allowing immediate point-cloud reconstruction of the entire house with full coverage.

CasaMaestro is the first model that supports house-scale reconstruction with multi-view panoramas, which is built following the minimal modeling strategy of Depth Anything 3.
We implement simple yet effective ERP data augmentation along with extra attention in final pose head.

Although no geometric modules specifically used (e.g., panoramic embedding), experiments show that our model achieves excellent quality in the task of sparse-viewport house-scale reconstruction, indicating that multiview 3D with panoramas is promising.

## 📊 Quantitative Results

The following tables show the quantitative pose metrics comparison of CasaMaestro and other baseline models.

*Note: The `*` denotes finetune or re-implementation on the Realsee training set, and `°` indicates panoramic methods.*

### Table 1: Results on Realsee-Real

| Method | AUC@10 ↑ | AUC@20 ↑ | AUC@30 ↑ | Rot. Mean ↓ | Trans. Mean ↓ | Pose Mean ↓ |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| PanoPose*° | 0.042 | 0.115 | 0.166 | 62.20 | 58.27 | 90.12 |
| SPR*° | 0.086 | 0.145 | 0.191 | 59.43 | 56.12 | 86.23 |
| VGGT | 0.189 | 0.313 | 0.388 | 37.74 | 32.31 | 47.65 |
| VGGT-Finetune* | 0.276 | 0.379 | 0.411 | 30.12 | 10.23 | 38.42 |
| PI3 | 0.222 | 0.365 | 0.450 | 26.69 | 25.48 | 36.61 |
| PI3-Finetune* | 0.301 | 0.397 | 0.510 | 21.36 | 7.442 | 28.97 |
| DepthAnything3 | 0.083 | 0.173 | 0.235 | 54.57 | 49.79 | 69.51 |
| StreamVGGT | 0.033 | 0.039 | 0.062 | 52.46 | 5.351 | 70.58 |
| InfiniteVGGT | 0.271 | 0.412 | 0.487 | 29.82 | 4.488 | 30.01 |
| **CasaMaestro° (Ours)** | **0.727** | **0.859** | **0.903** | **2.608** | **2.132** | **3.225** |

### Table 2: Results on Realsee-Syn

| Method | AUC@10 ↑ | AUC@20 ↑ | AUC@30 ↑ | Rot. Mean ↓ | Trans. Mean ↓ | Pose Mean ↓ |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| PanoPose*° | 0.082 | 0.135 | 0.179 | 57.44 | 57.39 | 88.04 |
| SPR*° | 0.093 | 0.162 | 0.201 | 51.24 | 50.16 | 67.42 |
| VGGT | 0.118 | 0.226 | 0.292 | 48.93 | 42.99 | 61.49 |
| VGGT-Finetune* | 0.244 | 0.386 | 0.423 | 29.20 | 9.672 | 30.78 |
| PI3 | 0.237 | 0.347 | 0.410 | 38.82 | 21.77 | 28.65 |
| PI3-Finetune* | 0.310 | 0.412 | 0.507 | 23.43 | 5.320 | 25.31 |
| DepthAnything3 | 0.030 | 0.073 | 0.110 | 71.24 | 59.67 | 87.20 |
| StreamVGGT | 0.007 | 0.007 | 0.010 | 57.84 | 11.56 | 102.0 |
| InfiniteVGGT | 0.068 | 0.196 | 0.288 | 61.37 | 6.991 | 61.83 |
| **CasaMaestro° (Ours)** | **0.792** | **0.892** | **0.927** | **1.553** | **1.682** | **2.281** |

## 🚀 Quick Start

### Installation

Our method is built on top of Depth Anything 3 (DA3), so the environment setup is consistent with the original DA3 project. You can follow the DA3 installation procedure if you already have that workflow in place.

```bash
pip install 'torch>=2' torchvision xformers
pip install -e .
```

### Run Inference

We provide a small example dataset so readers can quickly test the pipeline. This sample comes from the Realsee3D dataset.

`casameastro_vis.py` runs inference on a single panoramic scene and exports a merged point cloud. When using Realsee3D data, please enable `--use_mask`. This applies to both real and synthetic Realsee3D scenes as long as the data follows the expected directory structure.

For this script, we recommend using the `CasaMeastro_with_mask.pt` checkpoint released on Hugging Face together with the `da3-pano-large.yaml` configuration file.

```bash
python casameastro_vis.py \
  --scene_root example/scene_00001 \
  --cfg path/to/da3-pano-large.yaml  \
  --ckpt /path/to/CasaMeastro_with_mask.pt \
  --out_ply ./output/scene_00001.ply \
  --max_side 448 \
  --device cuda \
  --use_mask
```

If you want to limit the number of fused views or the number of sampled points, you can use a command such as:

```bash
python casameastro_vis.py \
  --scene_root example/scene_00001 \
  --cfg path/to/da3-pano-large.yaml \
  --ckpt /path/to/CasaMeastro_with_mask.pt \
  --out_ply ./output/scene_00001_sparse.ply \
  --max_views 20 \
  --max_points_per_view 10000 \
  --conf_quantile 0.1 \
  --height_quantile 0.5 \
  --use_mask
```

We also provide `casameastro_vis_no_mask.py` for visualizing synthetic scenes that do not use masks. For this script, we recommend using `da3-base.yaml` together with the `CasaMeastro_no_mask.pt` checkpoint.

```bash
python casameastro_vis_no_mask.py \
  --data_root example/scene_00001 \
  --da3_cfg src/depth_anything_3/configs/da3-base.yaml \
  --pose_ckpt /path/to/CasaMeastro_no_mask.pt \
  --out_ply ./output/scene_00001_no_mask.ply \
  --viewpoints all \
  --max_side 448 \
  --device cuda \
  --conf_quantile 0.1
```

If you want to reduce the number of fused points for faster visualization or smaller output files, you can additionally specify `--max_points_per_view`, `--yaw_cut_deg`, or `--height_quantile`:

```bash
python casameastro_vis_no_mask.py \
  --data_root example/scene_00001 \
  --da3_cfg src/depth_anything_3/configs/da3-base.yaml \
  --pose_ckpt /path/to/CasaMeastro_no_mask.pt \
  --out_ply ./output/scene_00001_no_mask_sparse.ply \
  --viewpoints all \
  --max_side 448 \
  --device cuda \
  --max_points_per_view 20000 \
  --yaw_cut_deg 5 \
  --conf_quantile 0.1 \
  --height_quantile 0.5
```

### Run CasaMaestro Panoramic Evaluation

`eval.py` evaluates CasaMaestro directly on panoramic scene data. If you want to use a predefined split, you can pass `scene_list_train_mixed.txt` or another scene list file. If some viewpoints do not contain their own `pano_mask.png`, you can provide a fallback mask through `--global_mask_path`.

```bash
python eval.py \
  --data_root /path/to/realsee_eval_root \
  --scene_list scene_list_train_mixed.txt \
  --cfg src/depth_anything_3/configs/da3-base.yaml \
  --ckpt /path/to/casameastro_checkpoint.pt \
  --global_mask_path /path/to/default/pano_mask.png \
  --views 60 \
  --max_side 448 \
  --num_workers 4 \
  --device cuda
```

If you want to use the script's built-in default `scene_list`, you can omit `--scene_list`.

## 🛠️ Training

The actual training entry point is `src/depth_anything_3/train_pano/trainer.py`. After running `pip install -e .`, the recommended way to launch training from the repository root is:

```bash
python -m depth_anything_3.train_pano.trainer \
  --real-data-root /path/to/realsee_real \
  --syn-data-root /path/to/realsee_syn \
  --scene-list-file scene_list_train_mixed.txt \
  --cfg-path src/depth_anything_3/configs/da3-base.yaml \
  --ckpt-path /path/to/initial_weights.safetensors \
  --output-dir workspace/casameastro_train \
  --epochs 25 \
  --batch-size 1 \
  --num-workers 4 \
  --train-view 40 \
  --max-side 448 \
  --use-mask \
  --global-mask-path /path/to/default/pano_mask.png
```

If you want to enable rotation augmentation for synthetic panoramas, you can additionally use:

```bash
python -m depth_anything_3.train_pano.trainer \
  --real-data-root /path/to/realsee_real \
  --syn-data-root /path/to/realsee_syn \
  --scene-list-file scene_list_train_mixed.txt \
  --cfg-path src/depth_anything_3/configs/da3-base.yaml \
  --ckpt-path /path/to/initial_weights.safetensors \
  --output-dir workspace/casameastro_train_aug \
  --train-view 40 \
  --max-side 448 \
  --use-mask \
  --syn-rot-aug \
  --yaw-min-deg 0 \
  --yaw-max-deg 180 \
  --pitch-std-deg 2 \
  --roll-std-deg 2
```

## 📈 DA3 Evaluation on Panoramas

We also provide code for evaluating the original DA3 model on panoramic images under `DA3-evaluation/realsee3d`. Since DA3 and similar pinhole-based vision foundation models cannot directly take panoramic images as input, the panoramas must first be projected into pinhole views in spherical coordinates before evaluation.

### Single-Scene Evaluation

#### 1. `DA3-evaluation/realsee3d/prepare_pinhole_scene.py`

This script converts one panoramic scene into a pinhole-view dataset. If you want to compute depth metrics later, it is recommended to enable `--process_depth`.

```bash
python DA3-evaluation/realsee3d/prepare_pinhole_scene.py \
  --scene_dir /path/to/realsee_scene/scene_00001 \
  --out_scene_dir ./DA3-evaluation/realsee3d/outputs/pinhole/scene_00001 \
  --resolutions 800x800 \
  --fov 90 \
  --yaws 0,90,180,270 \
  --pitches 0 \
  --process_depth \
  --image_format png
```

#### 2. `DA3-evaluation/realsee3d/eval_cross_view.py`

This script evaluates a single scene using the pinhole data generated in the previous step. `--frames_res` must match the resolution used in `--resolutions` above.

```bash
python DA3-evaluation/realsee3d/eval_cross_view.py \
  --data_root ./DA3-evaluation/realsee3d/outputs/pinhole/scene_00001 \
  --frames_res 800x800 \
  --cfg /path/to/da3_config.json \
  --ckpt /path/to/da3_model.safetensors \
  --num_frames_per_scene 200 \
  --max_side 518 \
  --num_workers 4 \
  --device cuda
```

### Batch Evaluation

#### 1. Convert a panoramic dataset into pinhole views in batch

Script: `DA3-evaluation/realsee3d/batch_pinhole_transfer.py`

The command below uses real scenes as an example. For synthetic data, replace `--data_type real` with `--data_type syn`, and replace `--real_data_root` with `--syn_data_root`.

```bash
python DA3-evaluation/realsee3d/batch_pinhole_transfer.py \
  --scene_list DA3-evaluation/realsee3d/real_world_scene_test_filtered.txt \
  --data_type real \
  --real_data_root /path/to/realsee_real \
  --pinhole_output_root ./DA3-evaluation/realsee3d/outputs/pinhole \
  --resolution 518x518 \
  --fov 90 \
  --yaws 0,90,180,270 \
  --pitches 0 \
  --process_depth \
  --image_format png
```

#### 2. Run batch evaluation on many scenes

Script: `DA3-evaluation/realsee3d/batch_evaluate.py`

The `--resolution` here must be consistent with the resolution used during batch pinhole conversion.

```bash
python DA3-evaluation/realsee3d/batch_evaluate.py \
  --scene_list DA3-evaluation/realsee3d/real_world_scene_test_filtered.txt \
  --data_type real \
  --pinhole_output_root ./DA3-evaluation/realsee3d/outputs/pinhole \
  --eval_output_root ./DA3-evaluation/realsee3d/outputs/eval \
  --model_cfg /path/to/da3_config.json \
  --model_ckpt /path/to/da3_model.safetensors \
  --resolution 518x518 \
  --num_frames_per_scene 500 \
  --max_side 448
```

## 🙏 Acknowledgements

Our work builds upon the following projects:

- [Depth-Anything-3](https://github.com/ByteDance-Seed/Depth-Anything-3)
  - [RealSee3D](https://github.com/realsee-developer/RealSee3D/tree/main)

## 📜 Citation

If you find CasaMaestro useful in your research or projects, please cite our work:

```
@article{casameastro,
  title={CasaMaestro: Multi-View Panoramas for House-Scale 3D Reconstruction},
  author={Yuzhou Ji and Xiaotian Yang and Zhipeng Zhang},
  journal={arXiv preprint arXiv:2606.31086},
  year={2026}
}
```

CasaMaestro

CasaMaestro 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

CasaMaestro

CasaMaestro 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README