CalibrationTokens

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

CalibrationTokens

(★ 9)

No description available.

CalibrationTokens Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Calibration Tokens
### Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens [ICCV 2025]

[🌐 Project Page](https://suchisrit.github.io/calibration-tokens/) | [📄 arXiv](https://arxiv.org/abs/2508.04928) | [📑 Paper (PDF)](https://arxiv.org/pdf/2508.04928)

---

## Overview

We introduce **Calibration Tokens**, a lightweight set of learnable tokens that can be prepended to the patch sequence of a frozen Vision Transformer (ViT) depth model, modulating its attention to handle the radial distortion of fisheye cameras without any fine-tuning of the base network weights.

Given a standard perspective image and its fisheye counterpart (synthetically generated from the same scene), the tokens are trained to align the fisheye depth prediction with depth prediction from the (reference) perspective image. At inference, only a fisheye image and the tokens are needed.

Supported backbone depth models:
- **UniDepth** (default)
- **Depth Anything**
- **MiDaS**

The calibration tokens are injected into the ViT encoder forward pass of each backbone. If you want to apply this to a new model, the relevant files to reference are:

| Backbone | File |
|---|---|
| UniDepth | `external_src/monocular_depth/UniDepth/unidepth/models/backbones/dinov2.py` |
| Depth Anything | `external_src/monocular_depth/Depth_Anything/torchhub/facebookresearch_dinov2_main/vision_transformer.py` |
| MiDaS | `external_src/monocular_depth/MiDaS/midas/backbones/vit.py` |

---

## Environment Setup

Uses **Python 3.8** and a CUDA-capable GPU (tested on CUDA 11.8).

### 1. Create and activate a virtual environment

```bash
virtualenv -p /usr/bin/python3.8 ./caltoken_env
source ./caltoken_env/bin/activate
```

### 2. Pin typing-extensions for Python 3.8 compatibility

`typing-extensions` 4.13+ requires Python 3.9. Pin it first so PyTorch's installer does not upgrade it:

```bash
pip install "typing-extensions<4.13.0"
```

### 3. Install PyTorch with CUDA support

```bash
pip install torch==2.2.0+cu118 torchvision==0.17.0+cu118 torchaudio==2.2.0+cu118 --index-url https://download.pytorch.org/whl/cu118
```

> For CUDA 12.1, replace `cu118` with `cu121` throughout.

### 4. Install xformers (required by UniDepth)

```bash
pip install xformers==0.0.24 --index-url https://download.pytorch.org/whl/cu118
```

### 5. Install remaining dependencies

```bash
pip install -r requirements.txt
```

---

## Pretrained Model Checkpoints

Download the backbone model weights and place them under `model_ckpt/`:

```
model_ckpt/
├── dpt_large_384.pt            # MiDaS ViT-Large
└── depthanything/
    ├── depth_anything_vitl14.pth
    ├── depth_anything_vitb14.pth
    └── depth_anything_vits14.pth
```

**MiDaS** weights (from [isl-org/MiDaS](https://github.com/isl-org/MiDaS)):

| File | Link |
|---|---|
| `dpt_large_384.pt` | https://github.com/isl-org/MiDaS/releases/download/v3/dpt_large_384.pt |

**Depth Anything** weights (from [LiheYoung/Depth-Anything](https://github.com/LiheYoung/Depth-Anything), hosted on HuggingFace):

| File | Size | Link |
|---|---|---|
| `depth_anything_vitl14.pth` | 1.34 GB | https://huggingface.co/spaces/LiheYoung/Depth-Anything/resolve/main/checkpoints/depth_anything_vitl14.pth |
| `depth_anything_vitb14.pth` | 390 MB | https://huggingface.co/spaces/LiheYoung/Depth-Anything/resolve/main/checkpoints/depth_anything_vitb14.pth |
| `depth_anything_vits14.pth` | 99.2 MB | https://huggingface.co/spaces/LiheYoung/Depth-Anything/resolve/main/checkpoints/depth_anything_vits14.pth |

**UniDepth** weights are loaded automatically from HuggingFace Hub on first run.

---

## Dataset Setup

The training scripts expect a text file per dataset with one image path per line, and either a shared camera intrinsics `.npy` file or a per-image intrinsics list. Example usages are shown here, but you must create paths files for whatever data you choose to train on that match however your data is stored.

```
splits/
├── irs_images.txt          # one absolute/relative image path per line
├── waymo_images.txt
└── waymo_intrinsics.txt    # one .npy path per line (per-image intrinsics)

data/irs_derived/K.npy      # shared 3x3 intrinsic matrix for IRS
```

This shows an example usage with the IRS and Waymo datasets, but you will have to create your own splits/paths files.

These example datasets can be found at the following links:

| Dataset | Link |
|---|---|
| **IRS** (Indoor Robotics Stereo) | https://github.com/blackjack2015/IRS (download via [OneDrive](https://1drv.ms/f/s!AmN7U9URpGVGem0coY8PJMHYg0g?e=nvH5oB)) |
| **Waymo Open Dataset** | https://waymo.com/open/ |

---

## Training

### Single GPU

```bash
bash bash/train_unidepth.sh
```

### Multi-GPU (DDP)

```bash
bash bash/ddp_train_unidepth.sh
```

The DDP script launches with `torchrun --nproc_per_node=4` by default. Edit the script to change the number of GPUs, datasets, distortion ranges, or augmentation settings.

The training pipeline will multiply each dataset up to the size of the largest dataset, and repeatedly alternate through datasets at each step. `--n_batches_per_update` must be a multiple of the number of datasets, so each gradient update accumulates an equal number of batches from each dataset.

Key arguments:

| Argument | Description |
|---|---|
| `--model_name` | Backbone model: `unidepth`, `depthanything`, `midas` |
| `--num_tokens` | Number of calibration tokens to train |
| `--output_reference_frame` | Reference frame the tokens are trained to align to: `distance` (default, Euclidean distance R) or `depth` (raw z-depth). See the note below. |
| `--datasets` | Space-separated dataset tags (used for logging) |
| `--image_txts` | One `.txt` image list per dataset |
| `--intrinsics_paths` | One `.npy` (shared) or `.txt` (per-image) intrinsics file per dataset |
| `--crop_shapes` | `H W` pairs per dataset (try to use multiples of the patch size, which can be seen in model.py)  |
| `--k1_ranges` ... `--k4_ranges` | Radial distortion coefficient sampling ranges per dataset |
| `--dataset_weights` | Loss weight per dataset (default: 1.0) |
| `--n_batches_per_update` | Gradient accumulation steps |
| `--learning_rates` | Learning rates matching `--learning_schedule` |
| `--learning_schedule` | Epoch boundaries at which each learning rate applies |
| `--checkpoint_path` | Output directory for checkpoints and logs |
| `--n_step_per_checkpoint` | Save a checkpoint every N gradient update steps |
| `--n_step_per_summary` | Log to TensorBoard every N gradient update steps |
| `--restore_path` | Path to a checkpoint directory to resume from |
| `--enable_ddp` | Enable multi-GPU distributed training |

> **⚠️ Important — output reference frame (`--output_reference_frame`):** By default (`distance`), the reference perspective prediction is converted from z-depth to **Euclidean distance (R) from the camera center** before the alignment loss, so the calibration tokens are trained to produce depth in the **distance (R) reference frame**. If you instead want the tokens to stay in the **raw z-depth reference frame**, set `--output_reference_frame depth`, which turns off this conversion. This choice changes what the trained tokens (and therefore the model at inference) predict, so make sure it matches how you intend to use the outputs.

> **Note on step counting:** The step numbers printed in the log (e.g. `Step= 100 Losses=...`) and used for checkpointing correspond to **gradient update steps** (i.e. one weight update after accumulating `--n_batches_per_update` batches). The progress bar shown during training counts **individual data batches** across all datasets. These two counts will differ whenever `--n_batches_per_update > 1`.

Photometric augmentation arguments (all default to disabled):

| Argument | Description |
|---|---|
| `--aug_prob_brightness/contrast/saturation/hue/blur` | Per-augmentation probability |
| `--aug_brightness/contrast/saturation/hue/blur_range MIN MAX` | Sampling range for each augmentation |

---

## Inference

Run depth estimation on a single fisheye image using a trained calibration token checkpoint:

```bash
bash bash/run_unidepth.sh
```

Or directly:

```bash
python run.py \
  --num_tokens 8 \
  --checkpoint_path /path/to/checkpoint.pth \
  --image_path /path/to/image.png \
  --output_path /path/to/output \
  --model_name unidepth \
  --eval_resize 1120 1680 \
  --device cuda
```

This saves two files to the output folder: a raw depth map as `.npy` and a jet-colorized visualization as `.png`.

| Argument | Description |
|---|---|
| `--num_tokens` | Number of calibration tokens (must match the checkpoint) |
| `--checkpoint_path` | Path to the `.pth` checkpoint file |
| `--image_path` | Path to the input fisheye image |
| `--output_path` | Folder to save outputs |
| `--model_name` | Backbone model: `unidepth`, `depthanything`, `midas` |
| `--eval_resize` | Optional `H W` to resize the image before inference |
| `--device` | Device to run on (e.g. `cuda`, `cuda:0`, `cpu`) |

---

This software is property of Yale University, and is provided free of charge for research purposes only. It comes with no warranties, expressed or implied.

## Citation

```bibtex
@inproceedings{gangopadhyay2025extending,
  title={Extending foundational monocular depth estimators to fisheye cameras with calibration tokens},
  author={Gangopadhyay, Suchisrit and Kim, Jung-Hee and Chen, Xien and Rim, Patrick and Park, Hyoungseob and Wong, Alex},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={5198--5209},
  year={2025}
}
```

CalibrationTokens

CalibrationTokens Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

CalibrationTokens

CalibrationTokens Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README