no-time-to-train

KO | EN

gitlite — search

#llm #golang #machine-learning #deep-learning #linux #macos #nodejs #java #windows #kubernetes #vue #ios #mcp #openai

no-time-to-train

(★ 316)

Official code for "No time to train! Training-Free Reference-Based Instance Segmentation"

파일 탐색기

최종 버전 다운로드 (.zip)

- - __init__.py
- - depth_estimation.ipynb
  - semantic_segmentation.ipynb
- - lint.sh
- .gitignore
- CODE_OF_CONDUCT.md
- conda-extras.yaml
- conda.yaml
- CONTRIBUTING.md
- hubconf.py
- LICENSE
- MODEL_CARD.md
- pyproject.toml
- README.md
- requirements-dev.txt
- requirements-extras.txt
- requirements.txt
- setup.cfg
- setup.py
- cars.jpg
- cars_pca.png
- cars_sam2_amg.png
- cars_squared.jpg
- giraffe.jpg
- giraffe_pca.png
- giraffe_sam2_amg.png
- giraffe_squared.jpg
- - __init__.py
  - local_global_corre.py
  - local_global_oracleIoU.py
  - pcaScore_oracleIoU.py
  - pos_neg_by_class.py
  - scores_by_class.py
  - sim_predIoU_oracleIoU.py
- - __init__.py
  - cd_vito_paper_coco_zeroshot_categories.py
  - change_filename_pascal.py
  - coco_inst_to_segm.py
  - coco_ref_dataset.py
  - coco_to_pkl.py
  - create_jsons_per_class.py
  - data_utils.py
  - download_dataset.py
  - few_shot_sampling.py
  - lvis_add_filename.py
  - lvis_fix_minival_segm.py
  - metainfo.py
  - pascal_voc_to_coco.py
  - sam_bbox_to_segm_batch.py
  - sample_memory_semantic_ref.py
  - sample_sub_dataset.py
  - select_ref_img_heuristic.py
  - visualization.py
- - ablation-coco-std.csv
  - ablation_std.py
  - bbox_positions.py
  - feature_similarity.py
  - grid-images-plot.py
  - heuristics_analysis.ipynb
  - heuristics_analysis.py
  - mask_area_distribution.py
  - paper_fig_square_imgs.py
  - plot_blur_results.py
  - plot_reference_images.py
  - tsne-coco.py
- - __init__.py
  - matcher_utils.py
  - matching_baseline_utils.py
  - matching_foundpose_utils.py
  - model_utils.py
  - Sam2Matcher.py
  - Sam2MatchingBaseline.py
  - Sam2MatchingBaseline_noAMG.py
  - Sam2MatchingBaseline_noAMG_model_and_memory.py
  - SAM2Ref.py
- - coco_allclasses_10shot_Sam2L.yaml
  - coco_fewshot_10shot_Sam2L.yaml
- - coco2014_split1.yaml
  - coco2014_split2.yaml
  - coco2014_split3.yaml
  - coco_allClasses_sam2_t.yaml
  - coco_allClasses_sam2_t_1_1.yaml
  - matcher_test.yaml
  - matching_baseline_10shot.yaml
  - matching_baseline_10shot_allClass.yaml
  - matching_baseline_1shot.yaml
  - matching_baseline_bad.yaml
  - matching_baseline_instance.yaml
  - matching_baseline_instance_fewshot.yaml
  - matching_baseline_noAMG_test.yaml
  - matching_baseline_test.yaml
  - matching_cdfsod_template.yaml
  - matching_lvis_allClass.yaml
  - matching_pascal_voc_few_shot_template.yaml
  - negative_10shot_allClass.yaml
  - negative_10shot_novel.yaml
  - negative_test.yaml
  - semantic_ref_1_1.yaml
  - semantic_ref_1_1_cls.yaml
  - test.yaml
  - test_semantic_ref.yaml
- - sam2matcher_pl.py
  - sam2ref_pl.py
- __init__.py
- dev_log.md
- utils.py
- - cars.jpg
  - groceries.jpg
  - truck.jpg
- automatic_mask_generator_example.ipynb
- image_predictor_example.ipynb
- video_predictor_example.ipynb
- - connected_components.cu
- - __init__.py
  - memory_attention.py
  - memory_encoder.py
  - position_encoding.py
  - sam2_base.py
  - sam2_utils.py
- - __init__.py
  - amg.py
  - misc.py
  - transforms.py
- __init__.py
- automatic_mask_generator.py
- build_sam.py
- sam2_image_predictor.py
- sam2_video_predictor.py
- __init__.py
- sam2_hiera_b+.yaml
- sam2_hiera_l.yaml
- sam2_hiera_s.yaml
- sam2_hiera_t.yaml
- - sav_000001.mp4
  - sav_000001_auto.json
  - sav_000001_manual.json
- - sav_benchmark.py
  - sav_utils.py
- LICENSE
- LICENSE_DAVIS
- LICENSE_VOS_BENCHMARK
- README.md
- requirements.txt
- sav_evaluator.py
- sav_visualization_example.ipynb
- - gpu0.sh
  - gpu1.sh
  - gpu3.sh
  - gpu4.sh
  - gpu5.sh
  - gpu6.sh
  - gpu7.sh
  - setup.sh
  - template.sh
- - dinov2_sam_baseline.sh
- - blur_ablation.sh
- - clipb16.sh
  - clipb32.sh
  - clipl14.sh
  - clipl14@336px.sh
- - convert_fast_to_dior_format.py
  - convert_hrsid_to_dior_format.py
  - convert_isaid_to_dior_format.py
  - convert_mapping_to_dior_format.py
  - convert_nwpu_vhr10_to_dior_format.py
  - convert_rareplanes_to_dior_format.py
  - convert_roof_to_dior_format.py
  - convert_sior_to_dior_format.py
  - convert_soda_a_to_dior_format.py
  - convert_sota_to_dior_format.py
  - convert_ssdd_to_dior_format.py
  - convert_vedai1024_to_dior_format.py
  - convert_vedai512_to_dior_format.py
  - convert_xview_to_dior_format.py
  - summary_table_datasets.py
  - tile_isaid_dataset.py
  - tile_roof_dataset.py
  - tile_sodaa_dataset.py
  - verify_isaid_categories.py
- - dinov3b.sh
  - dinov3h.sh
  - dinov3l.sh
- - EO_claptrap_template.sh
  - EO_template.sh
  - fast.sh
  - hrsid.sh
  - isaid.sh
  - mapping.sh
  - nwpu.sh
  - rareplanes.sh
  - rareplanes_single_class.sh
  - sior.sh
  - sodaa.sh
  - sota.sh
  - ssdd.sh
  - vedai1024.sh
  - vedai512.sh
  - xview.sh
- - dinov2_mobilesam.sh
- - plot_airplane_bird.py
  - plot_airplane_bird.sh
  - plot_dinov2_pca.py
  - plot_EO_accuracy.py
  - plot_EO_grid.py
  - plot_EO_heuristic.py
  - plot_EO_runtime.py
  - plot_sam2_amg.py
  - prepare_figures.py
  - table_EO_results.py
- - PEg14.sh
  - PEl14.sh
- - sam2_base_plus.sh
  - sam2_small.sh
  - sam2_tiny.sh
- - ablation-vl-qwen.sh
- ablation_coco_std.sh
- artaxor_clipart.sh
- batch-points-no-embed.sh
- batch-points-with-embed.sh
- cdfsod_pipeline_only_visualisation.sh
- dior_fish.sh
- few_shot_coco_semantic_pipeline.sh
- launch_cdfsod_visualisation.sh
- launch_cdfsod_visualisation_5_shot.sh
- launch_coco_semantic.sh
- launch_coco_semantic_eval.sh
- launch_pascal_split.sh
- lvis_pipeline.sh
- make_custom_dataset.py
- make_links.sh
- matching_cdfsod_pipeline.sh
- neu-det_uodd.sh
- pascal_split.sh
- pascal_voc_box_to_segmentation.sh
- evaluate_results.py
- run_qwen_coco_detections.py
.gitignore
environment.yml
few_shot_full_pipeline.sh
full_seg.sh
pyproject.toml
README.md
RobotoMono-MediumItalic.ttf
run_lightening.py
setup.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<div align="right">
 <details>
 <summary >🌐 Language</summary>
 <div>
 <div align="center">
 <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=en">English</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=zh-CN">简体中文</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=zh-TW">繁體中文</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=ja">日本語</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=ko">한국어</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=hi">हिन्दी</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=th">ไทย</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=fr">Français</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=de">Deutsch</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=es">Español</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=it">Italiano</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=ru">Русский</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=pt">Português</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=nl">Nederlands</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=pl">Polski</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=ar">العربية</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=fa">فارسی</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=tr">Türkçe</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=vi">Tiếng Việt</a>
 | <a href="https://openaitx.github.io/view.html?user=miquel-espinosa&project=no-time-to-train&lang=id">Bahasa Indonesia</a>
 </div>
 </div>
 </details>
</div>

# 🚀 No Time to Train!  
### Training-Free Reference-Based Instance Segmentation  
[![GitHub](https://img.shields.io/badge/%E2%80%8B-No%20Time%20To%20Train-black?logo=github)](https://github.com/miquel-espinosa/no-time-to-train)
[![Website](https://img.shields.io/badge/🌐-Project%20Page-grey)](https://miquel-espinosa.github.io/no-time-to-train/)
[![arXiv](https://img.shields.io/badge/arXiv-2507.02798-b31b1b)](https://arxiv.org/abs/2507.02798)

**State-of-the-art (Papers with Code)**

[**_SOTA 1-shot_**](https://paperswithcode.com/sota/few-shot-object-detection-on-ms-coco-1-shot?p=no-time-to-train-training-free-reference) | [![PWC](https://img.shields.io/badge/State%20of%20the%20Art-Few--Shot%20Object%20Detection%20on%20MS--COCO%20(1--shot)-21CBCE?style=flat&logo=paperswithcode)](https://paperswithcode.com/sota/few-shot-object-detection-on-ms-coco-1-shot?p=no-time-to-train-training-free-reference)

[**_SOTA 10-shot_**](https://paperswithcode.com/sota/few-shot-object-detection-on-ms-coco-10-shot?p=no-time-to-train-training-free-reference) | [![PWC](https://img.shields.io/badge/State%20of%20the%20Art-Few--Shot%20Object%20Detection%20on%20MS--COCO%20(10--shot)-21CBCE?style=flat&logo=paperswithcode)](https://paperswithcode.com/sota/few-shot-object-detection-on-ms-coco-10-shot?p=no-time-to-train-training-free-reference)

[**_SOTA 30-shot_**](https://paperswithcode.com/sota/few-shot-object-detection-on-ms-coco-30-shot?p=no-time-to-train-training-free-reference) | [![PWC](https://img.shields.io/badge/State%20of%20the%20Art-Few--Shot%20Object%20Detection%20on%20MS--COCO%20(30--shot)-21CBCE?style=flat&logo=paperswithcode)](https://paperswithcode.com/sota/few-shot-object-detection-on-ms-coco-30-shot?p=no-time-to-train-training-free-reference)

<!-- [**_SOTA 1-shot_**](https://paperswithcode.com/sota/few-shot-object-detection-on-ms-coco-1-shot?p=no-time-to-train-training-free-reference) | [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/no-time-to-train-training-free-reference/few-shot-object-detection-on-ms-coco-1-shot)](https://paperswithcode.com/sota/few-shot-object-detection-on-ms-coco-1-shot?p=no-time-to-train-training-free-reference)

[**_SOTA 10-shot_**](https://paperswithcode.com/sota/few-shot-object-detection-on-ms-coco-10-shot?p=no-time-to-train-training-free-reference) | [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/no-time-to-train-training-free-reference/few-shot-object-detection-on-ms-coco-10-shot)](https://paperswithcode.com/sota/few-shot-object-detection-on-ms-coco-10-shot?p=no-time-to-train-training-free-reference)

[**_SOTA 30-shot_**](https://paperswithcode.com/sota/few-shot-object-detection-on-ms-coco-30-shot?p=no-time-to-train-training-free-reference) | [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/no-time-to-train-training-free-reference/few-shot-object-detection-on-ms-coco-30-shot)](https://paperswithcode.com/sota/few-shot-object-detection-on-ms-coco-30-shot?p=no-time-to-train-training-free-reference) -->

</div>

---
> 🚨 **Update (5th February 2026)**: The paper manuscript has been updated with extensive ablation studies, visualisations and additional experiments.
> 
> 🚨 **Update (22nd July 2025):** Instructions for custom datasets have been added!
> 
> 🔔 **Update (16th July 2025):** Code has been updated with instructions!

---

## 📋 Table of Contents

- [🎯 Highlights](#-highlights)
- [📜 Abstract](#-abstract)
- [🧠 Architecture](#-architecture)
- [🛠️ Installation instructions](#️-installation-instructions)
  - [1. Clone the repository](#1-clone-the-repository)
  - [2. Create conda environment](#2-create-conda-environment)
  - [3. Install SAM2 and DINOv2](#3-install-sam2-and-dinov2)
  - [4. Download datasets](#4-download-datasets)
  - [5. Download SAM2 and DINOv2 checkpoints](#5-download-sam2-and-dinov2-checkpoints)
- [📊 Inference code: Reproduce 30-shot SOTA results in Few-shot COCO](#-inference-code)
  - [0. Create reference set](#0-create-reference-set)
  - [1. Fill memory with references](#1-fill-memory-with-references)
  - [2. Post-process memory bank](#2-post-process-memory-bank)
  - [3. Inference on target images](#3-inference-on-target-images)
  - [Results](#results)
- [🔍 Custom dataset](#-custom-dataset)
  - [0. Prepare a custom dataset ⛵🐦](#0-prepare-a-custom-dataset)
  - [0.1 If only bbox annotations are available](#01-if-only-bbox-annotations-are-available)
  - [0.2 Convert coco annotations to pickle file](#02-convert-coco-annotations-to-pickle-file)
  - [1. Fill memory with references](#1-fill-memory-with-references)
  - [2. Post-process memory bank](#2-post-process-memory-bank)
- [📚 Citation](#-citation)

## 🎯 Highlights
- 💡 **Training-Free**: No fine-tuning, no prompt engineering—just a reference image.  
- 🖼️ **Reference-Based**: Segment new objects using just a few examples.  
- 🔥 **SOTA Performance**: Outperforms previous training-free approaches on COCO, PASCAL VOC, and Cross-Domain FSOD.

**Links:**
- 🧾 [**arXiv Paper**](https://arxiv.org/abs/2507.02798)  
- 🌐 [**Project Website**](https://miquel-espinosa.github.io/no-time-to-train/)  
- 📈 [**Papers with Code**](https://paperswithcode.com/paper/no-time-to-train-training-free-reference)

## 📜 Abstract

> The performance of image segmentation models has historically been constrained by the high cost of collecting large-scale annotated data. The Segment Anything Model (SAM) alleviates this original problem through a promptable, semantics-agnostic, segmentation paradigm and yet still requires manual visual-prompts or complex domain-dependent prompt-generation rules to process a new image. Towards reducing this new burden, our work investigates the task of object segmentation when provided with, alternatively, only a small set of reference images. Our key insight is to leverage strong semantic priors, as learned by foundation models, to identify corresponding regions between a reference and a target image. We find that correspondences enable automatic generation of instance-level segmentation masks for downstream tasks and instantiate our ideas via a multi-stage, training-free method incorporating (1) memory bank construction; (2) representation aggregation and (3) semantic-aware feature matching. Our experiments show significant improvements on segmentation metrics, leading to state-of-the-art performance on COCO FSOD (36.8% nAP), PASCAL VOC Few-Shot (71.2% nAP50) and outperforming existing training-free approaches on the Cross-Domain FSOD benchmark (22.4% nAP).

![cdfsod-results-final-comic-sans-min](https://github.com/user-attachments/assets/ab302c02-c080-4042-99fc-0e181ba8abb9)

## 🧠 Architecture

![training-free-architecture-comic-sans-min](https://github.com/user-attachments/assets/d84dd83a-505e-45a0-8ce3-98e1838017f9)

## 🛠️ Installation instructions

### 1. Clone the repository

```bash
git clone https://github.com/miquel-espinosa/no-time-to-train.git
cd no-time-to-train
```

### 2. Create conda environment

We will create a conda environment with the required packages.
```bash
conda env create -f environment.yml
conda activate no-time-to-train
```

### 3. Install SAM2 and DINOv2

We will install SAM2 and DINOv2 from source.
```bash
pip install -e .
cd dinov2
pip install -e .
cd ..
```

### 4. Download datasets

Please download COCO dataset and place it in `data/coco`

### 5. Download SAM2 and DINOv2 checkpoints

We will download the exact SAM2 checkpoints used in the paper.
(Note, however, that SAM2.1 checkpoints are already available and might perform better.)

```bash
mkdir -p checkpoints/dinov2
cd checkpoints
wget https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_large.pt
cd dinov2
wget https://dl.fbaipublicfiles.com/dinov2/dinov2_vitl14/dinov2_vitl14_pretrain.pth
cd ../..
```

## 📊 Inference code

⚠️ Disclaimer: This is research code — expect a bit of chaos!

### Reproducing 30-shot SOTA results in Few-shot COCO

Define useful variables and create a folder for results:

```bash
CONFIG=./no_time_to_train/new_exps/coco_fewshot_10shot_Sam2L.yaml
CLASS_SPLIT="few_shot_classes"
RESULTS_DIR=work_dirs/few_shot_results
SHOTS=30
SEED=33
GPUS=4

mkdir -p $RESULTS_DIR
FILENAME=few_shot_${SHOTS}shot_seed${SEED}.pkl
```

#### 0. Create reference set

```bash
python no_time_to_train/dataset/few_shot_sampling.py \
        --n-shot $SHOTS \
        --out-path ${RESULTS_DIR}/${FILENAME} \
        --seed $SEED \
        --dataset $CLASS_SPLIT
```

#### 1. Fill memory with references

```bash
python run_lightening.py test --config $CONFIG \
                              --model.test_mode fill_memory \
                              --out_path ${RESULTS_DIR}/memory.ckpt \
                              --model.init_args.model_cfg.memory_bank_cfg.length $SHOTS \
                              --model.init_args.dataset_cfgs.fill_memory.memory_pkl ${RESULTS_DIR}/${FILENAME} \
                              --model.init_args.dataset_cfgs.fill_memory.memory_length $SHOTS \
                              --model.init_args.dataset_cfgs.fill_memory.class_split $CLASS_SPLIT \
                              --trainer.logger.save_dir ${RESULTS_DIR}/ \
                              --trainer.devices $GPUS
```

#### 2. Post-process memory bank

```bash
python run_lightening.py test --config $CONFIG \
                              --model.test_mode postprocess_memory \
                              --model.init_args.model_cfg.memory_bank_cfg.length $SHOTS \
                              --ckpt_path ${RESULTS_DIR}/memory.ckpt \
                              --out_path ${RESULTS_DIR}/memory_postprocessed.ckpt \
                              --trainer.devices 1
```

#### 3. Inference on target images

```bash
python run_lightening.py test --config $CONFIG  \
                              --ckpt_path ${RESULTS_DIR}/memory_postprocessed.ckpt \
                              --model.init_args.test_mode test \
                              --model.init_args.model_cfg.memory_bank_cfg.length $SHOTS \
                              --model.init_args.model_cfg.dataset_name $CLASS_SPLIT \
                              --model.init_args.dataset_cfgs.test.class_split $CLASS_SPLIT \
                              --trainer.logger.save_dir ${RESULTS_DIR}/ \
                              --trainer.devices $GPUS
```

If you'd like to see inference results online (as they are computed), add the argument:
```bash
    --model.init_args.model_cfg.test.online_vis True
```
To adjust the score threshold `score_thr` parameter, add the argument (for example, visualising all instances with score higher than `0.4`):
```bash
    --model.init_args.model_cfg.test.vis_thr 0.4
```
Images will now be saved in `results_analysis/few_shot_classes/`. The image on the left shows the ground truth, the image on the right shows the segmented instances found by our training-free method.

Note that in this example we are using the `few_shot_classes` split, thus, we should only expect to see segmented instances of the classes in this split (not all classes in COCO).

#### Results

After running all images in the validation set, you should obtain:

```
BBOX RESULTS:
  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.368

SEGM RESULTS:
  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.342
```
---

## 🔍 Custom dataset

We provide the instructions for running our pipeline on a custom dataset. Annotation format are always in COCO format.

> **TLDR;** To directly see how to run full pipeline on *custom datasets*, find `scripts/matching_cdfsod_pipeline.sh` together with example scripts of CD-FSOD datasets (e.g. `scripts/dior_fish.sh`)

### 0. Prepare a custom dataset ⛵🐦

Let's imagine we want to detect **boats**⛵ and **birds**🐦 in a custom dataset. To use our method we will need:
- At least 1 *annotated* reference image for each class (i.e. 1 reference image for boat and 1 reference image for bird)
- Multiple target images to find instances of our desired classes.

We have prepared a toy script to create a custom dataset with coco images, for a **1-shot** setting.
```bash
mkdir -p data/my_custom_dataset
python scripts/make_custom_dataset.py
```
This will create a custom dataset with the following folder structure:
```
data/my_custom_dataset/
    ├── annotations/
    │   ├── custom_references.json
    │   ├── custom_targets.json
    │   └── references_visualisations/
    │       ├── bird_1.jpg
    │       └── boat_1.jpg
    └── images/
        ├── 429819.jpg
        ├── 101435.jpg
        └── (all target and reference images)
```

**Reference images visualisation (1-shot):**

| 1-shot Reference Image for BIRD 🐦 | 1-shot Reference Image for BOAT ⛵ |
|:---------------------------------:|:----------------------------------:|
| <img src="https://github.com/user-attachments/assets/e59e580d-a7db-42ac-b386-892af211fc85" alt="bird_1" width="500"/> | <img src="https://github.com/user-attachments/assets/f94ee025-ae37-4a45-9c3e-0cfe8f8cd2bc" alt="boat_1" width="500"/> |

### 0.1 If only bbox annotations are available

We also provide a script to generate instance-level segmentation masks by using SAM2. This is useful if you only have bounding box annotations available for the reference images.

```bash
# Download sam_h checkpoint. Feel free to use more recent checkpoints (note: code might need to be adapted)
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth -O checkpoints/sam_vit_h_4b8939.pth
# Run automatic instance segmentation from ground truth bounding boxes.
python no_time_to_train/dataset/sam_bbox_to_segm_batch.py \
    --input_json data/my_custom_dataset/annotations/custom_references.json \
    --image_dir data/my_custom_dataset/images \
    --sam_checkpoint checkpoints/sam_vit_h_4b8939.pth \
    --model_type vit_h \
    --device cuda \
    --batch_size 8 \
    --visualize
```

**Reference images with instance-level segmentation masks (generated by SAM2 from gt bounding boxes, 1-shot):**

Visualisation of the generated segmentation masks are saved in `data/my_custom_dataset/annotations/custom_references_with_SAM_segm/references_visualisations/`.

| 1-shot Reference Image for BIRD 🐦 (automatically segmented with SAM) | 1-shot Reference Image for BOAT ⛵ (automatically segmented with SAM) |
|:---------------------------------:|:----------------------------------:|
| <img src="https://github.com/user-attachments/assets/65d38dc4-1454-43cd-9600-e8efc67b3a82" alt="bird_1_with_SAM_segm" width="500"/> | <img src="https://github.com/user-attachments/assets/43a558ad-50ca-4715-8285-9aa3268843c6" alt="boat_1_with_SAM_segm" width="500"/> |

### 0.2 Convert coco annotations to pickle file

```bash
python no_time_to_train/dataset/coco_to_pkl.py \
    data/my_custom_dataset/annotations/custom_references_with_segm.json \
    data/my_custom_dataset/annotations/custom_references_with_segm.pkl \
    1
```

### 1. Fill memory with references

First, define useful variables and create a folder for results. For correct visualisation of labels, class names should be ordered by category id as appears in the json file. E.g. `bird` has category id `16`, `boat` has category id `9`. Thus, `CAT_NAMES=boat,bird`.

```bash
DATASET_NAME=my_custom_dataset
DATASET_PATH=data/my_custom_dataset
CAT_NAMES=boat,bird
CATEGORY_NUM=2
SHOT=1
YAML_PATH=no_time_to_train/pl_configs/matching_cdfsod_template.yaml
PATH_TO_SAVE_CKPTS=./tmp_ckpts/my_custom_dataset
mkdir -p $PATH_TO_SAVE_CKPTS
```

Run step 1:
```bash
python run_lightening.py test --config $YAML_PATH \
    --model.test_mode fill_memory \
    --out_path $PATH_TO_SAVE_CKPTS/$DATASET_NAME\_$SHOT\_refs_memory.pth \
    --model.init_args.dataset_cfgs.fill_memory.root $DATASET_PATH/images \
    --model.init_args.dataset_cfgs.fill_memory.json_file $DATASET_PATH/annotations/custom_references_with_segm.json \
    --model.init_args.dataset_cfgs.fill_memory.memory_pkl $DATASET_PATH/annotations/custom_references_with_segm.pkl \
    --model.init_args.dataset_cfgs.fill_memory.memory_length $SHOT \
    --model.init_args.dataset_cfgs.fill_memory.cat_names $CAT_NAMES \
    --model.init_args.model_cfg.dataset_name $DATASET_NAME \
    --model.init_args.model_cfg.memory_bank_cfg.length $SHOT \
    --model.init_args.model_cfg.memory_bank_cfg.category_num $CATEGORY_NUM \
    --trainer.devices 1
```

### 2. Post-process memory bank

```bash
python run_lightening.py test --config $YAML_PATH \
    --model.test_mode postprocess_memory \
    --ckpt_path $PATH_TO_SAVE_CKPTS/$DATASET_NAME\_$SHOT\_refs_memory.pth \
    --out_path $PATH_TO_SAVE_CKPTS/$DATASET_NAME\_$SHOT\_refs_memory_postprocessed.pth \
    --model.init_args.model_cfg.dataset_name $DATASET_NAME \
    --model.init_args.model_cfg.memory_bank_cfg.length $SHOT \
    --model.init_args.model_cfg.memory_bank_cfg.category_num $CATEGORY_NUM \
    --trainer.devices 1
```

#### 2.1 Visualise post-processed memory bank

```bash
python run_lightening.py test --config $YAML_PATH \
    --model.test_mode vis_memory \
    --ckpt_path $PATH_TO_SAVE_CKPTS/$DATASET_NAME\_$SHOT\_refs_memory_postprocessed.pth \
    --model.init_args.dataset_cfgs.fill_memory.root $DATASET_PATH/images \
    --model.init_args.dataset_cfgs.fill_memory.json_file $DATASET_PATH/annotations/custom_references_with_segm.json \
    --model.init_args.dataset_cfgs.fill_memory.memory_pkl $DATASET_PATH/annotations/custom_references_with_segm.pkl \
    --model.init_args.dataset_cfgs.fill_memory.memory_length $SHOT \
    --model.init_args.dataset_cfgs.fill_memory.cat_names $CAT_NAMES \
    --model.init_args.model_cfg.dataset_name $DATASET_NAME \
    --model.init_args.model_cfg.memory_bank_cfg.length $SHOT \
    --model.init_args.model_cfg.memory_bank_cfg.category_num $CATEGORY_NUM \
    --trainer.devices 1
```
PCA and K-means visualisations for the memory bank images are stored in `results_analysis/memory_vis/my_custom_dataset`.

### 3. Inference on target images

If `ONLINE_VIS` is set to True, prediction results will be saved in `results_analysis/my_custom_dataset/` and displayed as they are computed. NOTE that running with online visualisation is much slower.

Feel free to change the score threshold `VIS_THR` to see more or less segmented instances.
```bash
ONLINE_VIS=True
VIS_THR=0.4
python run_lightening.py test --config $YAML_PATH \
    --model.test_mode test \
    --ckpt_path $PATH_TO_SAVE_CKPTS/$DATASET_NAME\_$SHOT\_refs_memory_postprocessed.pth \
    --model.init_args.model_cfg.dataset_name $DATASET_NAME \
    --model.init_args.model_cfg.memory_bank_cfg.length $SHOT \
    --model.init_args.model_cfg.memory_bank_cfg.category_num $CATEGORY_NUM \
    --model.init_args.model_cfg.test.imgs_path $DATASET_PATH/images \
    --model.init_args.model_cfg.test.online_vis $ONLINE_VIS \
    --model.init_args.model_cfg.test.vis_thr $VIS_THR \
    --model.init_args.dataset_cfgs.test.root $DATASET_PATH/images \
    --model.init_args.dataset_cfgs.test.json_file $DATASET_PATH/annotations/custom_targets.json \
    --model.init_args.dataset_cfgs.test.cat_names $CAT_NAMES \
    --trainer.devices 1
```

### Results

Performance metrics (with the exact same parameters as commands above) should be:

```
BBOX RESULTS:
  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.478

SEGM RESULTS:
  Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.458
```

Visual results are saved in `results_analysis/my_custom_dataset/`. Note that our method works for false negatives, that is, images that do not contain any instances of the desired classes.

*Click images to enlarge ⬇️*

| Target image with boats ⛵ (left GT, right predictions) | Target image with birds 🐦 (left GT, right predictions) |
|:----------------------:|:----------------------:|
| ![000000459673](https://github.com/user-attachments/assets/678dc15a-dd3b-49d5-9287-6290da16aa6b) | ![000000407180](https://github.com/user-attachments/assets/fe306e48-af49-4d83-ac82-76fac6c456d1) |

| Target image with boats and birds ⛵🐦 (left GT, right predictions) | Target image without boats or birds 🚫 (left GT, right predictions) |
|:---------------------------------:|:----------------------------------:|
| ![000000517410](https://github.com/user-attachments/assets/9849b227-7f43-43d7-81ea-58010a623ad5) | ![000000460598](https://github.com/user-attachments/assets/7587700c-e09d-4cf6-8590-3df129c2568e) |

## 🔬 Ablations

### Backbone ablation

To evaluate the transferability of our method across foundation models, we replace both the semantic
encoder (DINOv2) and the SAM-based segmenter with several alternatives.

**Semantic encoder ablation:**

```bash
# CLIP (Sizes: b16, b32, l14, l14@336px)
bash scripts/clip/clipl14@336px.sh
bash scripts/clip/clipl14.sh
bash scripts/clip/clipb16.sh
bash scripts/clip/clipb32.sh

# DINOV3 (Sizes: b, l, h)
bash scripts/dinov3/dinov3b.sh
bash scripts/dinov3/dinov3l.sh
bash scripts/dinov3/dinov3h.sh

# PE (Sizes: g14, l14)
bash scripts/pe/PEg14.sh
bash scripts/pe/PEl14.sh
```

**Segmenter ablation:**

```bash
# SAM2 (Sizes: tiny, small, base+, large)
bash scripts/sam2/sam2_tiny.sh
bash scripts/sam2/sam2_small.sh
bash scripts/sam2/sam2_base_plus.sh
bash scripts/baseline/dinov2_sam_baseline.sh # SAM2 Large
```

### VLM evaluation on COCO few-shot dataset

We evaluate QWEN VLM on COCO few-shot dataset.

```bash
bash scripts/vl-qwen/ablation-vl-qwen.sh
```

### Reference image heuristics

To understand why different reference images lead to performance variation, we analyse the statistical properties of COCO novel classes annotations.

#### ANALYSIS

We study three annotation characteristics: (1) mask area (object size),
(2) mask center location, and (3) distance to image edges.

<details>
<summary>Instructions:</summary>

```bash
# Mask area distribution
python no_time_to_train/make_plots/mask_area_distribution.py \
  --input data/coco/annotations/instances_val2017.json \
  --output no_time_to_train/make_plots/mask_area_distribution/mask_area_distribution.png \
  --edges-output no_time_to_train/make_plots/mask_area_distribution/bbox_edge_distance_histograms.png \
  --center-output no_time_to_train/make_plots/mask_area_distribution/bbox_center_density.png \
  --bins 80 \
  --distance-bins 80 \
  --disable-center-density

# Bbox center positions
python no_time_to_train/make_plots/bbox_positions.py \
	--per-class-root data/coco/annotations/per_class_instances \
	--filename centeredness_2d_hist_plain.png \
	--max-cols 6 \
	--output-dir ./no_time_to_train/make_plots/bbox_positions \
	--outfile grid_bbox_positions.png
```

</details>

<details>
<summary>[OUTPUT] Mask area distribution</summary>
<img width="600" height="600" alt="mask_area_distribution" src="https://github.com/user-attachments/assets/ece21119-3622-4a2f-8319-1d52ff05bf99" />

</details>

<details>
<summary>[OUTPUT] Bbox center density</summary>
<img width="3165" height="1627" alt="grid_bbox_positions" src="https://github.com/user-attachments/assets/dff4ddb2-a3f1-45e1-af12-8e9fffbb4d6c" />

</details>

<details>
<summary>[OUTPUT] Bbox edge distance histograms</summary>
<img width="1800" height="1200" alt="bbox_edge_distance_histograms" src="https://github.com/user-attachments/assets/e23d1360-599c-46a2-af59-3d071112e76e" />

</details>

#### SELECTION

We sample 100 diverse reference images per class, explicitly
covering a range of mask sizes, centers, and edge distances. Each
reference is evaluated on a fixed reduced validation subset.

<details>
<summary>Instructions:</summary>

**Setup script:** `scripts/1shot_ref_ablation/setup.sh`:
1. Create per class json file
2. Analyse specific class
3. Create reference set with different heuristics

```bash
bash scripts/1shot_ref_ablation/setup.sh
```

**Run scripts:** `scripts/1shot_ref_ablation/gpu*.sh`:

4. Run pipeline for each reference set
```bash
# Example launch script that calls template script for each reference set
bash scripts/1shot_ref_ablation/gpu0.sh
```

</details>

#### RESULTS

We analyze how detection scores correlate with reference image characteristics
(mask size, center position, edge distance).

<details>
<summary>Instructions:</summary>

```bash
python no_time_to_train/make_plots/heuristics_analysis.py
# Outputs: 
# - no_time_to_train/make_plots/heuristics_analysis/heatmap_bbox_norm_scores.png
# - no_time_to_train/make_plots/heuristics_analysis/heatmap_segm_norm_scores.png
# - no_time_to_train/make_plots/heuristics_analysis/heatmap_center_bbox_norm_scores_kde_smooth.png
# - no_time_to_train/make_plots/heuristics_analysis/heatmap_center_bbox_norm_scores.png
# - no_time_to_train/make_plots/heuristics_analysis/heatmap_center_segm_norm_scores_kde_smooth.png
# - no_time_to_train/make_plots/heuristics_analysis/heatmap_center_segm_norm_scores.png
# - no_time_to_train/make_plots/heuristics_analysis/per_class_area_vs_raw_scores.png
# - no_time_to_train/make_plots/heuristics_analysis/all_classes_area_vs_norm_scores.png
# - no_time_to_train/make_plots/heuristics_analysis/edge_distance_vs_norm_scores.png
# - no_time_to_train/make_plots/heuristics_analysis/bars_area_category_norm_scores.png
# - no_time_to_train/make_plots/heuristics_analysis/bars_centered_norm_scores.png
# - no_time_to_train/make_plots/heuristics_analysis/bars_avoid_sides_norm_scores.png
```

</details>

<details>
<summary>[OUTPUT] Barplots. Effect of mask area (left) and centeredness (right) on performance</summary>
<img width="1190" height="846" alt="barplot" src="https://github.com/user-attachments/assets/e900aff5-523d-4563-aebc-0135dcbb5eb6" />

</details>

<details>
<summary>[OUTPUT] Heatmaps. 2D score maps of performance as a function of mask-center location</summary>
<img width="1250" height="545" alt="heatmap" src="https://github.com/user-attachments/assets/c2c59ffe-b19e-4907-b0be-68249cf5db51" />

</details>

<details>
<summary>[OUTPUT] Reference-image performance vs. mask area for all COCO novel classes</summary>
<img width="2500" height="1432" alt="class_performance" src="https://github.com/user-attachments/assets/05a0e213-3ba5-4b4f-80ed-9b7ca782642a" />

</details>

### Reference-image degradation

We evaluate our method under progressively degraded reference images by applying increasing
levels of Gaussian blur.
<img width="2640" height="1194" alt="ablation-blur" src="https://github.com/user-attachments/assets/c2abf0ab-1578-41cf-abcf-50e43f7691f5" />

<details>
<summary>Instructions:</summary>

```bash
# Run different blur levels
bash scripts/blur_ablation/blur_ablation.sh

# Plot grid of blur ablation results
python no_time_to_train/make_plots/plot_blur_results.py \
    --results-root ./work_dirs/blur_ablation \
    --class-id 0 \
    --max-cols 4 \
    --output-dir ./no_time_to_train/make_plots/blur_ablation \
    --outfile grid_blur_ablation_class_0.png
```

</details>

### Feature similarity

Script for visualising feature similarity between reference images and target images.

It generates single-feature similarity (path features), and prototype-based similarity (aggregated features).
<img width="2500" height="1030" alt="feature_similarity_small" src="https://github.com/user-attachments/assets/d56ec9aa-c60e-4fe6-92cd-aa6270b1d6ed" />

<details>
<summary>Instructions:</summary>

```bash
python no_time_to_train/make_plots/feature_similarity.py \
 --classes orange \ 
 --num-images 20 \
 --min-area 12 \
 --max-area 25000 \
 --min-instances 2 \
 --seed 123 \
 --max-per-class 12
```
</details>

### T-SNE plots (DINOv2 feature separability)

t-SNE of DINOv2 features shows clear separation for dissimilar classes
but heavy overlap for similar ones, suggesting that confusion stems from
backbone feature geometry rather than prototypes selection.
<img width="2500" height="1444" alt="tsne" src="https://github.com/user-attachments/assets/baffc430-1600-44a1-9a14-1b08e25a9d55" />

<details>
<summary>Instructions:</summary>

Extract features

```bash
python no_time_to_train/make_plots/tsne-coco.py --extract
```

Plot T-SNE plots

```bash
# Example spoon vs fork
python no_time_to_train/make_plots/tsne-coco.py --classes cat dog
```

</details>

## 🛠️ Helpers

### Visualise memory

add image feature_comparison_small.png here

<details>
<summary>Instructions</summary>

To visualise the memory bank (PCA and K-means visualisations) for a given experiment, adjust the following command.

Set `DO_NOT_CROP` to True/False (in `no_time_to_train/models/Sam2MatchingBaseline_noAMG.py`) to visualise the reference image with/without the cropped mask.

```bash
python run_lightening.py test --config $CONFIG \
 --model.test_mode vis_memory \
 --ckpt_path $RESULTS_DIR/memory_postprocessed.ckpt \
 --model.init_args.dataset_cfgs.fill_memory.memory_pkl $RESULTS_DIR/$FILENAME \
 --model.init_args.dataset_cfgs.fill_memory.memory_length $SHOT \
 --model.init_args.dataset_cfgs.fill_memory.class_split $CLASS_SPLIT \
 --model.init_args.model_cfg.dataset_name $CLASS_SPLIT \
 --model.init_args.model_cfg.memory_bank_cfg.length $SHOT \
 --model.init_args.model_cfg.memory_bank_cfg.category_num $CATEGORY_NUM \
 --trainer.devices 1
```
</details>

### Resize images to 512x512 (make the images square)

To resize the images to 512x512 and save them to a new directory, run the following command. This is for the paper figures.

<details>
<summary>Instructions:</summary>

```bash
python no_time_to_train/make_plots/paper_fig_square_imgs.py
```
</details>

### Model size and memory

To calculate the model size and memory, run the following command.

<details>
<summary>Instructions:</summary>

- See `no_time_to_train/models/Sam2MatchingBaseline_noAMG_model_and_memory.py` for the model size and memory calculation.
(Easiest: temporarily replace by Sam2MatchingBaseline_noAMG.py, then rename back.)
</details>

## 🌍 EO datasets

### Evaluation scripts (EO datasets)

Evaluation scripts can be found in the `scripts/EO` directory. The EO datasets use the `./scripts/EO/EO_template.sh` script to run the evaluation.

Each EO experiment run is saved under the `./EO_results` directory. In the experiment folder we store:
- The summary.txt file with the configuration, and runtime of the experiment.
- The prediction visualisations on the test set (`results_analysis` folder).
- The memory visualisations (`memory_vis` folder).
- The few-shot annotation pickle file.
- The checkpoints of the model (if not cleaned up).

### Figures and tables
Additional scripts for generating figures and tables.

<details>
<summary>Summary latex table of the EO datasets:</summary>

```bash
python scripts/convert_datasets/summary_table_datasets.py
```

</details>

<details>
<summary>Generate LaTeX table of the EO datasets:</summary>

```bash
python scripts/paper_figures/table_EO_results.py ./EO_results_no_heuristics
```

</details>

<details>
<summary>Accuracy plot of the EO datasets:</summary>

```bash
python scripts/paper_figures/plot_EO_accuracy.py \
  --input-root ./EO_results \
  --output-root ./EO_results
```

</details>

<details>
<summary>Summary of heuristics effect on the EO datasets:</summary>
 
```bash
python scripts/paper_figures/plot_EO_heuristic.py \
 --no-heuristics ./EO_results_no_heuristics \
 --heuristics ./EO_results
```

</details>

<details>
<summary>Runtime plot of the EO datasets:</summary>

```bash
python scripts/paper_figures/plot_EO_runtime.py \
  --input-root ./EO_results \
  --output-root ./EO_results
```

</details>

<details>
<summary>Generate EO grid visualisations for paper figure:</summary>

```bash
python scripts/paper_figures/plot_EO_grid.py \
  --root ./EO_results_no_heuristics \
  --dataset ISAID \
  --shots 1
```

</details>

<details>
<summary>DINOv2 PCA feature visualisation:</summary>

```bash
python scripts/paper_figures/plot_dinov2_pca.py --image example_images/giraffe.jpg
python scripts/paper_figures/plot_dinov2_pca.py --image example_images/cars.jpg
```

</details>

<details>
<summary>SAM2 AMG feature visualisation:</summary>

```bash
python scripts/paper_figures/plot_sam2_amg.py --image example_images/giraffe.jpg
python scripts/paper_figures/plot_sam2_amg.py --image example_images/cars.jpg
```

</details>

## ⭐ Star History

## 📚 Citation

If you use this work, please cite us:

```bibtex
@article{espinosa2025notimetotrain,
  title={No time to train! Training-Free Reference-Based Instance Segmentation},
  author={Miguel Espinosa and Chenhongyi Yang and Linus Ericsson and Steven McDonagh and Elliot J. Crowley},
  journal={arXiv preprint arXiv:2507.02798},
  year={2025},
  primaryclass={cs.CV}
}
```

no-time-to-train

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

no-time-to-train

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README