conformal-od

KO | EN

gitlite — search

#python #ai #llm #golang #go #android #machine-learning #docker #windows #kubernetes #awesome-list #mcp #ios #openai

(★ 13)

Code repository for the paper "Adaptive Bounding Box Uncertainties via Two-Step Conformal Prediction" @ ECCV 2024 (Oral)

conformal-od Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Overview

This is the public code repository for our work
[Adaptive Bounding Box Uncertainties via Two-Step Conformal Prediction](https://arxiv.org/abs/2403.07263) presented as an oral paper at ECCV 2024.

#### Abstract :memo:
---

Quantifying a model's predictive uncertainty is essential for safety-critical applications such as autonomous driving. We consider quantifying such uncertainty for multi-object detection. In particular, we leverage conformal prediction to obtain uncertainty intervals with guaranteed coverage for object bounding boxes. One challenge in doing so is that bounding box predictions are conditioned on the object's class label. Thus, we develop a novel two-step conformal approach that propagates uncertainty in predicted class labels into the uncertainty intervals of bounding boxes. This broadens the validity of our conformal coverage guarantees to include incorrectly classified objects, thus offering more actionable safety assurances. Moreover, we investigate novel ensemble and quantile regression formulations to ensure the bounding box intervals are adaptive to object size, leading to a more balanced coverage. Validating our two-step approach on real-world datasets for 2D bounding box localization, we find that desired coverage levels are satisfied with practically tight predictive uncertainty intervals.

---

#### Citation
If you find this repository useful, please consider citing our work:

```
@inproceedings{timans2024conformalod,
    title = {Adaptive Bounding Box Uncertainties via Two-Step Conformal Prediction}, 
    author = {Alexander Timans and Christoph-Nikolas Straehle and Kaspar Sakmann and Eric Nalisnick},
    booktitle = {Proceedings of the European Conference on Computer Vision},
    year = {2024}
}
```

#### Acknowledgements
The [Robert Bosch GmbH](https://www.bosch.com) is acknowledged for financial support.

## Repo structure
The folder structure of the repo is quite self-explanatory, but below are some comments on each folder or file on the main repository level. The most important folder is ```control```.
```
conformal-od/
├── calibration: Conformal nonconformity scores, quantile selection and multiple testing correction, prediction interval construction, random data splitting.
├── config: YAML files with config params per dataset and conformal method, incl. baselines. A description of the config file setup is given in cfg_description.yaml.
├── control: Individual files to run for every conformal method and baseline (both conformal and object detector). These scripts will produce the final output files with performance metrics and results.
├── data: Files data_loader.py for custom dataset loading in detectron2, data_collector.py for custom pre-collection of prediction information, cityscapes.py for custom dataset transforms.
├── detectron2: A local copy of the detectron2 library which is tightly integrated in the data and model prediction aspects of the code.
├── evaluation: Metrics definitions, custom AP evaluation script, final results table generation, notebook with ablation study.
├── gaussian_yolo: Minimal local integration of relevant files to run the GaussianYOLO and YOLOv3 baselines.
├── model: Model- and prediction-related functions such as model definitions that have been slightly modified from the original detectron2 code, model loading, matching predictions to ground truths, and custom quantile regression head training.
├── plots: Plotting utilities and the notebook plots.ipynb to reproduce final plots contained in the paper.
├── util: read/write and other utilities, soume required for the DETR baseline only.
├── commands.py: Script to create a file with python run commands to start a suite of experiments.
├── commands_abl.py: Script to create a file with python run commands to ablate for different coverage levels.
├── env_conf1m.yml: The package environment file as used by myself, with exact package versions.
├── env_minimal.yml: A (heuristic) minimal environment file that contains the most important packages to run the repo.
├── main.py: The main script to run experiments, with CLI args that override any YAML configs.
└── run.sh: A simple shell script that executes the run commands generated by commands.py in series, and logs errors.
```

To run the code, guidelines are provided below.

## Preparing the local setup
1. Clone repo, e.g. using git
```
git clone https://github.com/alextimans/conformal-od
```
- **Note:** Make sure to be in the parent directory of ```conformal-od``` as working directory. Let's call it ```run-code```, so set working directory to ```run-code```. This directory is also the one from which to launch code runs, which is why the provided sample runs below follow the scheme ```python conformal-od/script.py -args```.

2. Set-up python env (e.g. with conda package manager)
```
conda env create -f conformal-od/env_conf1m.yml
conda activate conf1m
```
- Either generate python env from ```env_minimal.yml``` and add potentially missing packages, or use ```env_conf1m.yml``` for a more similar but memory-heavy environment.
- Make sure that the PYTHONPATH environment variable also points to the ```conformal-od``` repo to detect all python modules.
- Code is written to also be compatible with ```cuda``` and GPU support.
- **Note**: We have tried to outsource any modifications made to original detectron2 files into separate files (e.g., ```model/fast_rcnn.py```) for ease of use. However, a few minimal changes have been left directly in our local copy of detectron2: (1) one changed import statement in ```detectron2/modeling/roi_heads/roi_heads.py```; (2) one changed import statement in ```detectron2/modeling/meta_arch/rcnn.py```; (3) one line comment in ```detectron2/data/dataset_mapper.py``` to retain instance annotations. Should a fresh install of ```detectron2``` be used, these changes need to be manually integrated to point correctly.

3. Get and prepare the data

- Download COCO, Cityscapes and BDD100k datasets from their respective websites, which may require requesting access and log-in.
- Keep the data in their original folder structures, except for Cityscapes: since we never train on it, we move all the data (which is structured by city) to the ```train``` folder and leave ```test``` and ```val``` folders empty. This permits to use all available data for calibration/test splitting.
- You should now have a data folder accessible from the working directory, e.g., ```run-code/conformal-od/data/...``` directly, or ```run-code/data/...``` and access from within ```conformal-od``` with a symbolic link. Its structure should look something like

```
data
└── bdd100k
    └── images/100k
        └── test
        └── train
        └── val
    └── labels
└── cityscapes
    └── gtFine
        └── test (empty)
        └── train (all files)
        └── val (empty)
    └── leftImg8bit
        └── test (empty)
        └── train (all files)
        └── val (empty)
└── coco
    └── annotations
    └── labels/val2017
    └── val2017
```

4. Download necessary model checkpoints

- For Box-CQR, the Faster R-CNN model with trained quantile regression head can be downloaded from [this Google Drive](https://drive.google.com/drive/folders/1zzv8Q38GvwRKfDgSEgE_hrNGQ7fZCks5?usp=sharing) (```x101fpn_train_qr_5k_postprocess```). The link also provides the model checkpoints for some of the original pre-trained object detectors (Faster R-CNN, (Gaussian)YOLO, Sparse R-CNN) for good measure.
- The pretrained models should in principle all be obtainable from the respective original repositories: [Faster R-CNN](https://github.com/facebookresearch/detectron2), [YOLOv3](https://github.com/motokimura/PyTorch_Gaussian_YOLOv3), [DETR](https://github.com/facebookresearch/detr), [Sparse R-CNN](https://github.com/PeizeSun/SparseR-CNN/).
- Place them in a folder accessible by the working directory, e.g., ```run-code/checkpoints```.

## Example code runs

- **Note**: A name-change in the class label sets means that the 'Naive' method from the paper is referred to as ```oracle``` in the code. This should not be confused with the 'Oracle' from the paper, which translates to not using any label set method (i.e., the obtained results from only applying our box interval methods) and thus is not an explicit label set option.
- **Note**: The config and script files include path directory names, which will most likely initially throw errors due to non-matching paths. Modify/remove as necessary to obtain the correct directory pointers (see also ```config/cfg_description.yaml```).

- To check the main script CLI arguments that can be given and their default values and short explanations:
```
python conformal-od/main.py -h
```

- To run a conformal bounding box method with a conformal label set method for a dataset:
  - Choose ```risk_control``` from ```["std_conf", "ens_conf", "cqr_conf", "base_conf"]```
  - Choose ```label_set``` from ```["top_singleton", "full", "oracle", "class_threshold"]```
  - Set ```config_path``` thus deciding on dataset choice
  - Default object detector: Faster R-CNN X101-FPN
```
python conformal-od/main.py --config_file=cfg_std_rank --config_path=conformal-od/config/coco_val --run_collect_pred --save_file_pred --risk_control=std_conf --alpha=0.1 --label_set=class_threshold --label_alpha=0.01 --run_risk_control --save_file_control --save_label_set --run_eval --save_file_eval --file_name_suffix=_std_rank_class --device=cuda
```

- To run the full suite of experiments (all combinations & conformal baselines):
```
python conformal-od/commands.py  # generate commands.txt
sh conformal-od/run.sh  # read and run all combos
```

- To run the model baselines (DeepEns, GaussianYOLO, YOLOv3, DETR, Sparse-RCNN):
    - Follow the instructions in their respective script (found in ```control/baseline_*.py```)
    - These scripts are runnables since these methods require additional customization not easily amenable with ```main.py```.
```
python conformal-od/control/baseline_gaussian_yolo.py --config_file=cfg_base_gaussian_yolo --config_path=conformal-od/config/coco_val --run_collect_pred --save_file_pred --risk_control=gaussian_yolo --alpha=0.1 --run_risk_control --save_file_control --run_eval --save_file_eval --file_name_suffix=_base_gaussian_yolo --device=cuda
```

- To generate final aggregated results tables as ```.csv``` files, see the interactive ```evaluation/results_tables.ipynb``` notebook.
- To generate final bounding box figures that can be saved to file, see the interactive ```plots/plots.ipynb``` notebook.

## Interpreting the output

Consider a standard command is executed that runs and saves all intermediate outputs, for example

```
python conformal-od/main.py --config_file=cfg_std_rank --config_path=conformal-od/config/coco_val --run_collect_pred --save_file_pred --risk_control=std_conf --alpha=0.1 --label_set=class_threshold --label_alpha=0.01 --run_risk_control --save_file_control --save_label_set --run_eval --save_file_eval --file_name_suffix=_std_rank_class --device=cuda
```

The executable will create an associated output folder which logs all the results, and which is located in the output directory specified in the config file. In this case, the outputs will be located in ```.../output/std_conf_x101fpn_std_rank_class```.

The folder will contain a (sub)set of the possible files listed below. Most important are the ```.csv``` files which contain readable, aggregate-level results over evaluation metrics.

- ```log.txt```: The logging file capturing standard print output. This is quite verbose and nice to track if all the settings and behaviour is as expected.
- ```*_ap_info.json, *_ap_scores.json```: Outputs from the AP metric evaluations on predictive model performance (if the option is on).
- ```*_ap_table.csv```: The file with readable AP evaluation results.
- ```*_img_list.json```: An index list tracking which object classes appear in which images (see ```data/data_collector.py```)
- ```*_ist_list.json```: A dictionary list where each dictionary collects all the object-level information for objects of a particular class (see ```data/data_collector.py```)
- ```*_test_idx.pt```: A tensor tracking which images (and associated objects) are assigned to calibration/test across trials.
- ```*_control.pt```: A tensor storing all the evaluation metrics for the conformal box method across trials, classes and coordinates (used to generate the ```.csv``` files)
- ```*_res_table_*.csv```: A table with aggregate results for the **conformal box method only** (i.e., using 'oracle' box quantiles and no label sets). The second ```*``` denotes the assigned name of the nonconformity function (e.g., ```abs_res```).
- ```*_box_set_table_*.csv```: A table with aggregate results for the **conformal box method and selected conformal label set method** (i.e., after the full two-step approach).
- ```*_label_table.csv```: A table with aggregate results for the **conformal label set method only** (i.e., no bounding box procedure).
- ```*_label.pt```: A tensor storing all the evaluation metrics for the conformal label set method across trials, classes and coordinates (used to generate the ```.csv``` file).

Since storing all these files for every run can become storage-heavy, boolean toggle arguments can be leveraged (see ```main.py -h```) to reduce the intermediate write operations. However, it can be useful to store intermediate outputs once and then leverage those files to perform cheaper evaluations for other experiments. For example, ```*_img_list.json``` and 
```*_ist_list.json``` contain all the necessary prediction information and can be reused for multiple different post-hoc conformal method combinations.

#### Still open questions?

If there are any problems you encounter which have not been addressed, please feel free to create an issue or reach out!

conformal-od

conformal-od Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

conformal-od

conformal-od Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README