GrUMoDepth

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

(★ 56)

Gradient-based Uncertainty for Monocular Depth Estimation (ECCV 2022)

GrUMoDepth Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Gradient-based Uncertainty for Monocular Depth Estimation

This repository contains the official implementation of our  ECCV 2022 paper.

![Overview](https://raw.githubusercontent.com/jhornauer/GrUMoDepth/main/images/github_overview.png)

## Requirements
We provide the `environment.yml` file with the required packages. The file ca be used to create an Anaconda environment.

Prepare monodepth2: 
1. Download the `monodepth2` repository: [Monodepth2](https://github.com/nianticlabs/monodepth2)
2. Replace `from kitti_utils import generate_depth_map` with `from ..kitti_utils import generate_depth_map` in the file `monodepth2/kitti_dataset.py` in `line 14`. 
3. Replace `from layers import *` with `from ..layers import *` in the file `monodeth2/networks/depth_decoder.py` in `line 14`.
4. Replace `class MonodepthOptions:` with `class MonodepthOptions(object):` in the file `monodepth2/options.py` in `line 15`.
5. Add `import sys` and `sys.path.append("monodepth2")` to the file `monodepth2/trainer.py` before `from utils.py import *`.

## Datasets
We conduct our evaluations on the datasets [NYU Depth V2](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html) and [KITTI](http://www.cvlibs.net/datasets/kitti/raw_data.php). 
[NYU Depth V2](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html) is downloaded as provided by [FastDepth](https://github.com/dwofk/fast-depth) into the folder `nyu_data`. [KITTI](http://www.cvlibs.net/datasets/kitti/raw_data.php) is downloaded according to the instructions from [mono-uncertainty](https://github.com/mattpoggi/mono-uncertainty) into the folder `kitti_data`.

## Pre-trained Models 
We conduct experiments on already trained depth estimation models. The pre-trained models are trained on [KITTI](http://www.cvlibs.net/datasets/kitti/raw_data.php) with monocular and stereo supervision in a self-supervised manner and on [NYU Depth V2](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html) in a supervised manner.
In case of [KITTI](http://www.cvlibs.net/datasets/kitti/raw_data.php), we rely on the already trained modes from [mono-uncertainty](https://github.com/mattpoggi/mono-uncertainty). Please follow their instructions to download the respective model weights. 
Our models trained on [NYU Depth V2](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html) can be downloaded from the following link: [NYU Models](https://cloudstore.uni-ulm.de/s/CbWd7odXepC53Xr).
The models can be trained with the following command: 
```
python3 train_supervised.py --data_path nyu_data --width 288 --height 224 --max_depth 10 --dataset nyu  
```
To train the log-likelihood maximization model use the additional option `--uncert`. To train the MC Dropout model use the additional option `--dropout`.

## Run Code 
We conduct our experiments on [KITTI](http://www.cvlibs.net/datasets/kitti/raw_data.php) (self-supervised) and [NYU Depth V2](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html) (supervised) data.

As explained in our paper, we apply our training-free uncertainty estimation method to already trained models. Therefore, we have different base models and compare the uncertainty estimation approaches on the base models.

### Evaluation Self-Supervised
In the self-supervised case, we have the base models MC Dropout (Drop), Bootstrapped Ensembles (Boot), Post-Processing (Post), Log-likelihood Maximization (Log) and Self-Teaching (Self).
We compare the post hoc uncertainty estimation approaches on *Post*, *Log* and *Self*. As post hoc uncertainty estimation approaches we consider the variance over different test-time augmentations (Var), inference-only dropout (In-Drop) and our approach.

For the evaluation of the Drop model and the Boot model as well as the Log and the Self base models you can refer to [mono-uncertainty](https://github.com/mattpoggi/mono-uncertainty).

#### Evaluation of our gradient-based uncertainty estimation method:
For the evaluation of our method on the Post base model run: 
```
python3 generate_maps.py --data_path kitti_data --load_weights_folder weights/S/Monodepth2-Post/models/weights_19/ --eval_split eigen_benchmark --eval_stereo --output_dir experiments/S/post_model/Grad --grad
python3 evaluate.py --ext_disp_to_eval experiments/S/post_model/Grad/raw/ --eval_stereo --max_depth 80 --eval_split eigen_benchmark --eval_uncert --output_dir experiments/S/post_model/Grad --grad
```

For the evaluation of our method on the Log base model run: 
```
python3 generate_maps.py --data_path kitti_data --load_weights_folder weights/S/Monodepth2-Log/models/weights_19/ --eval_split eigen_benchmark --eval_stereo --output_dir experiments/S/log_model/Grad --uncert --grad --w 2.0 
python3 evaluate.py --ext_disp_to_eval experiments/S/log_model/Grad/raw/ --eval_stereo --max_depth 80 --eval_split eigen_benchmark --eval_uncert --output_dir experiments/S/log_model/Grad --grad --uncert --w 2.0
```

For the evaluation of our method on the Self base model run: 
```
python3 generate_maps.py --data_path kitti_data --load_weights_folder weights/S/Monodepth2-Self/models/weights_19/ --eval_split eigen_benchmark --eval_stereo --output_dir experiments/S/self_model/Grad --uncert --grad --w 2.0 
python3 evaluate.py --ext_disp_to_eval experiments/S/self_model/Grad/raw/ --eval_stereo --max_depth 80 --eval_split eigen_benchmark --eval_uncert --output_dir experiments/S/self_model/Grad --grad --uncert --w 2.0
```

To change the decoder layer for the gradient extraction use the argument `--ext_layer` with values between `0` and `10`.

To change the augmentation for the generation of the reference depth use the argument `--gref` with one of the values: `flip`, `gray` , `noise` or `rot`.

##### Evaluation of In-Drop method: 
For the evaluation of the In-Drop method on the Post base model run: 
```
python3 generate_maps.py --data_path kitti_data --load_weights_folder weights/S/Monodepth2-Post/models/weights_19/ --eval_split eigen_benchmark --eval_stereo --output_dir experiments/S/post_model/In-Drop --infer_dropout
python3 evaluate.py --ext_disp_to_eval experiments/S/post_model/Infer-Drop/raw/ --eval_stereo --max_depth 80 --eval_split eigen_benchmark --eval_uncert --output_dir experiments/S/post_model/In-Drop --infer_dropout
```

To change the dropout probability use the argument `--infer_p` with values between `0.0` and `1.0`. Default ist `0.2`.

##### Evaluation of the Var method: 
For the evaluation of the Var method on the Post base model run: 
```
python3 generate_maps.py --data_path kitti_data --load_weights_folder weights/S/Monodepth2-Post/models/weights_19/ --eval_split eigen_benchmark --eval_stereo --output_dir experiments/S/post_model/Var --var_aug
python3 evaluate.py --ext_disp_to_eval experiments/S/post_model/Var/raw/ --eval_stereo --max_depth 80 --eval_split eigen_benchmark --eval_uncert --output_dir experiments/S/post_model/Var --var_aug
```

For the evaluation of the models trained with monocular supervision replace the folder `S` with `M` and the argument `--eval_stereo` with `--eval_mono`.

### Evaluation Supervised
In the supervised case, we have the base models MC Dropout (Drop), Post-Processing (Post) and Log-likelihood Maximization (Log).
We compare the post hoc uncertainty estimation approaches on *Post* and *Log*. As post hoc uncertainty estimation approaches we consider the variance over different test-time augmentations (Var), inference-only dropout (In-Drop) and our approach.

#### Evaluation of our gradient-based uncertainty estimation method: 
For the evaluation of our method on the Post base model run: 
```
python3 evaluate_supervised.py --max_depth 10 --load_weights_folder weights/NYU/Monodepth2/weights/ --data_path nyu_data --eval_uncert --output_dir experiments/NYU/post_model/Grad/ --grad 
```

For the evaluation of our method on the Log base model run: 
```
python3 evaluate_supervised.py --max_depth 10 --load_weights_folder weights/NYU/Monodepth2-Log/weights/ --data_path nyu_data --eval_uncert --output_dir experiments/NYU/log_model/Grad/ --grad --uncert -w 2.0 
```

##### Evaluation of the Drop model: 
For the evaluation of the Drop model run: 
```
python3 evaluate_supervised.py --max_depth 10 --load_weights_folder weights/NYU/Monodepth2-Drop/weights/ --data_path nyu_data --eval_uncert --output_dir experiments/NYU/drop_model/Drop/ --dropout
```

##### Evaluation of In-Drop method: 
For the evaluation of the In-Drop method on the Post base model run: 
```
python3 evaluate_supervised.py --max_depth 10 --load_weights_folder weights/NYU/Monodepth2/weights/ --data_path nyu_data --eval_uncert --output_dir experiments/NYU/post_model/In-Drop/ --infer_dropout
```

To change the dropout probability use the argument `--infer_p` with values between `0.0` and `1.0`. Default ist `0.2`.

##### Evaluation of the Var method: 
For the evaluation of the Var method on the Post base model run: 
```
python3 evaluate_supervised.py --max_depth 10 --load_weights_folder weights/NYU/Monodepth2/weights/ --data_path nyu_data --eval_uncert --output_dir experiments/NYU/post_model/Var/ --var_aug
```

## Reference
Please use the following citations when referencing our work:

**Gradient-based Uncertainty for Monocular Depth Estimation** \
*Julia Hornauer and Vasileios Belagiannis* **[[paper]](https://arxiv.org/abs/2208.02005)**
```
@inproceedings{Hornauer2022GradientbasedUF,
  title={Gradient-based Uncertainty for Monocular Depth Estimation},
  author={Julia Hornauer and Vasileios Belagiannis},
  booktitle={European Conference on Computer Vision},
  year={2022}
}
```

## Acknowledgement
We used and modified code parts from the open source projects [monodepth2](https://github.com/nianticlabs/monodepth2) and [mono-uncertainty](https://github.com/mattpoggi/mono-uncertainty). We like to thank the authors for making their code publicly available.

GrUMoDepth

GrUMoDepth Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

GrUMoDepth

GrUMoDepth Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README