Attentive_DFPrior

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

Attentive_DFPrior

★ 27 Open GitHub ↗

[NeurIPS'23] Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priors

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priors</h1>
  <p align="center">
    <a>Pengchong Hu</a>
    ·
    <a href="https://h312h.github.io/">Zhizhong Han</a>

</p>
  <h2 align="center">NeurIPS 2023</h2>
  <h3 align="center"><a href="http://arxiv.org/abs/2310.11598">Paper</a> | <a href="https://machineperceptionlab.github.io/Attentive_DF_Prior/">Project Page</a> </h3>
  <div align="center"></div>
</p>

<details open="open" style='padding: 10px; border-radius:5px 30px 30px 5px; border-width: 1px;'>
  <summary>Table of Contents</summary>
  <ol>
    <li>
      <a href="#installation">Installation</a>
    </li>
    <li>
      <a href="#dataset">Dataset</a>
    </li>
    <li>
      <a href="#run">Run</a>
    </li>
    <li>
      <a href="#evaluation">Evaluation</a>
    </li>
    <li>
      <a href="#acknowledgement">Acknowledgement</a>
    </li>
    <li>
      <a href="#citation">Citation</a>
    </li>
  </ol>
</details>

## Installation
Please install all dependencies by following the instructions here. You can use [anaconda](https://www.anaconda.com/) to finish the installation easily.

You can build a conda environment called `df-prior`. Note that for Linux users, you need to install `libopenexr-dev` first before building the environment.

```bash
git clone https://github.com/MachinePerceptionLab/Attentive_DFPrior.git
cd Attentive_DFPrior

sudo apt-get install libopenexr-dev
    
conda env create -f environment.yaml
conda activate df-prior
```

## Dataset

### Replica

Please download the Replica dataset generated by the authors of iMAP into `./Datasets/Replica` folder.

```bash
bash scripts/download_replica.sh # Released by authors of NICE-SLAM
```

### ScanNet

Please follow the data downloading procedure on [ScanNet](http://www.scan-net.org/) website, and extract color/depth frames from the `.sens` file using this [code](https://github.com/ScanNet/ScanNet/blob/master/SensReader/python/reader.py).

<details>
  <summary>[Directory structure of ScanNet (click to expand)]</summary>
  
  DATAROOT is `./Datasets` by default. If a sequence (`sceneXXXX_XX`) is stored in other places, please change the `input_folder` path in the config file or in the command line.

```
  DATAROOT
  └── scannet
      └── scans
          └── scene0000_00
              └── frames
                  ├── color
                  │   ├── 0.jpg
                  │   ├── 1.jpg
                  │   ├── ...
                  │   └── ...
                  ├── depth
                  │   ├── 0.png
                  │   ├── 1.png
                  │   ├── ...
                  │   └── ...
                  ├── intrinsic
                  └── pose
                      ├── 0.txt
                      ├── 1.txt
                      ├── ...
                      └── ...

```
</details>

## Run

To run our code, you first need to generate the TSDF volume and corresponding bounds. We provide the generated TSDF volume and bounds for Replica and ScanNet: <a href="https://drive.google.com/file/d/1RcPuyW6VZJySK8bOscvkCMDEJIa1aMaA/view?usp=sharing" target="_blank" rel="noopener">replica_tsdf_volume.tar</a>, <a href="https://drive.google.com/file/d/1EqZ-Q1kEPhZKHOGc8Y1eKxAz-va_0icN/view?usp=sharing" target="_blank" rel="noopener">scannet_tsdf_volume.tar</a>.

You can also generate the TSDF volume and corresponding bounds by using the following code:

```bash
CUDA_VISIVLE_DEVICES=0 python get_tsdf.py configs/Replica/room0.yaml --space 1 # For Replica
CUDA_VISIVLE_DEVICES=0 python get_tsdf.py configs/ScanNet/scene0050_00.yaml --space 10 # For ScanNet
```

You can run DF-Prior by using the following code:

```bash
CUDA_VISIVLE_DEVICES=0 python -W ignore run.py configs/Replica/room0.yaml # For Replica
CUDA_VISIVLE_DEVICES=0 python -W ignore run.py configs/ScanNet/scene0050.yaml # For ScanNet
```

The mesh for evaluation is saved as `$OUTPUT_FOLDER/mesh/final_mesh_eval_rec.ply`, where the unseen regions are culled using all frames.

## Evaluation
### Average Trajectory Error
To evaluate the average trajectory error. Run the command below with the corresponding config file:
```bash
python src/tools/eval_ate.py configs/Replica/room0.yaml
```

### Reconstruction Error
#### Replica
To evaluate the reconstruction error in Replica, first download the ground truth Replica meshes where unseen region have been culled.
```bash
bash scripts/download_cull_replica_mesh.sh # Released by authors of NICE-SLAM
```
Then run the command below. The 2D metric requires rendering of 1000 depth images, which will take some time (~9 minutes). Use `-2d` to enable 2D metric. Use `-3d` to enable 3D metric.
```bash
# Assign any output_folder and gt mesh you like, here is just an example
OUTPUT_FOLDER=output/Replica/room0
GT_MESH=cull_replica_mesh/room0.ply
python src/tools/eval_recon.py --rec_mesh $OUTPUT_FOLDER/mesh/final_mesh_eval_rec.ply --gt_mesh $GT_MESH -2d -3d
```

We also provide code to cull the mesh given camera poses. Here, we take the culling of the ground truth mesh of Replica room0 as an example.
```bash
python src/tools/cull_mesh.py --input_mesh Datasets/Replica/room0_mesh.ply --traj Datasets/Replica/room0/traj.txt --output_mesh cull_replica_mesh/room0.ply
```
#### ScanNet
To evaluate the reconstruction error in ScanNet, first download the <a href="https://drive.google.com/file/d/1_abRXNcEBGwYxvoA3MjO-RLds5672THY/view?usp=sharing" target="_blank" rel="noopener">ground truth ScanNet meshes.zip</a> into `./Datasets/scannet` folder. Then you can just run the command below.
```bash
python src/tools/evaluate_scannet.py configs/ScanNet/scene0050.yaml 
```
We also provide our reconstructed meshes in Replica and ScanNet for evaluation purposes: <a href="https://drive.google.com/file/d/1MUGPRbzL9yfWQ14QwLb91Gigsh0CegVX/view?usp=sharing" target="_blank" rel="noopener">meshes.zip</a>.

## Acknowledgement
We adapt codes from some awesome repositories, including [NICE-SLAM](https://github.com/cvg/nice-slam), [NeuralRGBD](https://github.com/dazinovic/neural-rgbd-surface-reconstruction), [tsdf-fusion](https://github.com/andyzeng/tsdf-fusion-python), [manhattan-sdf](https://github.com/zju3dv/manhattan_sdf), [MonoSDF](https://github.com/autonomousvision/monosdf). Thanks for making the code available. We also thank [Zihan Zhu](https://zzh2000.github.io/) of [NICE-SLAM](https://github.com/cvg/nice-slam), for prompt responses to our inquiries regarding the details of their methods.

## Citation
If you find our code or paper useful, please cite
```bibtex
@inproceedings{Hu2023LNI-ADFP,
      title = {Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priors},
      author = {Hu, Pengchong and Han, Zhizhong},
      booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
      year = {2023}
    }
```

Attentive_DFPrior

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README