HBEVOcc

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #cli #claude-code #codex #developer-tools #react #windows

★ 12 Open GitHub ↗

HBEVOcc: Height-aware Bird's-Eye-View Representation for 3D Occupancy Prediction from Multi-Camera Images

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# HBEVOcc: Height-aware Bird's-Eye-View Representation for 3D Occupancy Prediction from Multi-Camera Images

## 🤗 Model Weights

We utilize 8 RTX2080Ti GPUs to train our model.

|         Setting         | Epochs | Training MeM (Batch Size=1) | RayIoU | mAVE |                                                Weights (Batch Size=2) Logs                                              | 
|:-----------------------:|:------:|:-------------:|:------:|:----:|:-------------------------------------------------------------------------------------------------------:|
| r50_256x704_1f_openocc |  24   |  7.3GB   |  39.4  | 0.52 | [Google-drive](https://drive.google.com/drive/folders/1_RDLHJUcJ9vQMO9nNOBrShGdb5lgBMwR?usp=drive_link) |
| r50_256x704_8f_openocc |  24   |  7.5GB   |  40.8  | 0.41 | [Google-drive](https://drive.google.com/drive/folders/1_RDLHJUcJ9vQMO9nNOBrShGdb5lgBMwR?usp=drive_link) |
| r50_256x704_8f_openocc |  48   |  7.5GB   |  41.4  | 0.39 | [Google-drive](https://drive.google.com/drive/folders/1_RDLHJUcJ9vQMO9nNOBrShGdb5lgBMwR?usp=drive_link) |

The code for the Occ3D dataset is coming soon..

## 🛠️Environment

Install Pytorch 1.13 + CUDA 11.6

```bash
conda create --name hbevocc python=3.8
conda activate hbevocc
pip install torch==1.13.0+cu116 torchvision==0.14.0+cu116 torchaudio==0.13.0 --extra-index-url https://download.pytorch.org/whl/cu116
pip install mmcv-full==1.7.0 -f https://download.openmmlab.com/mmcv/dist/cu117/torch1.13/index.html
pip install mmdet==2.28.2
pip install mmsegmentation==0.30.0
pip install mmengine
pip install -v -e .
pip install numpy==1.23.4
pip install yapf==0.40.1
pip install setuptools==59.5.0
pip install ninja
pip install einops
pip install open3d==0.16.0 -i https://pypi.tuna.tsinghua.edu.cn/simple
```
## 🏗 Prepare Dataset

1. Download nuScenes from [nuScenes](https://www.nuscenes.org/nuscenes) 
2. Download Occ3D-nus from [Occ3D-nus](https://drive.google.com/file/d/1kiXVNSEi3UrNERPMz_CfiJXKkgts_5dY/view?usp=drive_link)
3. Download OpenOcc from [OpenOcc-OpenDataLab](https://opendatalab.com/OpenDriveLab/CVPR24-Occ-Flow-Challenge/tree/main) 
or [OpenOcc-Google Drive](https://drive.google.com/drive/folders/1lpqjXZRKEvNHFhsxTf0MOE13AZ3q4bTq)
4. Download the generated info file from [Google Drive](https://drive.google.com/file/d/1KP25b3excY4N-3rqfkijuUmJLZeMwxZw/view?usp=sharing)
and unzip it to the `data/nuscenes` folder. Rename them to `'hbevocc-nuscenes_infos_train.pkl'` and `'hbevocc-nuscenes_infos_val.pkl'`. These `*pkl` files can be generated by running the `tools/create_data_bevdet.py`
5. Download the pretrained weights from [BEVDet](https://github.com/HuangJunJie2017/BEVDet)
6. Organize your folder structure as below:

```
├── project
├── ckpts/
│   ├── bevdet-r50-4d-stereo-cbgs.pth
│   ├── bevdet-r50-4dlongterm-stereo-cbgs.pth
├── data/
│   ├── nuscenes/
│   │   ├── samples/ 
│   │   ├── v1.0-trainval/
│   │   ├── gts/ (Occ3D-nus)
│   │   ├── openocc_v2/
│   │   ├── hbevocc-nuscenes_infos_train.pkl
│   │   ├── hbevocc-nuscenes_infos_val.pkl
```

## 💡 Training and Evaluation

Train HBEVOcc with 8 GPUs:

```train
bash tools/dist_train.sh config/hbevocc/hbevocc-r50-256x704-1f-openocc.py 8
bash tools/dist_train.sh config/hbevocc/hbevocc-r50-256x704-8f-openocc.py 8
```

Evaluate HBEVOcc with 1 GPU or 8 GPUs:

```eval
bash tools/dist_test.sh config/hbevocc/hbevocc-r50-256x704-1f-openocc.py path/to/ckpts 1
bash tools/dist_test.sh config/hbevocc/hbevocc-r50-256x704-8f-openocc.py path/to/ckpts 8
```

## 🧩 Results
<div align="center">
<img src="asserts/comparison_gpu.png" width="90%">
</div>
<div align="center">
<img src="asserts/miou_occ3d.png" width="90%">
</div>
<div align="center">
<img src="asserts/rayiou-occ3d.png" width="90%">
</div>
<div align="center">
<img src="asserts/rayiou-openocc.png" width="90%">
</div>

## 🙏 Acknowledgement
Many thanks to these excellent projects:

- [STCOcc](https://github.com/lzzzzzm/STCOcc)
- [FlashOCC](https://github.com/Yzichen/FlashOCC)
- [BEVDet](https://github.com/HuangJunJie2017/BEVDet)
- [FB-Occ](https://github.com/NVlabs/FB-BEV)
- [BEVFormer](https://github.com/fundamentalvision/BEVFormer)
- [DAT](https://github.com/LeapLabTHU/DAT)
- [mmdetection3d](https://github.com/open-mmlab/mmdetection3d)

HBEVOcc

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README