Wheat-3DGS

KO | EN

gitlite — search

#javascript #python #llm #go #android #deep-learning #linux #docker #cli #kubernetes #vue #mcp #ios #open-source

(★ 27)

Official implementation of Wheat3DGS: In-field 3D Reconstruction, Instance Segmentation and Phenotyping of Wheat Heads with Gaussian Splatting

Wheat-3DGS Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<p align="center">
  <h2 align="center">🌾 Wheat3DGS <br> In-field 3D Reconstruction, Instance Segmentation and Phenotyping of Wheat Heads with Gaussian Splatting</h>
  <h3 align="center">CVPR 2025 Agriculture-Vision Workshop</h3>
</p>

[Project Page](https://joaquin-gajardo.github.io/wheat3dgs/) | [Paper](https://openaccess.thecvf.com/content/CVPR2025W/V4A/html/Zhang_Wheat3DGS_In-field_3D_Reconstruction_Instance_Segmentation_and_Phenotyping_of_Wheat_CVPRW_2025_paper.html) |  [Arxiv](https://arxiv.org/abs/2504.06978) | [Data](https://drive.google.com/drive/folders/1DJPs_E8-93dCysYkQ0-uxHrAcZGTZiVh)

## Updates
- <b>[5/16/2025]</b> 3D gaussian segmentation viewer
- <b>[5/2/2025]</b>  Thx [Tomi](https://github.com/tomedic) for releasing code for wheat head morphology calculation at `wheatheadsmorphology`!
- <b>[4/30/2025]</b> Initial code release

## 📝 TODO List
- [ ] Share Docker setup
- [ ] Script to to get initial masks
- [ ] Data to ETH research collection including laser scans

## 🛠️ Setup
The setup should be very similar to the original [3D Gaussian Splatting](https://github.com/graphdeco-inria/gaussian-splatting) except we used a modified version of [differential gaussian rasterization](https://github.com/ashawkey/diff-gaussian-rasterization/tree/8829d14f814fccdaf840b7b0f3021a616583c0a1) with support of depth & alpha rendering, and an additional [flashsplat-rasterization](https://github.com/florinshen/flashsplat-rasterization/tree/189c483ffa33dd6d5661343ce496df0c6eb80a0c) submodule.

```
git clone --recursive https://github.com/zdwww/Wheat-3DGS.git
conda env create --file environment.yml
conda activate wheat3dgs
```

## 🌾 Dataset
[Data](https://drive.google.com/drive/folders/1DJPs_E8-93dCysYkQ0-uxHrAcZGTZiVh) used for Wheat3DGS and the baseline results for each plot (from 461 to 467) are structured as follows:
```bash
plot_46x/
├── images/ # 36 images captured by FIP
│   ├──FPWW036_SR0461_FIP2_cam_01.png 
│   └──FPWW036_SR0461...
├── sparse/ # camera intrinsics/extrinsics in COLMAP format
├── yolosam.zip # please unzip to extract 2D masks formatted for Wheat3DGS
│   ├──bboxes/ # YOLO bounding boxes per image (xyxy)
│   │  ├──FPWW036_SR0461_1_FIP2_cam_01.pt
│   │  └──...
│   └──masks/ # 2D segmention mask corresponds to each box (per instance per image) 
│   │  ├──FPWW036_SR0461_1_FIP2_cam_01_000.png
│   │  ├──FPWW036_SR0461_1_FIP2_cam_01_001.png
│   │  └──...
│   ├──yolo_vis/ # for visualization purpose only
│   └──sam_vis/ # for vis only
├── semantics/ # combined 2D masks per image for FruitNeRF baseline
├── manual_label/ # contains a randomly selected test view image with human-annotated wheat heads
└── transforms.json # data in NeRF format
```
Please unzip `yolosam.zip` for each plot before running the pipeline. The image names are structured as follows. For example, in `FPWW036_SR0461_FIP2_cam_01.png`: `FPWW036_SR0461` is shared by all images from the same plot; `FIP2` indicates the batch ID, which may appear as `1_FIP2`, `6_FIP`, or `FIP2`; and the final element `01` is the camera ID, ranging from `01` to `12`. Our data preprocessing functions already handle the naming convention correctly. To get a clearer view of how the cameras are positioned in 3D space and how the train/test views are distributed, please run `vis_cam.py` for visualization.

## 💻 Using Wheat3DGS

### Quick start

The majority of the Wheat3DGS pipeline can be executed by running this script.
```
sbatch run_wheat_3dgs.sh
```

### Step-by-step instructions

**Step 1.** To train a vanilla 3DGS scene reconstruction using wheat plot images as initialization:
```
python train_vanilla_3dgs.py -s <dataset_path> -m <model_path> --resolution 1 --eval
```
Note that `resolution` is set to 1 to prevent image downscaling, and the training/test split for evaluation follows the default setting of our dataset.

You can also run `render.py` and `metrics.py` with the same input arguments to evaluate the quality of novel view synthesis on the reconstructed wheat fields (see Sec. 5.1).

**Step 2 (Optional).** Although wheat head bounding boxes generated by YOLOv5 and 2D segmentations by SAM are already provided in our dataset, please refer to this [Colab notebook](https://colab.research.google.com/drive/1Bz5MiyjpYY1iy37ceSBSfLZCI55roRKl?usp=sharing) to reproduce results.

**Step 3.** To run 3D segmentation on a wheat plot to identify and associate decoupled 2D wheat heads:
```
python run_3d_seg.py -s <dataset_path> -m <model_path> --iou_threshold 0.6 --exp_name run1
``` 
| Parameter | Type | Description |
| :-------: | :--: | :---------: |
| `--iou_threshold` | `float` | The IoU threshold for associating rendered and YOLO-SAM-generated 2D segmentations. A higher threshold results in more conservative mask matching. |
| `--exp_name` | `str` | The experiment name used to create the folder for saving all results under `<model_path>/wheat_head`.

**Step 4.** To generate a video rotating around the reconstructed wheat field for **qualitative** evaluation of the 3D segmentation results (as shown on our project page), run:
```
python render_360.py -s <dataset_path> -m <model_path> --render_type <field or head> --exp_name run1 --n_frames 200 --framerate 20 --elevation 45
```
| Parameter | Type | Description |
| :-------: | :--: | :---------: |
| `--render_type` | `field` or `head` | Either render a video rotating around the entire reconstructed wheat field with the 3D segmentation overlaid on it, or a list of close-up videos of individual identified wheat heads. |
|`--n_frames` | `int` | Number of video frames (It’s recommended to set a relatively small value for wheat heads, as hundreds of them will be processed.)|
|`--framerate`| `int` | Framerate. Length of video = `n_frames / framerate`|
|`--elevation`| `int` | Elevation angle for the camera trajectory rotating around the scene. The larger the angle, the more novel the rendered view is, since the training views are captured by an overhead camera. |

Results will be saved as a video `<model_path>/wheat_head/<exp_name>/wheat_field_360.mp4` for the whole wheat field and a list of videos under the directory `.../<exp_name>/wheat_head_360` for each wheat head id.

**Step 5.** To **quantitatively** evaluate the quality of 3D segmentation and reproduce Table 2 in the paper, project the 3D segmentation into 2D by
```
python eval_wheatgs.py -s <dataset_path> -m <model_path> --exp_name run1 --skip_train --load_counts
```
Rendered 2D segmentation will be saved under `model_path/test/ours_<iteration>/segmentation` and `overlay`, where they can be used for metric calculation.

## Baseline
To reproduce the baseline results presented in the paper (i.e. [FruitNeRF](https://github.com/meyerls/FruitNeRF)), please refer to the original repository and the scripts in the `scripts` folder.

## Visualization
Our viewer is based on the implementation of [viser](https://github.com/nerfstudio-project/viser), allowing users to freely explore the reconstructed wheat field with overlaid 3D segmentation of wheat heads. Please refer to the scripts under folder `wheat3dgsviewer` for usage instructions.

## Acknowledgement
Our implementation is based on the original [3D Gaussian Splatting](https://github.com/graphdeco-inria/gaussian-splatting) and [FlashSplat](https://github.com/florinshen/FlashSplat). We thank the authors for their revolutionary work and open-source contributions.

## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE.md) file for details. Please note that the 3D Gaussian Splatting code base, however, is licensed under a different license, which can be found in the [LICENSE_3DGS](LICENSE_3DGS.md) file.

## Citation
If you find our paper useful, please cite us:
```bib
@InProceedings{Zhang_Gajardo_2025_CVPR,
    author    = {Zhang, Daiwei and Gajardo, Joaquin and Medic, Tomislav and Katircioglu, Isinsu and Boss, Mike and Kirchgessner, Norbert and Walter, Achim and Roth, Lukas},
    title     = {Wheat{3DGS}: {I}n-field {3D} {R}econstruction, {I}nstance {S}egmentation and {P}henotyping of {W}heat {H}eads with {G}aussian {S}platting},
    booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR) Workshops},
    month     = {June},
    year      = {2025},
    pages     = {5369-5379}
}
```

Wheat-3DGS

Wheat-3DGS Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Wheat-3DGS

Wheat-3DGS Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README