KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #developer-tools #codex #claude-code #cli #react #mcp #open-source #deepseek-harness #dsh-plugin #ai

pangaea-bench

★ 281 Open GitHub ↗

Towards Robust Evaluation for Geospatial Foundation Models

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

[![Tests](https://github.com/yurujaja/geofm-bench/actions/workflows/python-test.yml/badge.svg)](https://github.com/yurujaja/geofm-bench/actions/workflows/python-test.yml)

 <img src=".github/pangaea-banner.png" alt="PANGAEA" width="480"/>

# PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models

📢 **News**
 - [04/06/2025] We integrate [Geo-Bench](https://arxiv.org/abs/2306.03831) Datasets, including six segmentation and six classification tasks.
 
 - [22/04/2025] on EarthDay, PANGAEA was officialy adopted to benchmark TerraMind. Read the [news](https://www.linkedin.com/posts/simonetta-cheli-7669879b_earthday-earthobservation-activity-7320439907028467712-LSzl?utm_source=share&utm_medium=member_desktop&rcm=ACoAACdT8q0BDNWYKAdDYGUe_X4fQOzSHO8jgAs) and the [pre-print](https://arxiv.org/abs/2504.11171). We will release the benchmarking code in PANGAEA very soon!
 - [05/12/2024] the [pre-print](https://arxiv.org/abs/2412.04204) is out!

## 📚 Introduction

While geospatial foundation models (GFMs) have proliferated rapidly, their evaluations remain inconsistent and narrow. Existing works often utilize suboptimal downstream datasets (e.g., EuroSAT) and tasks (e.g., land cover classification), which constrain comparability and real-world usability. Additionally, a lack of diversity in evaluation protocols, including image resolution and sensor types, further complicates the extensive assessments of GFM performance.

To bridge this gap, we propose a standardized evaluation protocol that incorporates a wide-ranging selection of datasets, tasks, resolutions, and sensor types, establishing a robust and widely applicable benchmark for GFMs.

In this repo, you can find the code to benchmark GFMs. For the moment we included several GFMs that present different approaches. We look forward to adding new models and datasets.

For the moment, we support the following **models**:

| | Paper | GitHub | Keywords |
|:-----------:|:-----:|:------:|:--------:|
| [SSL4EOS12](https://arxiv.org/abs/2211.07044) | SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation | [link](https://github.com/zhu-xlab/SSL4EO-S12) | DINO, MAE, DATA2VEC, MOCO|
| [Scale-MAE](https://arxiv.org/pdf/2212.14532) | Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning | [link](https://github.com/bair-climate-initiative/scale-mae) | Masked Autoencoders, Multiscale|
| [SatlasNet](https://arxiv.org/pdf/2211.15660) | SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding | [link](https://github.com/allenai/satlas/tree/main) | Supervised, Multi-temporal |
| [GFM](https://arxiv.org/pdf/2302.04476) | Towards Geospatial Foundation Models via Continual Pretraining | [link](https://github.com/mmendiet/GFM) | Swin, Continual Pre-training |
| [SpectralGPT](https://arxiv.org/abs/2311.07113) | SpectralGPT: Spectral Remote Sensing Foundation Model | [link](https://github.com/danfenghong/IEEE_TPAMI_SpectralGPT) | MAE, Multi-spectral |
| [DOFA](https://arxiv.org/pdf/2403.15356) | Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation | [link](https://github.com/zhu-xlab/DOFA) | MAE, Dynamic bands |
| [CROMA](https://arxiv.org/pdf/2311.00566) | CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders | [link](https://github.com/antofuller/CROMA) | Contrastive Learning, MAE |
| [Prithvi](https://arxiv.org/pdf/2310.18660) | Foundation Models for Generalist Geospatial Artificial Intelligence | [link](https://github.com/NASA-IMPACT/hls-foundation-os) | MAE, Multi-temporal |
| [RemoteCLIP](https://arxiv.org/pdf/2306.11029) | RemoteCLIP: A Vision Language Foundation Model for Remote Sensing | [link](https://github.com/ChenDelong1999/RemoteCLIP) | Contrastive Learning |

And the following **datasets**:

**Note**: The following datasets are **community-contributed** and are not part of the original benchmark repository. We are grateful for these contributions, which help enrich the benchmark's diversity and applicability.
- **Potsdam dataset** [[Link](https://www.isprs.org/education/benchmarks/UrbanSemLab/2d-sem-label-potsdam.aspx)]. Contributed by [@pierreadorni](https://github.com/pierreadorni).
- **Open-Canopy** [[Link](https://arxiv.org/abs/2407.09392)]. Contributed by [@pierreadorni](https://github.com/pierreadorni).
- **Geo-Bench datasets** [[Link](https://github.com/ServiceNow/geo-bench)]. Contributed by [@yurujaja](https://github.com/yurujaja).

The repository supports the following **tasks** using geospatial (foundation) models:
 - [Single Temporal Semantic Segmentation](#single-temporal-semantic-segmentation)
 - [Multi-Temporal Semantic Segmentation](#multi-temporal-semantic-segmentation)
 - [Change Detection](#change-detection)
 - [Single Temporal Regression](#single-temporal-regression)
 - [Multi-Temporal Regression](#multi-temporal-regression)
 - [Linear Classification](#linear-classification)
 - [KNN Probe Classification](#knn-probe-classification)

It is also possible to train some [supervised baselines](#-fully-supervised-baseline), based on UNet and ViT.

## 🗺️ Datasets details
Please refer to [**Dataset Guide**](DATASET_GUIDE.md) to understand the processing requirements and commands specific to each dataset.

If you want to fast-prototype your model, maybe you want to run fast experiments on smaller datasets. We suggest starting with MADOS, HLSBurnScars, SpaceNet7 and Sen1Floods11 and AI4SmallFarms. They offer good diversity in satellites and domains. In the future, we will release stratified subsets for each dataset to facilitate fast prototyping across all datasets.

## 🛠️ Setup
Clone the repository:
```
git clone https://github.com/VMarsocci/pangaea-bench.git
cd pangaea-bench
```

**Dependencies**

We provide several ways to install the dependencies.

1. **Using either Conda or Mamba**:
    ```
    conda env create -f environment.yaml
    conda activate pangaea-bench
    ```

Optional: install [Mamba](https://github.com/conda-forge/miniforge/releases/) for faster resolution times
    ```
    wget https://github.com/conda-forge/miniforge/releases/download/24.3.0-0/Mambaforge-24.3.0-0-Linux-x86_64.sh
    sh ./Mambaforge-24.3.0-0-Linux-x86_64.sh

mamba env create -f environment.yaml
    mamba activate pangaea-bench
    ```

2. **Using pip**, create a Python native virtual environment and install dependencies into it:
   ```
   export PANGAEA_PATH=/path/to/venv/pangaea-bench # change this
   python3 -m venv ${PANGAEA_PATH}
   source ${PANGAEA_PATH}/bin/activate
   
   pip install -r requirements.txt
   ```
 **Then install the code repository as a development package**
   ```
   pip install --no-build-isolation --no-deps -e .
   ```

## 🏋️ Training

To run experiments, please refer to `configs/train.yaml`. In it, in addition to some basic info about training (e.g. `finetune` for fine-tuning also the encoder, `limited_label_train` to train the model on a stratified subset of labels, `num_workers`, `batch_size` and so on), there are 5 different basic configs:
- `dataset`: Information of downstream datasets such as image size, band_statistics, classes etc.
- `decoder`: Downstream task decoder fine-tuning related parameters, like the type of architecture (e.g. UPerNet), which multi-temporal strategy to use, and other related hparams (e.g. nr of channels)
- `encoder`: GFM encoder related parameters. `output_layers` is used for which layers are used for Upernet decoder.  
- `preprocessing`: Both preprocessing and augmentations steps required for the dataset, such as bands adaptation, normalization, resize/crop.
- `task`: Information about the trainer and evaluator. Most of the parameters are overwritten in run. Trainer and evaluator can be used for segmentation (`SegTrainer`) or regression (`RegTrainer`). Different parameter like precision training (`precision`) can be set in it.

Other 3 configs are used to set other training parameters:
- `criterion`: in which you can choose the loss for the training. Consider that if you want to add a custom loss, you should add to `pangaea/utils/losses.py`. Currently, we support `cross_entropy`, `weigthed_cross_entropy`, `dice` and `mae` loss functions.
- `lr_scheduler`: in which you can choose the scheduler. Consider that if you want to add a custom one, you should add to `pangaea/utils/schedulers.py`. 
- `optimizer`: in which you can choose the optimizer. Consider that if you want to add a custom one, you should add to `pangaea/utils/optimizers.py`.

We provide several examples of command lines to initialize different training tasks on single GPU.

Please note:
 - The repo adopts [hydra](https://github.com/facebookresearch/hydra), so you can easily log your experiments and overwrite parameters from the command line. More examples are provided later.
 - To use more gpus or nodes, set `--nnodes` and `--nproc_per_node` correspondingly. Please refer to the [torchrun doc](https://pytorch.org/docs/stable/elastic/run.html).

### 💻 Decoder Finetuning
#### Single Temporal Semantic Segmentation

Take HLSBurnScars dataset, RemoteCLIP Encoder and Upernet Segmentation Decoder as example:
```
torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
   --config-name=train \
   dataset=hlsburnscars \
   encoder=remoteclip \
   decoder=seg_upernet\
   preprocessing=seg_default \
   criterion=cross_entropy \
   task=segmentation
```

If you want to overwrite some parameters (e.g. turn off wandbe, change batch size and the path to the dataset, and use 50% stratified sampled subset for training):
```
torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
   --config-name=train \
   dataset=hlsburnscars \
   encoder=remoteclip \
   decoder=seg_upernet\
   preprocessing=seg_default \
   criterion=cross_entropy \
   task=segmentation \
   dataset.root_path= /path/to/the/dataset/hlsburnscars \
   batch_size=16 \
   use_wandb=False \
   limited_label_train=0.5 \
   limited_label_strategy=stratified
```

#### Multi-Temporal Semantic Segmentation
- Multi-temporal decoder config (e.g. `configs/decoder/seg_upernet_mt_ltae.yaml` if you want to use `ltae` as a strategy to combine multi-temporal info) should be used. 
- In addition, in the dataset config, indicate the number of time frames, e.g., `multi_temporal: 6`

An example of using SSL4EO-DINO on CropTypeMapping is as below
```
torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
   --config-name=train \
   dataset=croptypemapping \
   encoder=ssl4eo_dino \
   decoder=seg_upernet_mt_ltae \
   preprocessing=seg_resize \
   criterion=cross_entropy \
   task=segmentation
```

To use SatlasNet encoder, the `configs/encoder/satlasnet_mi.yaml` is required
```
torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
   --config-name=train \
   dataset=croptypemapping \
   encoder=satlasnet_mi \
   decoder=seg_upernet_mt_ltae \
   preprocessing=seg_resize \
   criterion=cross_entropy \
   task=segmentation
```

To overwrite parameters, please check the Single Temporal Semantic Segmentation example.

#### Change Detection

One of the change detection decoder should be used: `configs/decoder/seg_siamupernet_conc.yaml` employs feature concatenation strategy while `configs/decoder/seg_siamupernet_diff.yaml` uses feature differencing strategy. For example, Prithvi encoder on xView2:

```
torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
   --config-name=train \
   dataset=xview2 \
   encoder=prithvi \
   decoder=seg_siamupernet_conc \
   preprocessing=seg_default \
   criterion=cross_entropy \
   task=change_detection
```

To overwrite parameters, please check the Single Temporal Semantic Segmentation example.

#### Single Temporal Regression

The regression decoder (e.g. `configs/decoder/reg_upernet.yaml`) and the regression task (e.g. `configs/task/regression.yaml`) configs should be used. 
E.g. Prithvi encoder on BioMassters

```
torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
   --config-name=train \
   dataset=biomassters \
   encoder=prithvi \
   decoder=reg_upernet \
   preprocessing=reg_default \
   criterion=mse \
   task=regression
```
To use SatlasNet encoder, the `configs/encoder/satlasnet_si.yaml` is required.
To overwrite parameters, please check the Single Temporal Semantic Segmentation example.

#### Multi-Temporal Regression

The multi-temporal regression decoder (e.g. `configs/decoder/reg_upernet_mt_ltae.yaml` or `configs/decoder/reg_upernet_mt_linear.yaml`) and the regression task (e.g. `configs/task/regression.yaml`) configs should be used.

Take Prithvi encoder on BioMassters as example:

```
torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
   --config-name=train \
   dataset=biomassters \
   encoder=prithvi \
   decoder=reg_upernet_mt_ltae \
   preprocessing=reg_default \
   criterion=mse \
   task=regression
```

To use SatlasNet encoder, please refer to the multi-temporal semantic segmentation example.
To overwrite parameters, please check the Single Temporal Semantic Segmentation example.

#### Linear Classification

For linear classification, you can use either single-label or multi-label classification depending on your dataset. The main difference is in the task config and criterion used.

For single-label classification (e.g. m-Brick-Kiln dataset):
```
export GEO_BENCH_DIR=YOUR/PATH/DIR   # Note that `export GEO_BENCH_DIR=YOUR/PATH/DIR` is required for Geo-Bench datasets.
torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
   --config-name=train \
   dataset=mbrickkiln \
   encoder=remoteclip \
   decoder=cls_linear \
   preprocessing=cls_resize \
   criterion=cross_entropy \
   task=linear_classification \
   task.trainer.n_epochs=50 \
   batch_size=16 \
   finetune=false
```

For multi-label classification (e.g. MBigEarthNet dataset):
```
export GEO_BENCH_DIR=YOUR/PATH/DIR   # Note that `export GEO_BENCH_DIR=YOUR/PATH/DIR` is required for Geo-Bench datasets.
torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
   --config-name=train \
   dataset=mbigearthnet \
   encoder=remoteclip \
   decoder=cls_linear \
   preprocessing=cls_resize \
   criterion=binary_cross_entropy \
   task=linear_classification_multi_label \
   task.trainer.n_epochs=50 \
   batch_size=16 \
   finetune=false
```

#### KNN Probe Classification

For KNN probe classification, you'll use a different decoder and task config. This is useful for evaluating the quality of learned representations without any fine-tuning. Here's an example using the m-EuroSat dataset:

```
export GEO_BENCH_DIR=YOUR/PATH/DIR   # Note that `export GEO_BENCH_DIR=YOUR/PATH/DIR` is required for Geo-Bench datasets.
torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
   --config-name=train \
   dataset=meurosat \
   encoder=remoteclip \
   decoder=cls_knn \
   preprocessing=cls_resize \
   criterion=none \
   task=knn_probe \
   batch_size=32 \
   finetune=false
```

Note that for KNN probe:
- The criterion is set to `none` since no training is performed
- The batch size can be larger since we're only doing inference
- `finetune` is set to `false` as we're only using the pre-trained encoder

### 💻 End-to-end Finetuning

It is enough to add `finetune=True` to the command line.

For example, for single-temporal semantic segmentation:
```
torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
   --config-name=train \
   dataset=hlsburnscars \
   encoder=remoteclip \
   decoder=upernet\
   preprocessing=default \
   criterion=cross_entropy \
   task=segmentation \
   finetune=True
```

### 💻 Fully Supervised Baseline

The repo supports also training fully supervised baselines (i.e. UNet and ViT). To run these, follow the same command line rules as for other models. Keep in mind that setting finetune=True is necessary since this fully supervised approach trains the model from scratch. 
An example for single temporal semantic segmentation with UNet is provided (Sen1Floods11 dataset):
```
torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
   --config-name=train \
   dataset=sen1floods11 \
   encoder=unet_encoder \
   decoder=seg_unet \
   preprocessing=seg_default \
   criterion=cross_entropy \
   task=segmentation \
   finetune=True
```
There is no multi-temporal UNet supported.

An example for multi-temporal semantic segmentation with ViT is provided (CropTypeMapping-SS dataset):
```
torchrun --nnodes=1 --nproc_per_node=1 pangaea/run.py \
   --config-name=train \
   dataset=croptypemapping \
   encoder=vit_scratch \
   decoder=seg_upernet_mt_ltae \
   preprocessing=seg_default \
   criterion=cross_entropy \
   task=segmentation \
   task.evaluator.inference_mode=whole \
   finetune=true
```

## 🔧 Customization

### Using Your Own Dataset

Refer to: [Adding a new downstream dataset](CONTRIBUTING.md#adding-a-new-downstream-dataset)

### Using Your Own Model

Refer to: [Adding a new geospatial foundation model](CONTRIBUTING.md#adding-a-new-geospatial-foundation-model)

## 🏃 Evaluation

An evaluation step is always run after the training.

If you want to just run an evaluation, indicate the `ckpt_dir` where the checkpoints and configurations are stored.

```
torchrun pangaea/run.py --config-name=test ckpt_dir=path_to_ckpt_dir
```

## ✏️ Contributing
We appreciate all contributions. Please refer to [Contributing Guidelines](CONTRIBUTING.md).

## ⚠️ TO DO

- host all weights/datasets/subsets on HF (the automatic download is working for all the datasets and models' weights but, respectively, **Five Billion Pixels**, **BioMassters**, and **GFM**. The GFM pretrained model can be downloaded from [OneDrive](https://onedrive.live.com/?authkey=%21AIx%5FHoX7JG4Ai18&id=93B3D3BDA9EFE744%21100937&cid=93B3D3BDA9EFE744).)
 - add hyperparameters search ([Optuna](https://optuna.org/))
 - support automatic running of all the experiments
 - create an Arena to fast benchmark all the GFMs

## 🧮 Some results

Check the paper for all the insights!

NOTE: if you want to benchmark the results of your model, for a fair comparison do not change the hparams in the configs! Soon we will publish also a set of "benchmark-configs", to support automatic running.

## 📝 Citation

If you find this work useful, please cite:

```
@misc{marsocci2024pangaeaglobalinclusivebenchmark,
      title={PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models}, 
      author={Valerio Marsocci and Yuru Jia and Georges Le Bellier and David Kerekes and Liang Zeng and Sebastian Hafner and Sebastian Gerard and Eric Brune and Ritu Yadav and Ali Shibli and Heng Fang and Yifang Ban and Maarten Vergauwen and Nicolas Audebert and Andrea Nascetti},
      year={2024},
      eprint={2412.04204},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.04204}, 
}
```
##  Acknowledge

The computations/data handling were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725.

pangaea-bench

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README