GeoLink_NeurIPS2025

KO | EN

gitlite — search

#javascript #hacktoberfest #ai #llm #docker #macos #nodejs #java #cli #kubernetes #pytorch #mcp #ios #open-source

GeoLink_NeurIPS2025

(★ 68)

Official implementation and checkpoints of GeoLink remote sensing foundation model in NeurIPS2025.

GeoLink_NeurIPS2025 Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data 🌍🛰️

Welcome to the official repository for the NeurIPS 2025 paper **"GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data"**!

📄 **Read our paper**: [arXiv Preprint](https://arxiv.org/abs/2509.26016)

![GeoLink Framework](https://github.com/bailubin/GeoLink_NeurIPS2025/blob/main/framework.png)

## 📖 About This Project

GeoLink is a multimodal framework that empowers remote sensing foundation models by integrating OpenStreetMap (OSM) data in both pretraining stage and downstream tasks. 
Most traditional remote sensing models have focused solely on imagery, but GeoLink bridges the critical modality gap between RS and OSM data, enabling synergy for geospatial intelligence.

### 🚀 Key characteristics:
- **Finegrained fusion**: GeoLink can fusion RS and OSM data at entity-patch level, and obtain hybrid RS-OSM patch encodings for pixel-level tasks like semantic segmentation.
Example can be found in [**Using Multimodal GeoLink**](#using-multimodal-geolink).
- **Versatile Applications**: Supports both unimodal and multimodal fine-grained encodings for diverse tasks from land cover classification to urban function zone mapping.

Our extensive experiments demonstrate that incorporating OSM data significantly boosts RS image encoder performance and enhances adaptability to complex geographic scenarios.

## 🏋️‍♂️ Pre-trained Models

We provide two pre-trained models for different use cases:

### 1. Unimodal GeoLink(ViT) 🔍
- **Input**: Remote sensing images only. Pretrained by GeoLink and we extract the image encoder out for unimodal application. Using cases can be found in [**Using Unimodal GeoLink (ViT)**](#using-unimodal-vit)
- **Output**: Remote sensing encodings
- **Use case**: Standard RS interpretation tasks
- **Download**: [PKU Disk Link](https://disk.pku.edu.cn/link/AACFE5713FC4724F92B07DF68FF2FCB2D6) or [Google Drive](https://drive.google.com/file/d/12u0goOohBYHjlkKIs11bVeTYyscd2Z9g/view?usp=drive_link)

### 2. Multimodal GeoLink 🌐
- **Input**: Remote sensing images + OSM data. Using canses can be found in [**Using Multimodal GeoLink**](#using-multimodal-geolink).
- **Output**: Fused RS-OSM features
- **Use case**: Comprehensive geographic tasks requiring multimodal understanding
- **Download**: [PKU Disk Link](https://disk.pku.edu.cn/link/AA0C4453EBBA4D4C979FA737047555ACDA) or [Google Drive](https://drive.google.com/file/d/1bAeNurdrH9nEI7qzwNWyLBWBSZnfwj_9/view?usp=drive_link)

## ⚙️ Installation & Dependencies

First, install the required dependencies:

```bash
pip install torch==1.10.0+cu111 torchvision==0.11.0+cu111
pip install torch-cluster==1.6.0 torch_geometric==2.4.0 torch-scatter==2.0.9 torch-sparse==0.6.12 torch-spline-conv==1.2.1
pip install timm==0.6.12 transformers==4.46.3
```

## 🛠️ Quick Start
Here are some using examples of GeoLink. There are some example data in ./exapmle_data dir, including 
ufz_example to demonstrate how to use GeoLink to extract multimodal encodings and perform downstream semantic segmentation tasks, 
as well as osm2graph_example to illustrate how to convert OSM data into standard GeoLink input.
The following examples are from ./using_example.ipynb.

<a name="using-unimodal-vit"></a>
### Using Unimodal GeolInk (ViT)

If you only need remote sensing image processing capabilities, use GeoLink pretrained unimodal ViT model.
After download the checkpoint, you can use timm ViT to load it directly:

```python
import timm
import torch

# Load the pre-trained model
model_path = 'geolink_vit_large_patch16_224.pth'  # path to ViT checkpoint
checkpoint = torch.load(model_path, map_location='cpu')
config = checkpoint['model_config']

# Create and initialize the model
model = timm.create_model(
    config['architecture'], 
    pretrained=False, 
    num_classes=config['num_classes'],  # 0
    global_pool=config['global_pool']   # ''
)
model.load_state_dict(checkpoint['model_state_dict'])

# Using sinusoidal position embedding with no back-propagation
model.pos_embed.requires_grad = config['pos_embed_requires_grad']
# Then you can use it to generate embeddings or add the task-specific decoder for a given task.
```

<a name="using-multimodal-geolink"></a>
### Using Multimodal GeoLink

To use the full multimodal GeoLink model for generating RS-OSM fusion embeddings.
After download the checkpoint, you should load GeoLink model:

```python
import timm
import torch
from model import *
from dataset import *

'''
    You can use the following code to obtain the fusion embeddings.
    The default output embeedings are from the 7/11/15th layers of ViT-L and the hybrid RS-OSM embeddings.
    The hybrid RS-OSM embeddings are generated by the integration of features from last(23th) layer of ViT-L
    and OSM embeddings (detail structure are shown in the right of Fig 8b in the paper).
    Such embeddings can be applied to downstream tasks through a task-specific decoder, like UperNet for semantic segmentation.
'''

ckpt_fp = r'geolink_mutimodal_vit_large_patch16_224.pth' # path to multimodal geolink checkpoint dir
checkpoint = torch.load(ckpt_fp, map_location='cpu')
config = checkpoint['model_config']
img_encoder = timm.create_model(
    config['architecture'], 
    pretrained=False, 
    num_classes=config['num_classes'], 
    global_pool=config['global_pool']
)

osm_encoder = OSMHeteroGAT()
geolink = GeoLink(img_encoder, osm_encoder)
msg = geolink.load_state_dict(checkpoint['model_state_dict'])
print(msg)

multi_encoder = GeoLink_Fusion_Embedding(geolink, output_layers=[7, 11, 15, 23])
# 23 means the fusion embedding with shape of [256,14,14]

data_root = r'./example_data/ufz_example'
samples = os.listdir(os.path.join(data_root, 'graph'))

trainset = DownstreamDataset(data_root, file_names=samples)
trainloader = torch_geometric.data.DataLoader(trainset, batch_size=2,
                             pin_memory=True, num_workers=1, drop_last=True, shuffle=True)
train_iter = iter(trainloader)
# get a batch
img, graph = next(train_iter)
result = multi_encoder(img, graph) #list size [4, batch_size, embed_dim, 14, 14]
```

### Using Multimodal GeoLink for semantic segmentation tasks

To use the full multimodal GeoLink model for RS-OSM fusion:

```python
import timm
import torch
from model import *
from dataset import *

osm_encoder = OSMHeteroGAT()
geolink = GeoLink(img_encoder, osm_encoder)
msg = geolink.load_state_dict(checkpoint['model_state_dict'])
print(msg)

multi_encoder = GeoLink_Fusion_Embedding(geolink, output_layers=[7, 11, 15, 23])
model = SegUPerNet(encoder=multi_encoder, num_classes=9, channels=512) # using UperNet as the decoder

data_root = r'./example_data/ufz_example'
samples = os.listdir(os.path.join(data_root, 'graph'))

trainset = DownstreamDataset_UFZ(data_root, file_names=samples)
trainloader = torch_geometric.data.DataLoader(trainset, batch_size=2,
                             pin_memory=True, num_workers=1, drop_last=True, shuffle=True)
train_iter = iter(trainloader)
# get a batch
img, graph, label = next(train_iter)
result = model(img, graph) #list [batch_size, cls_num, 224, 224]
```
### Formatting your own OSM data to standard input

You can use your own OSM data and format to the GeoLink standard input. The OSM data corresponding to a remote sensing image area 
should be stored as vector types in three separate files: polygon, polyline, and point (at least one vector type must be present).
The OSM data can be Shapefile/Geojson/Geopackage/..., and it should has two main columns, OSM Tags and Geometry.
- **OSM Tags:** record the OSM tags in format 'tag:value' which joined by ';'
- **Geometry:** record the geometry information, and it can be POLYGON/LINESTRING/POINT

| Index | OSM Tags | Geometry |
|-------|----------|----------|
| 0 | `leisure:golf_course` | `POLYGON ((-104.88896 39.38870, -104.88824 39.3...))` |
| 1 | `landuse:grass;golf:tee` | `POLYGON ((-104.88803 39.38820, -104.88802 39.3...))` |
| 2 | `landuse:grass;golf:green` | `POLYGON ((-104.88855 39.38835, -104.88847 39.3...))` |
| 3 | `landuse:grass;golf:tee` | `POLYGON ((-104.88786 39.38859, -104.88786 39.3...))` |
| 4 | `building:building` | `POLYGON ((-104.88830 39.38886, -104.88842 39.3...))` |

The following code shows how to format the OSM data the GeoLink standard input. 
Please ensure your environment can connect to Hugging Face, as it requires using CLIP's BERT model for tag encoding (alternatively, you can download the parameters locally for loading).
```python
import geopandas as gpd
import pickle
import os
from prepare_data import *

example_path = r'./example_data/osm2graph_example/' # path to the osm example dir
tagw_path = r'all_tags30_frequency1.json'

osm_process = OSM2Graph(tagw_path, 'cuda:1')
name = 'a472140250_FI_21' # example name

# obatin the geographic boundingbox of the give RS image
# here, the boundingbox is saved in the meta file
# you can also calculate it from the original RS data like TIFF, tools are prepared in the prepare_data/utils.py
# note that the geographic coodinate system should be the same between RS and OSM data

with open(os.path.join(example_path, name, name+'.pickle'), 'rb') as file:
    meta_data = pickle.load(file)
bbox = meta_data['bbox']
north, south, east, west = bbox[3], bbox[1], bbox[2], bbox[0]
width = east - west
height = north - south

# OSM2Graph can handle situations where one or two vector types are missing
# If none of the three vector types are available, then just use RS image.
try:
    polygon_file = gpd.read_file(os.path.join(example_path, name, name + '_polygon.geojson'))
except:
    polygon_file = None
try:
    line_file = gpd.read_file(os.path.join(example_path, name, name + '_line.geojson'))
except:
    line_file = None
try:
    point_file = gpd.read_file(os.path.join(example_path, name, name + '_point.geojson'))
except:
    point_file = None

data = osm_process.process(polygon_file, line_file, point_file, north, south, east, west)
```

---

*This repository contains the official implementation of GeoLink. If you find our work helpful, please consider citing our paper!* 📚
```
@misc{bai2025geolinkempoweringremotesensing,
      title={GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data}, 
      author={Lubian Bai and Xiuyuan Zhang and Siqi Zhang and Zepeng Zhang and Haoyu Wang and Wei Qin and Shihong Du},
      year={2025},
      eprint={2509.26016},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2509.26016}, 
}
```

GeoLink_NeurIPS2025

GeoLink_NeurIPS2025 Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

GeoLink_NeurIPS2025

GeoLink_NeurIPS2025 Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README