GCM-downloader

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #claude-code #codex #cli #developer-tools #react #windows

★ 26 Open GitHub ↗

CMIP6 interactive data downloader — OPeNDAP subsetting, multi-server failover, 61 models, 197 countries

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# CMIP6 Interactive Data Downloader

**Author.** Thanh-Nhan-Duc Tran ([ductran27](https://github.com/ductran27))

A command-line tool that downloads CMIP6 climate model output from the Earth System Grid Federation (ESGF). It supports OPeNDAP spatial subsetting, multi-server failover, subprocess isolation, and parallel downloads.

Downloading CMIP6 data in bulk through the ESGF web portal is impractical when you need multiple models, scenarios, and variables. Files sit on servers worldwide, many of which are intermittently unavailable or block certain regions. This tool automates the process and works around server failures without user intervention.

---

## Features

An interactive wizard walks you through 8 steps, which are experiments, variables, frequency, grid label, models, ensemble variants, time range, and region of interest. The download engine then handles the transfer.

**OPeNDAP server-side subsetting.** The tool requests only the spatial and temporal subset it needs, instead of downloading full global fields. For Vietnam, a daily precipitation file shrinks from approximately 700 MB to 400 KB, which is a 1750-fold reduction. When OPeNDAP fails, the tool falls back to HTTP with resume support and crops the file locally.

**Subprocess isolation.** Each OPeNDAP request runs in its own process, because the netCDF4 C library can segfault under concurrent connections. When a subprocess crashes, the main process detects the failure and falls back to HTTP. The session continues without interruption.

**Multi-server failover with circuit breaker.** For every file, the tool collects download URLs from all ESGF replica nodes. When one server is unreachable, the next is tried automatically. Hosts that fail three or more times are disabled for the rest of the session, with separate tracking for OPeNDAP and HTTP.

**Filename-based time filter.** CMIP6 filenames encode the date range they cover. When you select 1980 to 2014 from a historical run that spans 1850 to 2014, the tool skips the approximately 130 pre-1980 files without any network requests.

**Curated catalog.** The tool ships with a verified catalog of 19 experiments, 46 variables, and 61 models, together with institution, resolution, and GCM/ESM classification. Live ESGF queries supplement the catalog when a model or variant is not in it. Spatial subsetting supports 197 countries and 25 predefined regions, with a 1-degree bounding box buffer that accounts for coarse model grids.

**Rich terminal interface.** Every wizard step renders as a bordered panel with a live selection summary, so you can see your choices without scrolling back. The download phase shows an overall progress bar together with a recent-activity log tail, i.e., the last few status lines, inside a single live frame.

---

## Installation

**Python 3.10 or later** is required.

```bash
git clone https://github.com/ductran27/GCM-downloader.git
cd GCM-downloader
pip install -r requirements.txt
```

The tool uses these dependencies.

| Package  | Purpose                                             |
|----------|-----------------------------------------------------|
| numpy    | Array operations                                    |
| xarray   | NetCDF reading and coordinate handling              |
| netCDF4  | OPeNDAP access and NetCDF I/O backend               |
| dask     | Lazy loading for large datasets                     |
| requests | HTTP downloads                                      |
| rich     | Terminal UI (panels, tables, live progress display) |

No ESGF account or API key is needed. All data access uses the public ESGF search API and the public OPeNDAP and HTTP endpoints.

---

## Usage

```bash
python download_cmip6.py
```

The wizard walks you through 8 steps.

1. **Experiments**, e.g., historical, ssp126, ssp245, ssp370, ssp585 (19 available)
2. **Variables**, e.g., tas, pr, hurs, rsds, sfcWind (46 available across atmosphere, ocean, land, and sea ice)
3. **Frequency**, i.e., daily, monthly, 3-hourly, 6-hourly, or annual
4. **Grid label**, i.e., native (gn), regridded (gr), or any available
5. **Models**, including 61 curated models plus any additional models found on ESGF
6. **Variants**, i.e., ensemble members such as r1i1p1f1, queried live per model
7. **Time range**, an optional year filter, e.g., 1980-2014
8. **Region of interest**, that is, a country, a predefined region, a custom bounding box, or global

Type `back` at any prompt to return to the previous step. Type `quit` to exit.

### Output structure

```
data/
  Vietnam/
    EC-Earth3-Veg/
      historical/
        pr/
          pr_day_EC-Earth3-Veg_historical_r1i1p1f1_gr_19800101-19801231.nc
          pr_day_EC-Earth3-Veg_historical_r1i1p1f1_gr_19810101-19811231.nc
          ...
      ssp245/
        pr/
          ...
    manifest.json
    download.log
```

### Configuration

Edit the constants at the top of `download_cmip6.py`.

| Constant                      | Default  | Description                                      |
|-------------------------------|----------|--------------------------------------------------|
| `OUTPUT_DIR`                  | `./data` | Root directory for downloaded files              |
| `MAX_CONCURRENT`              | 3        | Parallel download threads                        |
| `MAX_RETRIES`                 | 5        | HTTP retry attempts per URL                      |
| `OPENDAP_MAX_RETRIES`         | 2        | OPeNDAP retry attempts per URL                   |
| `OPENDAP_TIMEOUT`             | 300      | Seconds before an OPeNDAP subprocess is killed   |
| `OPENDAP_HOST_FAIL_THRESHOLD` | 3        | Host failures before the circuit breaker trips   |

---

## File Descriptions

| File                | Description                                                           |
|---------------------|-----------------------------------------------------------------------|
| `download_cmip6.py` | Main downloader, including wizard entry, ESGF search, and the OPeNDAP and HTTP engine |
| `ui.py`             | Rich-based terminal interface, prompt wrappers, and download progress display |
| `cmip6_catalog.py`  | Curated catalog of 19 experiments, 46 variables, and 61 models        |
| `countries.py`      | 197 country bounding boxes and 25 predefined regions                  |

---

## Data Conventions

- **Precipitation (pr)** is in kg m-2 s-1. Multiply by 86400 to get mm/day.
- **Temperature (tas, tasmin, tasmax)** is in Kelvin. Subtract 273.15 to get Celsius.
- **Longitude** is converted to the -180 to 180 convention after download, regardless of the model native grid.
- **Calendar** varies by model. Most use standard Gregorian. Some models, e.g., CESM2, E3SM, and GISS, use `noleap`, which is a 365-day calendar without February 29. Check `ds.time.encoding['calendar']` when this matters.
- **Floating-point noise** in regridded precipitation fields may produce tiny negatives on the order of 10^-21. Clip with `ds['pr'].clip(min=0)`.

---

## Troubleshooting

**Segmentation fault during OPeNDAP downloads.**
Handled automatically. OPeNDAP downloads run in isolated subprocesses. When one crashes, the main process detects the failure and falls back to HTTP.

**All downloads fail or a server appears blocked.**
The tool collects URLs from multiple ESGF replica nodes. When one server blocks your region, downloads route through the alternative mirrors. When every server is unreachable, ESGF may be experiencing an outage. Try again later.

**Slow downloads.**
Try the regridded grid (`gr`) instead of the native grid (`gn`), which usually produces fewer and smaller files. Use spatial subsetting by selecting a country instead of global. Reduce `MAX_CONCURRENT` when your connection is unstable.

**Files outside the time range are being downloaded.**
The filename date range is parsed before any network request, so files entirely outside your selection are skipped. When unwanted files still appear, they contain partial overlap with your time range and are cropped after download.

---

## How It Works

The downloader queries the ESGF Solr search API for files matching your selections. It groups results by filename and merges URLs from all replica servers, so every file has multiple download sources. For each file, the engine first tries OPeNDAP subsetting in an isolated subprocess. When that fails, e.g., from a timeout, a segfault, or a structural incompatibility such as a curvilinear grid, the engine falls back to a full HTTP download followed by local cropping. A thread-safe circuit breaker tracks per-host failures and disables unreliable servers after three strikes, with separate counters for each protocol. Three download threads run in parallel by default.

---

## Citation

If you use this tool in your research, please cite it as follows.

> Tran, T. N. D. (2026). CMIP6 Interactive Data Downloader (Version 1.0) [Software]. GitHub. https://github.com/ductran27/GCM-downloader

BibTeX entry.

```bibtex
@software{tran2026cmip6downloader,
  author    = {Tran, Thanh-Nhan-Duc},
  title     = {{CMIP6 Interactive Data Downloader}},
  year      = {2026},
  version   = {1.0},
  url       = {https://github.com/ductran27/GCM-downloader},
  note      = {Command-line tool for downloading CMIP6 climate model output from ESGF}
}
```

---

## Acknowledgements

This tool accesses CMIP6 climate model output distributed by the **Earth System Grid Federation (ESGF)**. We acknowledge ESGF, its member institutions, and the modeling centers that produce and share these data. The publications listed below define the CMIP6 framework, its experimental protocols, and the data standards that this tool builds upon.

- Eyring, V., Bony, S., Meehl, G. A., et al. (2016). Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization. *Geosci. Model Dev.*, 9, 1937-1958. https://doi.org/10.5194/gmd-9-1937-2016

- O'Neill, B. C., Tebaldi, C., van Vuuren, D. P., et al. (2016). The Scenario Model Intercomparison Project (ScenarioMIP) for CMIP6. *Geosci. Model Dev.*, 9, 3461-3482. https://doi.org/10.5194/gmd-9-3461-2016

- Tebaldi, C., Debeire, K., Eyring, V., et al. (2021). Climate model projections from the Scenario Model Intercomparison Project (ScenarioMIP) of CMIP6. *Earth Syst. Dynam.*, 12, 253-293. https://doi.org/10.5194/esd-12-253-2021

- Kravitz, B., Robock, A., Tilmes, S., et al. (2015). The Geoengineering Model Intercomparison Project Phase 6 (GeoMIP6) simulation design and preliminary results. *Geosci. Model Dev.*, 8, 3379-3392. https://doi.org/10.5194/gmd-8-3379-2015

- Gillett, N. P., Shiogama, H., Funke, B., et al. (2016). The Detection and Attribution Model Intercomparison Project (DAMIP v1.0) contribution to CMIP6. *Geosci. Model Dev.*, 9, 3685-3697. https://doi.org/10.5194/gmd-9-3685-2016

- Eyring, V., Bock, L., Lauer, A., et al. (2020). Earth System Model Evaluation Tool (ESMValTool) v2.0, an extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP. *Geosci. Model Dev.*, 13, 3383-3438. https://doi.org/10.5194/gmd-13-3383-2020

- Taylor, K. E., Juckes, M., Balaji, V., et al. (2025). CMIP6 Model Output Metadata Requirements, Data Reference Syntax (DRS) and Controlled Vocabularies (CVs), Version 6.2.8. https://doi.org/10.5281/zenodo.15670624

- Balaji, V., Taylor, K. E., Juckes, M., et al. (2018). Requirements for a global data infrastructure in support of CMIP6. *Geosci. Model Dev.*, 11, 3659-3680. https://doi.org/10.5194/gmd-11-3659-2018

---

## License

The CMIP6 data accessed through this tool is subject to the [CMIP6 Terms of Use](https://pcmdi.llnl.gov/CMIP6/TermsOfUse/TermsOfUse6-2.html). Please cite the relevant modeling groups and ESGF when using downloaded data in publications.

GCM-downloader

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README