ESMcat

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #claude-code #codex #cli #developer-tools #react #windows

★ 21 Open GitHub ↗

Earth System Modelling Catalogue + Loader for CMIP6

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# ESMcat

[![All Contributors](https://img.shields.io/badge/all_contributors-4-orange.svg?style=flat-square)](https://raw.githubusercontent.com/scotthosking/ESMcat/main/)

ESMcat is a Python package for working with large climate model datasets. It provides a catalogue system for indexing and filtering CMIP6 datasets.

---

## Requirements

- Python 3.7+
- [pandas](https://pandas.pydata.org/)
- [NumPy](https://numpy.org/)
- [pyarrow](https://arrow.apache.org/docs/python/)
- [Xarray](https://docs.xarray.dev/en/stable/)

---

## Installation

Install directly from GitHub:

```bash
pip install git+https://github.com/scotthosking/ESMcat.git
```

Or clone and install in editable mode for development:

```bash
git clone https://github.com/scotthosking/ESMcat.git
cd ESMcat
pip install -e .
```

---

## Machine configuration

ESMcat uses a config file at `~/.esmcat/config.json` to know which machine you are on, and therefore which dataset paths and catalogue files to use.

Set your machine on first use:

```python
import esmcat as ecat
ecat.set_config('jasmin')
```

This writes `{"machine": "jasmin"}` to `~/.esmcat/config.json`. ESMcat will then load `datasets_jasmin.json` from the package for dataset root paths and directory structures.

To check your current config:

```python
ecat.get_config()
```

To add support for a new machine, create a `datasets_{machine}.json` file in the `esmcat/` package directory following the same structure as `datasets_jasmin.json`.

---

## Catalogue files

ESMcat uses pre-built catalogue files (Parquet format) stored in `~/.esmcat/`. These index the files available on your system for each dataset.

Bundled catalogue files are included in the package under `esmcat/catalogues/` and are copied to `~/.esmcat/` automatically on first use. Currently bundled:

| Dataset | Coverage |
|---------|----------|
| `cmip6` | CMIP and ScenarioMIP activities |

To rebuild a catalogue from scratch (e.g. after new data has been added to the archive):

```python
ecat.catalogue(dataset='cmip6', refresh=True)
```

Existing CSV catalogues are automatically migrated to Parquet on first use.

---

## Usage

### Filter the CMIP6 catalogue

```python
import esmcat as ecat

df = ecat.catalogue(dataset='cmip6',
                  Experiment='historical',
                  Var=['tas', 'pr'],
                  CMOR='Amon')

print(df.head())
```

Use `CMOR` to select frequency and realm (e.g. `Amon` for monthly atmosphere, `day` for daily).

### Available columns

| Column | Description | Example values |
|--------|-------------|----------------|
| `MIP` | CMIP6 activity | `CMIP`, `ScenarioMIP` |
| `Centre` | Modelling centre | `MOHC`, `CNRM-CERFACS` |
| `Model` | Model name | `HadGEM3-GC31-LL`, `CNRM-ESM2-1` |
| `Experiment` | Experiment ID | `historical`, `ssp245`, `ssp585` |
| `RunID` | Ensemble member | `r1i1p1f1`, `r2i1p1f2` |
| `CMOR` | CMOR table (encodes frequency and realm) | `Amon`, `Omon`, `day`, `fx` |
| `Var` | Variable name | `tas`, `pr`, `tos` |
| `Grid` | Grid label | `gn` (native), `gr` (regridded) |
| `Version` | Data version | `v20190621` |
| `StartDate` | Start date of files (YYYYMMDD) | `19500101` |
| `EndDate` | End date of files (YYYYMMDD) | `21001231` |
| `Path` | Relative path to data directory | |
| `DataFiles` | Semicolon-separated list of filenames | |

### Open a dataset

Pass a single-row catalogue entry to `ecat.open_dataset()` to load it as an Xarray Dataset. Multiple files (e.g. a variable split across decades) are combined automatically via `xarray.open_mfdataset`.

```python
import esmcat as ecat

catlg = ecat.catalogue(dataset='cmip6',
                     Experiment='historical',
                     Var='tas',
                     CMOR='Amon',
                     Model='HadGEM3-GC31-LL',
                     RunID='r1i1p1f3')

ds = ecat.open_dataset(catlg.iloc[0])
print(ds)
```

To loop over multiple variables:

```python
for _, row in catlg.iterrows():
    ds = ecat.open_dataset(row)
```

`ecat.open_dataset` requires [Xarray](https://docs.xarray.dev/en/stable/) and access to the underlying data files.

### Read everything (bypass default filters)

```python
df = ecat.catalogue(dataset='cmip6', read_everything=True)
```

---

## Adding or editing datasets

Dataset configurations (root paths, directory structures, filename structures) are defined in `datasets_{machine}.json`. To add support for a new dataset on an existing machine, add an entry to the relevant JSON file following the same structure as the existing ones.

---

## Contributors ✨

Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):

<table>
 <tbody>
 <tr>
 <td align="center" valign="top" width="14.28%"><a href="https://scotthosking.com"><img src="https://avatars.githubusercontent.com/u/10783052?v=4?s=100" width="100px;" alt="Scott Hosking"/> Scott Hosking</a> <a href="https://github.com/scotthosking/ESMcat/commits?author=scotthosking" title="Code">💻</a></td>
 <td align="center" valign="top" width="14.28%"><a href="https://github.com/TomBracegirdle"><img src="https://avatars.githubusercontent.com/u/18678126?v=4?s=100" width="100px;" alt="TomBracegirdle"/> TomBracegirdle</a> <a href="https://github.com/scotthosking/ESMcat/commits?author=TomBracegirdle" title="Code">💻</a></td>
 <td align="center" valign="top" width="14.28%"><a href="https://github.com/TonyP-BAS"><img src="https://avatars.githubusercontent.com/u/113535863?v=4?s=100" width="100px;" alt="Tony Phillips"/> Tony Phillips</a> <a href="https://github.com/scotthosking/ESMcat/commits?author=TonyP-BAS" title="Code">💻</a></td>
 <td align="center" valign="top" width="14.28%"><a href="https://github.com/C-H-Simpson"><img src="https://avatars.githubusercontent.com/u/20053498?v=4?s=100" width="100px;" alt="Charles H. Simpson"/> Charles H. Simpson</a> <a href="https://github.com/scotthosking/ESMcat/commits?author=C-H-Simpson" title="Code">💻</a></td>
 </tr>
 </tbody>
</table>

This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!

ESMcat

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README