gaggle

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 17 Open GitHub ↗

A DuckDB extension for working with Kaggle datasets

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h2>Gaggle</h2>

[![Tests](https://img.shields.io/github/actions/workflow/status/CogitatorTech/gaggle/tests.yml?label=tests&style=flat&labelColor=282c34&logo=github)](https://github.com/CogitatorTech/gaggle/actions/workflows/tests.yml)
[![Examples](https://img.shields.io/badge/examples-view-green?style=flat&labelColor=282c34&logo=github)](https://github.com/CogitatorTech/gaggle/tree/main/docs/examples)
[![Docs](https://img.shields.io/badge/docs-read-blue?style=flat&labelColor=282c34&logo=read-the-docs)](https://github.com/CogitatorTech/gaggle/tree/main/docs)
[![License](https://img.shields.io/badge/license-MIT%2FApache--2.0-007ec6?style=flat&labelColor=282c34&logo=open-source-initiative)](https://github.com/CogitatorTech/gaggle)

Access and query Kaggle datasets from DuckDB

</div>

---

Gaggle is a DuckDB extension that allows you to work with [Kaggle datasets](https://www.kaggle.com/datasets)
directly in SQL queries, as if they were DuckDB tables.
It is written in Rust and uses the [Kaggle API](https://www.kaggle.com/docs/api)
to search, download, and manage the datasets.

Kaggle hosts a large collection of very useful datasets for data science and machine learning.
Accessing these datasets typically involves manually downloading a dataset (as a ZIP file),
extracting it, loading the files in the dataset into your data science environment, and managing storage and dataset
updates, etc.
This workflow can quickly become complex, especially when working with multiple datasets or when datasets are updated
frequently.
Gaggle tries to help simplify this process by hiding the complexity and letting you work with datasets directly inside
DuckDB that allows you to run fast analytical queries on the data.

In essence, Gaggle makes DuckDB into a SQL-enabled frontend for Kaggle datasets.

### Features

- Provides a simple API to interact with Kaggle datasets from DuckDB
- Allows you to search, download, and read datasets from Kaggle
- Supports datasets that contain CSV, Parquet, JSON, and XLSX files
- Supports dataset updates and versioning
- Configurable and has built-in caching support to avoid re-downloading
- Thread-safe, fast, and has a low memory footprint

See the [ROADMAP.md](ROADMAP.md) for the list of implemented and planned features.

> [!IMPORTANT]
> Gaggle is in early development, so bugs and breaking changes are expected.
> Please use the [issues page](https://github.com/CogitatorTech/gaggle/issues) to report bugs or request features.

---

### Quickstart

#### Install from Community Extensions Repository

You can install and load Gaggle from
the [DuckDB community extensions](https://duckdb.org/community_extensions/extensions/gaggle) repository by running the
following SQL commands in the DuckDB shell:

```sql
install gaggle from community;
load gaggle;
```

#### Build from Source

Alternatively, you can build Gaggle from source and use it by following these steps:

1. Clone the repository and build the Gaggle extension from source:

```bash
git clone --recursive https://github.com/CogitatorTech/gaggle.git
cd gaggle

# This might take a while to run
make release
```

2. Start DuckDB shell (with Gaggle statically linked to it):

```bash
./build/release/duckdb
```

> [!NOTE]
> After building from source, the Gaggle binary will be `build/release/extension/gaggle/gaggle.duckdb_extension`.
> You can load it using the `load 'build/release/extension/gaggle/gaggle.duckdb_extension';` in the DuckDB shell.
> Note that the extension binary will only work with the DuckDB version that it was built against.
> You can download the pre-built binaries from the [releases page](https://github.com/CogitatorTech/gaggle/releases) for
> your platform.

#### Trying Gaggle

```sql
-- Get extension version
select gaggle_version();

-- List files in the dataset (recursively)
-- (Note that if the dataset is not downloaded, it will be downloaded and cached)
select *
from gaggle_ls('habedi/flickr-8k-dataset-clean', true) limit 5;

-- Read a Parquet file from the local cache using a prepared statement
prepare rp as select * from read_parquet(?) limit 10;
execute rp(gaggle_file_path('habedi/flickr-8k-dataset-clean', 'flickr8k.parquet'));

-- Alternatively, we can use a replacement scan to read directly via `kaggle:` prefix
select count(*)
from 'kaggle:habedi/flickr-8k-dataset-clean/flickr8k.parquet';

-- Then, we check cache info
select gaggle_cache_info();

-- Check if cached dataset is current (is the newest version?)
select gaggle_is_current('habedi/flickr-8k-dataset-clean');
```

[![Simple Demo 1](https://asciinema.org/a/do6g8xv1G5tkRc4e3bExbNYwZ.svg)](https://asciinema.org/a/do6g8xv1G5tkRc4e3bExbNYwZ)

---

### Documentation

Check out the [docs](docs/README.md) directory for the API documentation, how to build Gaggle from source, and more.

#### Examples

Check out the [examples](docs/examples) directory for SQL scripts that show how to use Gaggle.

---

### Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to make a contribution.

### License

Gaggle is available under either of the following licenses:

* MIT License ([LICENSE-MIT](LICENSE-MIT))
* Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE))

### Acknowledgements

* The logo is from [here](https://www.svgrepo.com/svg/322445/goose) with some modifications.

gaggle

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README