datafusion-ducklake

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

datafusion-ducklake

(★ 162)

A Rust-native DuckLake engine built on Apache DataFusion

datafusion-ducklake Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DataFusion-DuckLake

[![crates.io](https://img.shields.io/crates/v/datafusion-ducklake.svg)](https://crates.io/crates/datafusion-ducklake)
[![docs.rs](https://img.shields.io/docsrs/datafusion-ducklake)](https://docs.rs/datafusion-ducklake)
[![CI](https://github.com/hotdata-dev/datafusion-ducklake/actions/workflows/ci.yml/badge.svg)](https://github.com/hotdata-dev/datafusion-ducklake/actions/workflows/ci.yml)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://raw.githubusercontent.com/datafusion-contrib/datafusion-ducklake/main/LICENSE)
[![Discord](https://img.shields.io/badge/Discord-DataFusion%2BDuckLake-5865F2?logo=discord&logoColor=white)](https://discord.com/channels/885562378132000778/1492192627666321452)

A [DataFusion](https://datafusion.apache.org/) extension for reading and writing
[DuckLake](https://ducklake.select) catalogs. DuckLake is an integrated data lake and
catalog format that stores metadata in a SQL database and data as Parquet files on disk
or object storage.

The goal of this project is to make DuckLake a first-class, Arrow-native lakehouse
format inside DataFusion.

This project is maintained by [Hotdata](https://www.hotdata.dev) with support from the
community. Come talk to us on [the Hotdata Discord](https://discord.gg/cdHczfxxBc).

- 📦 **crates.io:** <https://crates.io/crates/datafusion-ducklake>
- 📖 **API docs:** <https://docs.rs/datafusion-ducklake>
- 🧩 **Feature & backend support:** see [COMPATIBILITY.md](COMPATIBILITY.md)
- 💬 **Project chat:** [DataFusion+DuckLake Discord](https://discord.com/channels/885562378132000778/1492192627666321452) — development and usage discussion
- 🧡 **Meet the team:** [Hotdata Discord](https://discord.gg/cdHczfxxBc)

---

## Quick start

Add the crate:

```bash
cargo add datafusion-ducklake
```

The default build includes the statically bundled DuckDB catalog backend. Applications configure
their object store implementation directly. Other catalog backends and write support are opt‑in
via feature flags. See [COMPATIBILITY.md](COMPATIBILITY.md) for the full matrix.

```toml
# Cargo.toml — read PostgreSQL catalogs
# (to write them too, use features = ["write-postgres"])
[dependencies.datafusion-ducklake]
version = "0.7"
default-features = false
features = ["metadata-postgres", "tls-rustls-aws-lc-rs"]
```

`metadata-postgres`, `multicatalog-postgres`, and `write-postgres` do not select a TLS provider.
Plain local connections work without one. For TLS, also enable one of `tls-native-tls`,
`tls-rustls-aws-lc-rs`, or `tls-rustls-ring` on `datafusion-ducklake`.

The examples below also use `datafusion`, `object_store`, and `url` directly — add them
to your `[dependencies]` as well (this crate does not re-export them). The write example
additionally uses `sqlx` (with its `postgres` and `runtime-tokio` features) to open the
connection pool.

Run a query against an existing PostgreSQL catalog with the bundled example:

```bash
cargo run --example basic_query --features metadata-postgres -- \
  "postgresql://user:password@localhost:5432/database" "SELECT * FROM main.users"
```

Configure `object_store` directly for local, S3, or MinIO data files. Applications can instead
register another DataFusion `ObjectStore`, such as an OpenDAL‑backed connector.

DataFusion enables local filesystem support. S3 and MinIO applications must enable
`object_store/aws` themselves. That feature enables Ring through its HTTP client. If another
dependency enables AWS‑LC, install the intended process‑wide Rustls
[`CryptoProvider`](https://docs.rs/rustls/0.23/rustls/crypto/struct.CryptoProvider.html) before
creating TLS clients.

(The example also accepts DuckDB, SQLite, and MySQL connection strings with the matching
`metadata-*` feature — see [COMPATIBILITY.md](COMPATIBILITY.md).)

---

## Reading a catalog

```rust,ignore
use datafusion::execution::runtime_env::RuntimeEnv;
use datafusion::prelude::*;
use datafusion_ducklake::{DuckLakeCatalog, PostgresMetadataProvider};
use object_store::ObjectStore;
use object_store::aws::AmazonS3Builder;
use std::sync::Arc;
use url::Url;

// (inside an async fn)
// Read metadata from a PostgreSQL catalog
let provider = PostgresMetadataProvider::new("postgresql://user:pass@localhost:5432/db").await?;

// Register object stores for any non-local data (S3 / MinIO)
let runtime = Arc::new(RuntimeEnv::default());
let s3: Arc<dyn ObjectStore> = Arc::new(
    AmazonS3Builder::new()
        .with_endpoint("http://localhost:9000") // MinIO endpoint
        .with_bucket_name("ducklake-data")
        .with_access_key_id("minioadmin")
        .with_secret_access_key("minioadmin")
        .with_region("us-west-2") // any region works for MinIO
        .with_allow_http(true)    // required for http:// endpoints
        .build()?,
);
runtime.register_object_store(&Url::parse("s3://ducklake-data/")?, s3);

let catalog = DuckLakeCatalog::new(provider)?;
let ctx = SessionContext::new_with_config_rt(
    SessionConfig::new().with_default_catalog_and_schema("ducklake", "main"),
    runtime,
);
ctx.register_catalog("ducklake", Arc::new(catalog));

let df = ctx.sql("SELECT * FROM ducklake.main.my_table").await?;
df.show().await?;
```

---

## Writing a catalog

PostgreSQL has two writers, both behind the `write-postgres` feature:

- **`PostgresSingleCatalogMetadataWriter`** — the **standard, spec-compliant**
  single-catalog layout. Same catalog shape as the SQLite and MySQL writers, so the
  catalog is readable (and writable) by other DuckLake implementations including
  DuckDB's `ducklake` extension. SQL `CREATE TABLE AS SELECT` and `INSERT INTO` both
  work. **Prefer this one.**
- **`PostgresMetadataWriter`** — the **experimental multi-catalog layout** described in
  [its own section](#multi-catalog-postgresql-experimental), for hosting many catalogs
  in one database. Library-specific, not in the DuckLake spec, and no CTAS.

```rust,ignore
use datafusion::prelude::*;
use datafusion_ducklake::metadata_writer::MetadataWriter; // set_data_path
use datafusion_ducklake::{
    DuckLakeCatalog, PostgresMetadataProvider, PostgresSingleCatalogMetadataWriter,
};
use std::sync::Arc;

// Bootstrap the standard DuckLake tables and point the catalog at its data root
let writer = PostgresSingleCatalogMetadataWriter::new_with_init(
    "postgresql://user:pass@localhost:5432/db",
).await?;
writer.set_data_path("/abs/path/to/data")?;

// CTAS and INSERT both work on this path
let provider = PostgresMetadataProvider::new("postgresql://user:pass@localhost:5432/db").await?;
let catalog = DuckLakeCatalog::with_writer(Arc::new(provider), Arc::new(writer))?;
let ctx = SessionContext::new();
ctx.register_catalog("ducklake", Arc::new(catalog));
ctx.sql("CREATE TABLE ducklake.main.events AS SELECT 1 AS id").await?.collect().await?;
```

The multi-catalog path instead looks like this — tables are created through the writer
API (no CTAS), then appended to with SQL:

```rust,ignore
use datafusion::prelude::*;
use datafusion_ducklake::metadata_writer::MetadataWriter; // set_data_path
use datafusion_ducklake::{
    DuckLakeCatalog, DuckLakeTableWriter, MulticatalogManager, MulticatalogProvider,
    PostgresMetadataWriter, initialize_multicatalog_schema,
};
use sqlx::postgres::PgPoolOptions;
use std::sync::Arc;

let pool = PgPoolOptions::new().connect("postgresql://user:pass@localhost:5432/db").await?;

// One-time: bootstrap the multi-catalog tables, then create a named catalog
initialize_multicatalog_schema(&pool).await?;
let catalog_id = MulticatalogManager::new(pool.clone()).create_catalog("my_catalog").await?;

// Create a table by writing the first batch through the table writer
let writer = Arc::new(PostgresMetadataWriter::with_pool(pool.clone(), catalog_id).await?);
writer.set_data_path("/abs/path/to/data")?;
let object_store: Arc<dyn object_store::ObjectStore> =
    Arc::new(object_store::local::LocalFileSystem::new());
let table_writer = DuckLakeTableWriter::new(writer.clone(), object_store)?;
table_writer.write_table("public", "events", &[batch]).await?; // `batch` is your RecordBatch

// Now append with SQL, reading the same catalog back through MulticatalogProvider
let provider = MulticatalogProvider::with_pool(pool.clone(), "my_catalog").await?;
let catalog = DuckLakeCatalog::with_writer(Arc::new(provider), writer)?;
let ctx = SessionContext::new();
ctx.register_catalog("ducklake", Arc::new(catalog));
ctx.sql("INSERT INTO ducklake.public.events VALUES (1, 'a')").await?.collect().await?;
ctx.sql("SELECT count(*) FROM ducklake.public.events").await?.show().await?;
```

Writer output is configurable (Parquet compression, row-group sizing by row count and
byte size). See [`DuckLakeTableWriter`](https://docs.rs/datafusion-ducklake) for the
writer options.

> Writing to a **standard, single-catalog** DuckLake store (the spec-compliant layout) is
> supported today for **SQLite** via `SqliteMetadataWriter` (feature `write-sqlite`) and for
> **PostgreSQL** via `PostgresSingleCatalogMetadataWriter` (feature `write-postgres`), where
> SQL `CREATE TABLE AS SELECT` and `INSERT INTO` both work. See
> [`tests/it/sql_write_tests.rs`](tests/it/sql_write_tests.rs) and
> [`tests/it/postgres_single_catalog_write_tests.rs`](tests/it/postgres_single_catalog_write_tests.rs).

---

## Partitioning

Partition a table by one or more columns (optionally through a transform) so that queries
filtering on a partition column skip whole files:

```rust
// Declare the partition scheme *before* loading data, then INSERT as usual.
execute_ducklake_sql(
    &ctx,
    &catalog,
    "ALTER TABLE ducklake.main.sales SET PARTITIONED BY (region, year(sale_date))",
)
.await?;
```

Writes split rows into one Parquet file per partition value; reads then prune non-matching
files automatically. Supported transforms are `identity` (the raw value) and
`year`/`month`/`day`/`hour`; pruning currently applies to `identity` and `year`
(`month`/`day`/`hour` are recorded but not yet used to skip files). Partitioned **writes**
work on every writable backend — SQL `INSERT`/`UPDATE`, the low-level write entry points,
the streaming session, compaction, and promote all honour the live spec — and **read +
pruning** work on all backends. `RESET PARTITIONED BY` turns it off. See
[COMPATIBILITY.md](COMPATIBILITY.md) and
[`tests/it/partition_write_tests.rs`](tests/it/partition_write_tests.rs).

---

## Sort order

Order the rows inside each written file so that per-file statistics stay tight and
range-filtered scans skip more:

```rust
execute_ducklake_sql(
    &ctx,
    &catalog,
    "ALTER TABLE ducklake.main.sales SET SORTED BY (sale_date DESC NULLS LAST)",
)
.await?;
```

The spec is recorded in the catalog (`ducklake_sort_info` / `ducklake_sort_expression`) and
applied on insert, to `UPDATE` rewrites, and to compaction output; rows are sorted before the
partition split, so each per-partition file stays a sorted subsequence. Bare-column keys are
produced; other expressions are tolerated on read but never produced. `RESET SORTED BY`
turns it off.

---

## Multi-catalog (PostgreSQL, experimental)

A single PostgreSQL metadata store can host **multiple independent DuckLake catalogs** —
useful for multi-tenant deployments or keeping many logical lakehouses in one database.

> ⚠️ **Experimental and library-specific.** This multi-catalog layout is **not part of the
> DuckLake specification** and is not (yet) supported or accepted upstream. Catalogs
> written this way are read back only through this crate's `MulticatalogProvider` — they
> are **not** interchangeable with a standard single-catalog DuckLake store. The API and
> on-disk/in-catalog layout may change, so treat it as a preview. PostgreSQL writes no
> longer require this path — use `PostgresSingleCatalogMetadataWriter` for the
> spec-compliant layout.

- **Create and manage** catalogs with `MulticatalogManager` (feature `write-postgres`):
  `initialize_multicatalog_schema` bootstraps the shared tables, then `create_catalog`,
  and `drop_table_in_catalog` manage their contents.
- **Read** a specific catalog with `MulticatalogProvider::with_pool(pool, "name")`
  (feature `multicatalog-postgres`), which plugs into a `DuckLakeCatalog` like any other
  metadata provider.

See [`examples/multicatalog_write.rs`](examples/multicatalog_write.rs) for an end-to-end
walkthrough (bootstrap → create catalogs → write → read back).

---

## Maintenance

The `maintenance` API handles lakehouse upkeep from Rust: expiring old snapshots,
cleaning up superseded files, and reclaiming orphaned files. The concrete entry points
are backend-gated (`write-sqlite` / `write-postgres`). `DROP TABLE` is available through
`MetadataWriter`. See
[`examples/maintenance_demo.rs`](examples/maintenance_demo.rs) and
[`examples/orphan_cleanup_demo.rs`](examples/orphan_cleanup_demo.rs).

### Compaction

Two explicit, triggered operations on `DuckLakeTable` rewrite a table's data files
into a better physical layout without changing its logical rows:

- `merge_adjacent_files(state, MergeOptions)` coalesces several small files (of the
  same schema version) into fewer larger ones. A merged file spanning multiple
  origin snapshots is written as a DuckLake *partial data file* (preserving each
  row's original rowid and origin snapshot), so time travel and change feeds are
  unaffected.
- `rewrite_data_files(state, RewriteOptions)` rewrites a file whose deleted fraction
  exceeds a threshold (default `0.95`), dropping its deleted rows.

Both commit atomically in one snapshot and coexist with concurrent appends; superseded
files are scheduled for deletion and reclaimed later by `cleanup_old_files`. See
[`examples/compaction_demo.rs`](examples/compaction_demo.rs).

---

## Compatibility

For the full breakdown of catalog backends, object stores, types, capabilities, and
current limitations, see **[COMPATIBILITY.md](COMPATIBILITY.md)**.

A few highlights worth knowing up front:

- Reads work on DuckDB, SQLite, PostgreSQL, and MySQL; **writes are SQLite/PostgreSQL only**.
- Object stores: local filesystem and S3-compatible (S3, MinIO).
- Snapshots can be selected through `DuckLakeCatalog` (by id or timestamp) or per query with
  `ducklake_table_at`; DataFusion does not support `AS OF` syntax.
- Table partitioning: read + file pruning on all backends; partitioned writes on every
  writable backend.
- Data inlined by DuckDB's ducklake extension is **not read** — see COMPATIBILITY.md for
  the `COUNT(*)` undercount caveat and how to avoid it.

---

## Project status

This project is in alpha and evolving alongside DataFusion and DuckLake. APIs may change
as core abstractions are refined. See [CHANGELOG.md](CHANGELOG.md) for release history.
Feedback, issues, and contributions are welcome.

datafusion-ducklake

datafusion-ducklake Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

datafusion-ducklake

datafusion-ducklake Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README