duckdb-delta - GitLite

KO | EN

gitlite — search

#python #react #llm #golang #go #machine-learning #rust #deep-learning #nodejs #awesome #cli #awesome-list #vue #ios

duckdb-delta

(★ 228)

DuckDB extension for Delta Lake

File Explorer

Download Latest Version (.zip)

- - micro.csv
  - tpcds_sf1_local.csv
  - tpch_sf1_local.csv
- - CloudTesting.yml
  - LocalTesting.yml
  - MainDistributionPipeline.yml
- benchmark.Makefile
- README.md
- - generate_test_data.py
- - generate_delta_kernel_ffi_header
  - prefix.inc
  - suffix.inc
- apply_patches.py
- create_minio_credential_file.sh
- env_minio
- generate_golden_table_tests.py
- idempotent_writes_demo.py
- plot.py
- unwrap_golden_tables.sh
- upload_test_files_to_azurite.sh
- upload_test_files_to_minio.sh
- verify_column_mapping_roundtrip.py
- - delta_domain_metadata.cpp
  - delta_metadata_scan.cpp
  - expression_functions.cpp
  - metadata_functions.cpp
  - util_functions.cpp
- - delta_extension.hpp
  - delta_functions.hpp
  - delta_kernel_ffi.hpp
  - delta_log_types.hpp
  - delta_macros.hpp
  - delta_schema_builder.hpp
  - delta_time_travel.hpp
  - delta_utils.hpp
  - path_utils.hpp
- - delta_catalog.cpp
  - delta_insert.cpp
  - delta_schema_entry.cpp
  - delta_table_entry.cpp
  - delta_transaction.cpp
  - delta_transaction_manager.cpp
- delta_extension.cpp
- delta_functions.cpp
- delta_log_types.cpp
- delta_macros.cpp
- delta_schema_builder.cpp
- delta_utils.cpp
- path_utils.cpp
- README.md
.clang-format
.clang-tidy
.clangd
.editorconfig
.gitignore
.gitmodules
CMakeLists.txt
duckdb
extension-ci-tools
extension_config.cmake
LICENSE
Makefile
README.md
vcpkg.json

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DuckDB Delta Extension

This is the DuckDB extension for [Delta](https://delta.io/), built using the
[Delta Kernel](https://github.com/delta-incubator/delta-kernel-rs). The extension offers read and limited write (blind insert) support for delta
tables, both local and remote.

## Supported platforms

The supported platforms are:

- `linux_amd64` and `linux_amd64_gcc4` and `linux_arm64`
- `osx_amd64` and `osx_arm64`
- `windows_amd64`

Support for the [other](https://duckdb.org/docs/extensions/working_with_extensions#platforms) DuckDB platforms is
work-in-progress

## How to use

> [!NOTE]
> This extension requires the DuckDB v0.10.3 or higher

This extension is distributed as a binary extension. To use it, simply use one of its functions from DuckDB and the extension will be autoloaded:

```SQL
FROM delta_scan('s3://some/delta/table');
```

To scan a local table, use the full path prefixes with `file://`

```SQL
FROM delta_scan('file:///some/path/on/local/machine');
```

## Time travel

A table can be read at an earlier version, either by version number or by timestamp. A timestamp
selects the latest version committed at or before it. A timestamp carrying no zone -- a `TIMESTAMP`
rather than a `TIMESTAMPTZ`, or a string with no offset -- resolves through the session timezone:

```SQL
FROM delta_scan('file:///some/path', version => 3);
FROM delta_scan('file:///some/path', timestamp => TIMESTAMPTZ '2026-01-01 12:00:00+00');
```

The same applies to an attached table, both per query and for the whole attachment:

```SQL
ATTACH 'file:///some/path' AS tbl (TYPE delta);
FROM tbl AT (VERSION => 3);
FROM tbl AT (TIMESTAMP => TIMESTAMPTZ '2026-01-01 12:00:00+00');

ATTACH 'file:///some/path' AS tbl_old (TYPE delta, TIMESTAMP TIMESTAMPTZ '2026-01-01 12:00:00+00');
```

`version` and `timestamp` are mutually exclusive. An attached version or timestamp is the default for
the attachment rather than a pin, so a per-query `AT` clause overrides it.

## Cloud Storage authentication

Note that using DuckDB [Secrets](https://duckdb.org/docs/configuration/secrets_manager.html) for Cloud authentication is supported.

### S3 Example

```SQL
CREATE SECRET (
  TYPE S3,
  PROVIDER CREDENTIAL_CHAIN
);
FROM delta_scan('s3://some/delta/table/with/auth');
```

### Azure Example

```SQL
CREATE SECRET (
    TYPE AZURE,
    PROVIDER CREDENTIAL_CHAIN,
    CHAIN 'cli',
    ACCOUNT_NAME 'mystorageaccount'
);
FROM delta_scan('abfss://some/delta/table/with/auth');
```

### GCS Example

<https://duckdb.org/docs/guides/network_cloud_storage/gcs_import.html>
You need to create [HMAC keys](https://console.cloud.google.com/storage/settings;tab=interoperability) and declare a secret.

```SQL
CREATE SECRET (
    TYPE GCS,
    KEY_ID 'xxxx',
    SECRET 'yyy'
);
```

## Features

Many features/optimizations are supported in this extension as it reuses most of DuckDB's
regular parquet scanning logic:

- multithreaded scans and parquet metadata reading
- data skipping/filter pushdown
  - skipping row-groups in file (based on parquet metadata)
  - skipping complete files (based on delta partition info)
- projection pushdown
- blind inserts
- time travel by version or timestamp
- scanning tables with deletion vectors
- all primitive types
- structs
- VARIANT type support
- Cloud storage (AWS, Azure, GCP) support with secrets

## Building

See the [Extension Template](https://github.com/duckdb/extension-template) for generic build instructions

## Running tests

There are various tests available for the delta extension:

1. Delta Acceptence Test (DAT) based tests in `/test/sql/dat`
2. delta-kernel-rs based tests in `/test/sql/delta_kernel_rs`
3. Generated data based tests in `tests/sql/generated` (generated using [delta-rs](https://delta-io.github.io/delta-rs/), [PySpark](https://spark.apache.org/docs/latest/api/python/index.html), and DuckDB)

To run the first 2 sets of tests:

```shell
make test_debug
```

or in release mode

```shell
make test
```

To also run the tests on generated data:

```shell
make generate-data
GENERATED_DATA_AVAILABLE=1 make test
```

# Updating delta-kernel-rs / FFI version

Simply update the `GIT_TAG` definition found in `./CMakeLists.txt` and (re-)run
`make clean <debug|release>`. The FFI header is included directly from the
cargo build, and any breakage from the update should show up immediately.