delta-examples

KO | EN

gitlite — search

#javascript #python #llm #golang #android #linux #docker #nodejs #awesome #pytorch #security #ios #mcp #open-source

(★ 242)

Delta Lake examples

File Explorer

Download Latest Version (.zip)

- - students1.csv
  - students2.csv
  - students3.csv
- people.csv
- people_countries.csv
- small_file.csv
- delta-polars.yml
- deltalake-0100.yml
- deltalake-0110.yml
- deltalake-0130.yml
- deltalake-0151.yml
- deltalake-0201.yml
- deltalake-081.yml
- deltalake-090.yml
- deltalake-minimal.yml
- mosaic-340.yml
- mr-delta-rs.yml
- mr-delta.yml
- pandas-deltalake.yml
- pyspark-322-delta-121.yml
- pyspark-322-delta-200.yml
- pyspark-330-delta-200.yml
- pyspark-330-delta-210.yml
- pyspark-330-delta-220.yml
- pyspark-332-delta-230.yml
- pyspark-340-delta-240.yml
- pyspark-340-delta-240rc1.yml
- pyspark-340-delta-300rc1.yml
- pyspark-350-delta-310.yml
- pyspark-350-delta-320.yml
- 2.4.0rc1.xml
- 3.0.0rc1.xml
- 3.1.0.xml
- - csv-parquet-delta-lake.ipynb
  - parquet-to-delta.ipynb
- - example.ipynb
- - Dockerfile
  - intro-to-delta-rs.ipynb
- - deltatorch.ipynb
  - utils.py
- - database.ddb
  - duckdb-ibis-deltalake.ipynb
  - duckdb-kernel-demo.ipynb
  - duckdb-point-query.ipynb
  - duckdb-read-delta.ipynb
- - basic-operations.ipynb
- - create-delta-table-with-pandas.ipynb
  - create-delta-table-with-pandas2.ipynb
  - pandas-constraints.ipynb
  - pandas-ibis-deltalake.ipynb
  - pandas-versioned-dataset.ipynb
  - Untitled.ipynb
  - why-deltalake-for-pandas.ipynb
- - benchmark-alternatives.ipynb
  - polars-constraints.ipynb
  - polars-file-skipping.ipynb
  - read-delta-table-polars.ipynb
- - 01_quickstart.ipynb
  - 02_table_batch_read_and_writes.ipynb
  - cache-danger.ipynb
  - change-column-name.ipynb
  - change-data-feed.ipynb
  - column-mapping-enabled.ipynb
  - constraints.ipynb
  - convert-csv-to-delta.ipynb
  - convert-iceberg-to-delta-lake.ipynb
  - create-table-delta-lake.ipynb
  - delete-rows.ipynb
  - deletion-vectors.ipynb
  - deletion-vectors2.ipynb
  - delta-210-features.ipynb
  - delta-230-features.ipynb
  - delta-basic-operations.ipynb
  - delta-lake-arraytype-columns.ipynb
  - delta-lake-optimize.ipynb
  - delta-managed-vs-external-tables.ipynb
  - delta-materialized-view.ipynb
  - delta-merge.ipynb
  - delta-merge2.ipynb
  - delta-z-order.ipynb
  - drop-column.ipynb
  - drop-table.ipynb
  - file-sizes.ipynb
  - generated-columns.ipynb
  - hive-style-partitioning.ipynb
  - incremental-updates-with-trigger.ipynb
  - optimize-compact.ipynb
  - parquet-to-delta-lake.ipynb
  - pyspark-append-overwrite.ipynb
  - pyspark-ibis-delta-lake.ipynb
  - release-candidate-template.ipynb
  - replacewhere.ipynb
  - rollback-with-restore.ipynb
  - schema-enforcement.ipynb
  - schema-evolution.ipynb
  - speed-up-spark-with-delta.ipynb
  - time-travel.ipynb
  - transaction-log.ipynb
  - UniForm.ipynb
  - uniform2.ipynb
  - vacuum.ipynb
- - blog_0_7_0.ipynb
  - daft-deltalake.ipynb
  - dask-deltalake.ipynb
  - datafusion-read-delta.ipynb
  - delta-lake-transactions.ipynb
  - delta-lake-without-spark.ipynb
  - deltalake_0_9_0.ipynb
  - deltalake_python_0_8_1.ipynb
  - get_add_actions.ipynb
  - intro-to-delta-rs.ipynb
  - lakehouse-interop.ipynb
  - mage-delta-lake.ipynb
  - point-query-benchmarks.ipynb
  - python-deltalake-documentation.ipynb
  - z-order-delta-table.ipynb
- - 01a-Delta Lake Workshop - Delta Lake Primer.ipynb
  - 01b-Delta Lake Workshop - SageMaker Models.ipynb
  - README.md
- - delta-lake-quickstart.ipynb
  - JodieNotebook.ipynb
- - Soda Library Demo.ipynb
.gitignore
.java-version
LICENSE.txt
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# delta-examples

This repo provides notebooks with Delta Lake examples using PySpark, Scala Spark, and Python.

Running these commands on your local machine is a great way to learn about how Delta Lake works.

## PySpark setup

You can install PySpark and Delta Lake by creating the `pyspark-330-delta-220` conda environment.

Create the environment with this command: `conda env create -f envs/pyspark-330-delta-220`.

Activate the environment with this command: `conda activate pyspark-330-delta-220`.

Then you can run `jupyter lab` and execute all the PySpark notebooks.

## Release Candidate setup

The Delta Lake Spark connector release process typically includes an early release candidate release a few weeks prior to the official release. ([example](https://github.com/delta-io/delta/releases/tag/v2.4.0rc1))

The release candidate is typically hosted in non-standard repositories that require additional setup.

To configure a notebook to use a release candidate:

1. Make sure to use a conda environment that includes the correct `--extra-index-url`, see [here](https://github.com/delta-io/delta-examples/blob/master/envs/pyspark-340-delta-240rc1.yml#L19-L20) for an example.
2. Provide the appropriate Ivy settings file that has the correct non-standard repository, see [this one](https://github.com/delta-io/delta-examples/blob/master/ivy/2.4.0rc1.xml) for an example.
3. Initialize Spark with the correct Ivy configuration

```
    .config(
        "spark.jars.ivySettings",
        "../../ivy/2.4.0rc1.xml"
    )
    ```

## delta-rs setup (Python bindings)

You can run the delta-rs notebooks that use the Python bindings by creating the `mr-delta-rs` conda environment.

Create the environment with this command: `conda env create -f envs/mr-delta-rs.yml`.

Activate the environment with this command: `conda activate mr-delta-rs`.

## delta-rs Rust setup

Rust notebooks in `notebooks/delta-rs` were developed using [Evcxr Jupyter Kernel](https://github.com/evcxr/evcxr/tree/main/evcxr_jupyter).

You can either follow the instructions in [Evcxr Jupyter Kernel](https://github.com/evcxr/evcxr/tree/main/evcxr_jupyter) to set it up, or you can use the included Docker file using the following commands (in which case all you need is Docker installed):
```
cd notebooks/delta-rs

docker build -t delta-rs .
docker run -it --rm  -p 8888:8888 --name delta-rs -v $PWD/notebooks/delta-rs:/usr/src/delta-rs delta-rs
```

**Note:** *One of the main reasons for creating the Dockerfile was an issue with running Evcxr Jupyter Kernel on MacOS with Apple chip - despite being able to build Rust applications directly on on the host after setting `[target.aarch64-apple-darwin]` in the `~/.cargo/config` file, `:dep` builds are stil not working in the notebook.*

## Scala setup

You can install [almond](https://almond.sh/) to run the Scala Spark notebooks in this repo.

## Contributing

We welcome contributions, especially notebooks that illustrate important functions that will benefit the Delta Lake community.

Check out the open issues for ideas on good notebooks to create!

delta-examples

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

delta-examples

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README