docker-spark-iceberg

(★ 384)

No description available.

File Explorer

Download Latest Version (.zip)

- - check-notebooks.yml
  - docker-image.yml
  - stale.yml
- dependabot.yml
- .rat-excludes
- build-and-push-image.sh
- check-license
- .gitignore
- build.gradle
- docker-compose.yml
- gradlew
- README.md
- - Iceberg - An Introduction to the Iceberg Java API using Scala.ipynb
  - Iceberg - Berlin Buzzwords 2023.ipynb
  - Iceberg - Getting Started.ipynb
  - Iceberg - Integrated Audits Demo.ipynb
  - Iceberg - Table Maintenance Spark Procedures.ipynb
  - Iceberg - View Support.ipynb
  - Iceberg - Write-Audit-Publish (WAP) with Branches.ipynb
  - PyIceberg - Commits.ipynb
  - PyIceberg - Getting Started.ipynb
  - PyIceberg - Write support.ipynb
- .pyiceberg.yaml
- Dockerfile
- entrypoint.sh
- requirements.txt
- spark-defaults.conf
.dockerignore
.gitignore
.pre-commit-config.yaml
docker-compose.yml
LICENSE
README.md

# Installation Guide

1. Get the code

git clone https://github.com/databricks/docker-spark-iceberg

Downloads the entire project code from GitHub to your computer.

cd docker-spark-iceberg

Moves into the project folder you just downloaded.

2. Docker

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Docker Desktop Needed to build and run containers. Install it and keep it running in the background.

docker-compose up

Runs the command against the services defined in the compose file.

docker exec -it spark-iceberg spark-shell

Type this command into your terminal and run it.

docker exec -it spark-iceberg spark-sql

Type this command into your terminal and run it.

docker exec -it spark-iceberg pyspark

Type this command into your terminal and run it.

docker image rm tabulario/spark-iceberg && docker-compose pull

Runs the command against the services defined in the compose file.

✅ Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.

Pulled directly from this repo's README.

3. Gradle (Java/Kotlin)

Medium

Prerequisites

Git Needed to download the project code from GitHub.
JDK (Java) Required to build and run Java/Kotlin projects.
Gradle If the repo includes gradlew (Gradle Wrapper), you don't need to install Gradle separately.

cd flink-example

This project's files live in a subfolder, so move into it first.

./gradlew build

Builds the project using Gradle.

✅ A BUILD SUCCESSFUL message means it worked. The output is created under build/.

4. Python

Easy

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r spark/requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

jupyter notebook

Launches Jupyter in your browser so you can open and run the notebook (.ipynb) files.

✅ If it runs without errors and prints output in the terminal, it worked.