medallion-data-lakehouse

KO | EN

gitlite — search

#javascript #react #machine-learning #android #hacktoberfest #python #python3 #redux #typescript #flutter #cli #cpp

medallion-data-lakehouse

★ 3 Open GitHub ↗

A production-ready local Open Data Lakehouse implementing a Medallion Architecture (Bronze, Silver, Gold) with Apache Airflow, MinIO, DuckDB, dbt, and Apache Superset.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# 🌊 Local Open Data Lakehouse

An end-to-end local Data Lakehouse platform leveraging modern data stack tooling to ingest transactional data, store it in open Parquet format on object storage, transform it into star schema models using dbt, and serve interactive analytics in Superset.

---

## 📌 Features

- **ELT Architecture:** Incremental extraction from PostgreSQL OLTP into object storage before executing transformations.
- **Medallion Storage:** Structured object storage layout in MinIO (Bronze -> Silver -> Gold).
- **Open Standards:** Compressed Parquet file formats managed with DuckDB and dbt-duckdb.
- **Orchestration:** Automated daily runs scheduled via Apache Airflow.
- **BI & Analytics:** Apache Superset dashboards backed by DuckDB's high-performance query engine.

---

## 🚀 Quickstart

### 1. Launch Services via Docker Compose

Ensure Docker Desktop is running, then spin up the infrastructure:

```bash
# Build the custom Airflow image and start all services.
make build
```
or

```bash
docker compose -f .docker/docker-compose.yaml up -d --build
```

### 2. Service Access Points

---

## 💻 Local DAG Development & Environment Setup

Create a Python 3.10 environment for IDE support and local dbt work. Run Airflow through Docker: native Windows Python is not a supported Airflow runtime, including when commands are entered through Git Bash.

### 1. Setup Local Virtual Environment

Use `uv` to create the environment used by the editor and local dbt commands.

#### Option A: Using `uv` (Recommended - Ultra Fast)

```bash
# Create virtual environment with Python 3.10
uv venv .venv --python 3.10

# Install dependencies using the same Airflow constraint as the Docker image.
uv pip install --python .venv/Scripts/python.exe "apache-airflow==2.10.0" \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.10.0/constraints-3.10.txt"
uv pip install --python .venv/Scripts/python.exe -r .docker/config/requirements.txt
```

#### Repair a Stale Environment

If `.venv/Scripts/python.exe` reports that its Python runtime is missing, recreate the environment rather than reusing it:

```bash
mv .venv .venv.broken
uv python install 3.10
uv venv .venv --python 3.10
```

Then repeat the dependency-install commands above. Keep `.venv.broken` only until the new environment has been verified.

### 2. Configure Editor

Point your IDE (VS Code / PyCharm) interpreter to `.venv/Scripts/python.exe` (Windows) or `.venv/bin/python` (Linux/macOS).

### 3. Validate Airflow Through Docker

List the DAGs discovered by the running Airflow container:
```bash
docker compose -f .docker/docker-compose.yaml exec airflow-webserver airflow dags list
```

Dry-run a specific ingestion task in isolation (execution date format: `YYYY-MM-DD`):
```bash
docker compose -f .docker/docker-compose.yaml exec airflow-webserver \
  airflow tasks test ecommerce_db_fetch ingest_users_to_bronze 2026-08-01
```

---

## 📂 Repository Structure

```text
local-data-lakehouse/
|-- AGENTS.md                                  # Shared instructions for coding agents
|-- agent-skills/
|   `-- raw-file-generator/
|       `-- SKILL.md                           # Raw copy-pasteable file-content workflow
|-- .docker/
|   |-- airflow.Dockerfile                     # Custom Airflow image and dependencies
|   |-- config/
|   |   `-- requirements.txt                   # Pinned Python dependencies for the Airflow image
|   |-- docker-compose.yaml                    # Infrastructure definition
|   `-- .env                                   # Local environment variables
|-- airflow/
|   `-- dags/
|       `-- ecommerce_db_fetch/                # Ingestion DAG module
|           `-- _dag_ecommerce_db_fetch.py
|       `-- dbt_run/                           # Dynamic Silver and Gold dbt DAGs
|-- dbt/                                       # dbt transformation models and DuckDB configs
|-- ARCHITECTURE.md                            # Detailed technical design & system flow
`-- README.md
```

medallion-data-lakehouse

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README