aws-lakehouse-medallion

KO | EN

gitlite — search

#javascript #python #react #ai #llm #android #machine-learning #linux #cli #kubernetes #vue #security #pytorch #ios

aws-lakehouse-medallion

(★ 25)

AWS medallion lakehouse portfolio with governance signals, open table format positioning, and observability.

aws-lakehouse-medallion 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# AWS Lakehouse Medallion

Public AWS lakehouse portfolio project with medallion-style data products, dbt on DuckDB for local validation, and explicit operational telemetry delivered into the lake through Firehose, S3, Glue, and Athena.

## Quickstart

The default developer workflow uses `uv` and Python 3.11+.

```bash
make setup
make doctor
```

If Python 3.11 is not already available locally, `uv` can provision it during `make setup`. A manual virtual environment is only a fallback:

```bash
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[dev]'
```

## Run The Demo

For local validation, use the Make target. It runs the full pipeline with an explicit mock telemetry backend, so the repo stays reproducible without AWS credentials:

```bash
make demo
```

For the real cloud path, configure Firehose telemetry first and then run the CLI directly:

```bash
export TELEMETRY_AWS_REGION=us-east-1
export TELEMETRY_FIREHOSE_STREAM_NAME=aws-lakehouse-dev-events
uv run aws-lakehouse-medallion demo
```

Useful operational commands:

```bash
uv run aws-lakehouse-medallion show-config
uv run aws-lakehouse-medallion doctor
AWS_LAKEHOUSE_TELEMETRY_BACKEND=mock uv run aws-lakehouse-medallion demo
uv run aws-lakehouse-medallion backfill --start 2026-02-01 --end 2026-02-02
uv run aws-lakehouse-medallion clean-artifacts
```

Generated artifacts land in:

- `artifacts/raw`
- `artifacts/ops`
- `artifacts/published`
- `artifacts/warehouse/local.duckdb`
- `dbt/runtime_seeds`

## Validate Changes

`make check` is the official local validation path. It runs diagnostics, Ruff, pytest, the demo smoke flow, and Terraform formatting and validation. The Make targets use an explicit mock telemetry backend so CI and local validation do not depend on live AWS credentials.

```bash
make check
```

Additional focused commands:

```bash
make fmt
make lint
make test
make dbt
```

## Architecture Snapshot

The demo keeps the local execution path lightweight while telling a production-facing AWS story:

- `ingest` writes append-only bronze-style raw JSONL extracts.
- `validate` deduplicates, enforces contracts, and routes bad records into dead-letter artifacts.
- `transform` runs dbt locally on DuckDB.
- `publish` writes an auditable outbox and published payloads.
- every operational event is mirrored locally to `artifacts/ops/pipeline_events.jsonl` and published to Kinesis Data Firehose for landing in S3 and querying through Glue/Athena.

The infrastructure story intentionally separates:

- orchestration path: EventBridge Scheduler -> SQS
- observability path: application -> Firehose -> S3 -> Glue/Athena

That separation makes the repo easier to explain to AWS buyers and avoids mixing control-plane messaging with operational telemetry.

## Project Layout

- `src/aws_lakehouse_demo/`: pipeline code, CLI, config loading, and telemetry wiring.
- `configs/`: runtime configuration defaults.
- `dbt/`: dbt project and local DuckDB profile.
- `infra/terraform/`: baseline S3, Glue, Athena, Firehose, SQS, and EventBridge Scheduler resources.
- `docs/`: architecture notes and runbooks.
- `tests/`: unit, CLI, config, and smoke coverage.

Further reading:

- [Architecture](docs/architecture.md)
- [Runbook](docs/runbook.md)
- [Terraform README](infra/terraform/README.md)

## Troubleshooting

- `uv sync` fails with Python 3.9:
  run `uv sync --python 3.11 --extra dev` explicitly.
- `demo` or `backfill` fails with a telemetry configuration error:
  either set `TELEMETRY_AWS_REGION` and `TELEMETRY_FIREHOSE_STREAM_NAME`, or use the explicit mock override for local validation.
- `doctor` reports `dbt_module_available: false`:
  rerun `make setup` so `dbt-duckdb` is installed inside the managed environment.
- `doctor` reports `telemetry_ready: false`:
  confirm the Firehose env vars or rerun through `make doctor` for the mock-backed local workflow.
- Firehose data lands in S3 but Athena does not show recent events:
  verify the Glue table location matches the Firehose prefix and refresh the Athena metadata if needed.
- Terraform validation fails before initialization:
  run `terraform -chdir=infra/terraform init -backend=false` once, then retry `make check`.

aws-lakehouse-medallion

aws-lakehouse-medallion 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

aws-lakehouse-medallion

aws-lakehouse-medallion 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README