Containerised-Data_Stack

KO | EN

gitlite — search

#hacktoberfest #react #typescript #android #linux #docker #macos #java #awesome-list #pytorch #security #ios #openai #open-source

Containerised-Data_Stack

(★ 25)

A fully containerized modern data engineering platform that integrates data ingestion, orchestration, processing, storage, and analytics services using Docker. The project demonstrates how to build a portable, scalable, and reproducible data stack for development and production environments.

Containerised-Data_Stack 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# GitHub Analytics Stack

Containerized analytics pipeline: **GitHub commits** → **PostgreSQL** → **dbt** → **Apache Superset**.

Ingestion runs via a Python GitHub API client (local dev) with **Airbyte JSON templates** included for production deployments.

## Architecture

```
┌──────────────┐    ┌──────────────┐    ┌──────────┐    ┌──────────┐    ┌───────────┐
│ GitHub API   │───▶│  Ingestion   │───▶│ Postgres │───▶│   dbt    │───▶│ Superset  │
│ (or Airbyte) │    │  (Python)    │    │ raw data │    │ models   │    │ dashboard │
└──────────────┘    └──────────────┘    └──────────┘    └──────────┘    └───────────┘
```

## Project Structure

```
github-analytics-stack/
├── ingestion/                  # GitHub API ingestion
│   ├── extract_github.py
│   ├── load.py
│   └── airbyte_config/         # Airbyte templates for production
├── dbt_project/                  # Staging → intermediate → marts
├── superset/                     # Superset configuration
├── postgres/init.sql
├── scripts/
├── tests/
├── docker-compose.yml
└── Makefile
```

## Real-World Features

- **GitHub API ingestion** with retries, rate-limit handling, and upserts
- **Airbyte IaC templates** for production-scale ingestion
- **dbt medallion models** with source freshness and 15+ data tests
- **Superset** for contributor activity dashboards
- **Health checks** on Postgres and Superset containers
- **Port isolation** — Postgres on host port `5434`

## Quickstart

### Prerequisites

- Docker & Docker Compose
- Python 3.10+
- (Optional) `GITHUB_TOKEN` for higher API rate limits

### 1. Configure

```bash
cd github-analytics-stack
cp .env.example .env
make setup
source .venv/bin/activate
```

### 2. Run the full pipeline

```bash
make up
make pipeline
```

### 3. Initialize Superset (first time)

```bash
make superset-init
```

Open http://localhost:8088 and connect to `analytics.contributor_activity`.

## Operations

```bash
make ingest        # Fetch commits from configured repos
make dbt-run       # Build analytics models
make dbt-test      # Run dbt data tests
make health        # Verify ingestion and marts
make logs          # Tail container logs
make clean         # Tear down volumes
```

## Configuration

| Variable | Default | Description |
|----------|---------|-------------|
| `GITHUB_REPOS` | `apache/superset,dbt-labs/dbt-core` | Comma-separated repos |
| `GITHUB_TOKEN` | — | PAT for higher rate limits |
| `POSTGRES_PORT` | `5434` | Host Postgres port |
| `SUPERSET_PORT` | `8088` | Superset UI port |

## Production Ingestion (Airbyte)

Use the templates in `ingestion/airbyte_config/` to configure Airbyte Cloud or self-hosted Airbyte OSS. Point the destination at the same `raw_commits` schema used by dbt sources.

## Data Model

| Layer | Object | Description |
|-------|--------|-------------|
| Raw | `public.raw_commits` | GitHub commit JSON payloads |
| Staging | `analytics.stg_commits` | Cleaned commit records |
| Intermediate | `analytics.int_commits_by_contributor` | Weekly aggregates |
| Marts | `analytics.contributor_activity` | Dashboard-ready metrics |

## Testing

```bash
make test
```

## License

MIT

Containerised-Data_Stack

Containerised-Data_Stack 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Containerised-Data_Stack

Containerised-Data_Stack 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README