End-to-End-Data-Pipeline

(β˜… 137)

πŸ“ˆ A scalable, production-ready data pipeline for real-time streaming & batch processing, integrating Kafka, Spark, Airflow, AWS, Kubernetes, and MLflow. Supports end-to-end data ingestion, transformation, storage, monitoring, and AI/ML serving with CI/CD automation using Terraform & GitHub Actions.

파일 탐색기

μ΅œμ’… 버전 λ‹€μš΄λ‘œλ“œ (.zip)
  • .dockerignore
  • .editorconfig
  • .env.example
  • .gitignore
  • .prettierignore
  • .prettierrc
  • api.sh
  • ARCHITECTURE.md
  • CITATION.cff
  • data_pipeline_setup.sh
  • DEPLOYMENT_STRATEGIES.md
  • docker-compose.ci.yaml
  • docker-compose.lite.yaml
  • docker-compose.yaml
  • End_to_End_Data_Pipeline.ipynb
  • index.html
  • LICENSE
  • Makefile
  • pyproject.toml
  • QUICK_START.md
  • README.md
  • requirements.txt
  • serve_wiki.py

# μ„€μΉ˜ κ°€μ΄λ“œ

1. μ½”λ“œ λ‚΄λ €λ°›κΈ°
git clone https://github.com/hoangsonww/End-to-End-Data-Pipeline

κΉƒν—ˆλΈŒμ—μ„œ ν”„λ‘œμ νŠΈ μ½”λ“œ 전체λ₯Ό λ‚΄ μ»΄ν“¨ν„°λ‘œ λ‚΄λ €λ°›μŠ΅λ‹ˆλ‹€.

cd End-to-End-Data-Pipeline

방금 내렀받은 ν”„λ‘œμ νŠΈ 폴더 μ•ˆμœΌλ‘œ μ΄λ™ν•©λ‹ˆλ‹€.

2. Docker

쉬움 μΆ”μ²œ
사전 μ€€λΉ„λ¬Ό
  • Git GitHubμ—μ„œ ν”„λ‘œμ νŠΈ μ½”λ“œλ₯Ό λ‚΄λ €λ°›μœΌλ €λ©΄ ν•„μš”ν•©λ‹ˆλ‹€.
  • Docker Desktop μ»¨ν…Œμ΄λ„ˆλ₯Ό λΉŒλ“œν•˜κ³  μ‹€ν–‰ν•˜λ €λ©΄ ν•„μš”ν•©λ‹ˆλ‹€. μ„€μΉ˜ ν›„ μ‹€ν–‰ν•΄μ„œ λ°±κ·ΈλΌμš΄λ“œμ— μΌœλ‘μ„Έμš”.
DC[Docker Compose<br/>make deploy-local]

compose μ„€μ • νŒŒμΌμ— μ •μ˜λœ μ„œλΉ„μŠ€λ“€μ„ λŒ€μƒμœΌλ‘œ λͺ…령을 μ‹€ν–‰ν•©λ‹ˆλ‹€.

DL[Docker Compose Lite<br/>make deploy-lite]

compose μ„€μ • νŒŒμΌμ— μ •μ˜λœ μ„œλΉ„μŠ€λ“€μ„ λŒ€μƒμœΌλ‘œ λͺ…령을 μ‹€ν–‰ν•©λ‹ˆλ‹€.

β”œβ”€β”€ docker-compose.yaml # 20 services

compose μ„€μ • νŒŒμΌμ— μ •μ˜λœ μ„œλΉ„μŠ€λ“€μ„ λŒ€μƒμœΌλ‘œ λͺ…령을 μ‹€ν–‰ν•©λ‹ˆλ‹€.

β”œβ”€β”€ docker-compose.ci.yaml # CI-specific compose

compose μ„€μ • νŒŒμΌμ— μ •μ˜λœ μ„œλΉ„μŠ€λ“€μ„ λŒ€μƒμœΌλ‘œ λͺ…령을 μ‹€ν–‰ν•©λ‹ˆλ‹€.

βœ… 터미널에 docker compose ps λ₯Ό μž…λ ₯ν•΄ μ»¨ν…Œμ΄λ„ˆλ“€μ΄ Up μƒνƒœμΈμ§€ ν™•μΈν•˜μ„Έμš”. README에 포트 λ²ˆν˜Έκ°€ μ ν˜€μžˆλ‹€λ©΄ λΈŒλΌμš°μ €μ—μ„œ http://localhost:포트번호 둜 μ ‘μ†ν•΄λ³΄μ„Έμš”.

이 레포의 README에 적힌 μ‹€μ œ λͺ…λ Ήμ–΄λ₯Ό κ·ΈλŒ€λ‘œ κ°€μ Έμ™”μŠ΅λ‹ˆλ‹€.

3. .NET

보톡
사전 μ€€λΉ„λ¬Ό
  • Git GitHubμ—μ„œ ν”„λ‘œμ νŠΈ μ½”λ“œλ₯Ό λ‚΄λ €λ°›μœΌλ €λ©΄ ν•„μš”ν•©λ‹ˆλ‹€.
  • .NET SDK dotnet λͺ…λ Ήμ–΄λ₯Ό μ“°κΈ° μœ„ν•œ SDKμž…λ‹ˆλ‹€.
β”‚ └── templates/ # 8 templates (airflow, spark, dotnet-api, kafka-producer, configmap, secrets, namespace)

이 λͺ…λ Ήμ–΄λ₯Ό 터미널에 κ·ΈλŒ€λ‘œ μž…λ ₯ν•΄ μ‹€ν–‰ν•˜μ„Έμš”.

βœ… dotnet run μ‹€ν–‰ ν›„ 터미널에 λ‚˜νƒ€λ‚˜λŠ” μ•ˆλ‚΄ λ©”μ‹œμ§€λ‚˜ μ£Όμ†Œλ₯Ό ν™•μΈν•˜μ„Έμš”.

이 레포의 README에 적힌 μ‹€μ œ λͺ…λ Ήμ–΄λ₯Ό κ·ΈλŒ€λ‘œ κ°€μ Έμ™”μŠ΅λ‹ˆλ‹€.

4. Python

쉬움
사전 μ€€λΉ„λ¬Ό
  • Git GitHubμ—μ„œ ν”„λ‘œμ νŠΈ μ½”λ“œλ₯Ό λ‚΄λ €λ°›μœΌλ €λ©΄ ν•„μš”ν•©λ‹ˆλ‹€.
  • Python 3 μ„€μΉ˜ μ‹œ 'Add Python to PATH' μ˜΅μ…˜μ„ κΌ­ μ²΄ν¬ν•˜μ„Έμš” (Windows).
pip install -r airflow/requirements.txt

requirements.txt 등에 λͺ…μ‹œλœ 파이썬 라이브러리λ₯Ό μ„€μΉ˜ν•©λ‹ˆλ‹€.

jupyter notebook

λΈŒλΌμš°μ €μ—μ„œ λ…ΈνŠΈλΆ(.ipynb) νŒŒμΌλ“€μ„ μ—΄λžŒν•˜κ³  μ‹€ν–‰ν•  수 μžˆλŠ” Jupyter 화면을 μΌ­λ‹ˆλ‹€.

βœ… μ—λŸ¬ λ©”μ‹œμ§€ 없이 μ‹€ν–‰λ˜κ³  터미널에 μ•ˆλ‚΄ 문ꡬ가 좜λ ₯되면 μ •μƒμž…λ‹ˆλ‹€.

5. Make

보톡
사전 μ€€λΉ„λ¬Ό
  • Git GitHubμ—μ„œ ν”„λ‘œμ νŠΈ μ½”λ“œλ₯Ό λ‚΄λ €λ°›μœΌλ €λ©΄ ν•„μš”ν•©λ‹ˆλ‹€.
  • Make Linux/macOSλŠ” 보톡 κΈ°λ³Έ μ„€μΉ˜λ˜μ–΄ μžˆμŠ΅λ‹ˆλ‹€. WindowsλŠ” 별도 μ„€μΉ˜(예: MSYS2, WSL)κ°€ ν•„μš”ν•©λ‹ˆλ‹€.
make build

μƒμ„±λœ λΉŒλ“œ 섀정을 λ°”νƒ•μœΌλ‘œ μ‹€μ œ μ»΄νŒŒμΌμ„ 진행해 μ‹€ν–‰ νŒŒμΌμ„ λ§Œλ“­λ‹ˆλ‹€.

make up

μƒμ„±λœ λΉŒλ“œ 섀정을 λ°”νƒ•μœΌλ‘œ μ‹€μ œ μ»΄νŒŒμΌμ„ 진행해 μ‹€ν–‰ νŒŒμΌμ„ λ§Œλ“­λ‹ˆλ‹€.

make health

μƒμ„±λœ λΉŒλ“œ 섀정을 λ°”νƒ•μœΌλ‘œ μ‹€μ œ μ»΄νŒŒμΌμ„ 진행해 μ‹€ν–‰ νŒŒμΌμ„ λ§Œλ“­λ‹ˆλ‹€.

make urls

μƒμ„±λœ λΉŒλ“œ 섀정을 λ°”νƒ•μœΌλ‘œ μ‹€μ œ μ»΄νŒŒμΌμ„ 진행해 μ‹€ν–‰ νŒŒμΌμ„ λ§Œλ“­λ‹ˆλ‹€.

make trigger-batch

μƒμ„±λœ λΉŒλ“œ 섀정을 λ°”νƒ•μœΌλ‘œ μ‹€μ œ μ»΄νŒŒμΌμ„ 진행해 μ‹€ν–‰ νŒŒμΌμ„ λ§Œλ“­λ‹ˆλ‹€.

βœ… μ—λŸ¬ 없이 λλ‚˜λ©΄ μ„±κ³΅μž…λ‹ˆλ‹€. μƒμ„±λœ μ‹€ν–‰ νŒŒμΌμ„ 직접 μ‹€ν–‰ν•΄λ³΄μ„Έμš”.

이 레포의 README에 적힌 μ‹€μ œ λͺ…λ Ήμ–΄λ₯Ό κ·ΈλŒ€λ‘œ κ°€μ Έμ™”μŠ΅λ‹ˆλ‹€.

였늘의 μΆ”μ²œ ν”„λ‘œμ νŠΈ

hermes-agent β˜… 233,851

λ‹Ήμ‹ κ³Ό ν•¨κ»˜ μ„±μž₯ν•˜λŠ” agent

Python
ComfyUI β˜… 128,789

κ·Έλž˜ν”„/λ…Έλ“œ μΈν„°νŽ˜μ΄μŠ€λ₯Ό κ°–μΆ˜ κ°€μž₯ κ°•λ ₯ν•˜κ³  λͺ¨λ“ˆν™”λœ ν™•μ‚° λͺ¨λΈ(diffusion model) GUI, API 및 λ°±μ—”λ“œ.

Python
ollama β˜… 179,103

Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma 및 기타 λͺ¨λΈλ“€μ„ λ°”λ‘œ μ‹œμž‘ν•΄ λ³΄μ„Έμš”.

Go
project-based-learning β˜… 280,216

ν”„λ‘œμ νŠΈ 기반 ν”„λ‘œκ·Έλž˜λ° νŠœν† λ¦¬μ–Ό λͺ¨μŒμž…λ‹ˆλ‹€.

Python
the-algorithm β˜… 73,854

X Recommendation Algorithm의 μ†ŒμŠ€ μ½”λ“œ

Scala
ML-For-Beginners β˜… 89,646

12μ£Ό, 26개의 κ°•μ’Œ, 52개의 ν€΄μ¦ˆ, λͺ¨λ‘λ₯Ό μœ„ν•œ ν΄λž˜μ‹ λ¨Έμ‹ λŸ¬λ‹ 컀리큘럼

Jupyter Notebook
// repository documentation