End-to-End-Data-Pipeline
๐ A scalable, production-ready data pipeline for real-time streaming & batch processing, integrating Kafka, Spark, Airflow, AWS, Kubernetes, and MLflow. Supports end-to-end data ingestion, transformation, storage, monitoring, and AI/ML serving with CI/CD automation using Terraform & GitHub Actions.
ํ์ผ ํ์๊ธฐ
์ต์ข ๋ฒ์ ๋ค์ด๋ก๋ (.zip)- settings.local.json
- devcontainer.json
- cicd-pipeline.yml
- CODE_OF_CONDUCT.md
- CONTRIBUTING.md
- pull_request_template.md
- SECURITY.md
- profiles_settings.xml
- Project_Default.xml
- .gitignore
- copilot.data.migration.agent.xml
- copilot.data.migration.ask.xml
- copilot.data.migration.ask2agent.xml
- copilot.data.migration.edit.xml
- E2E-Data-Pipeline.iml
- google-java-format.xml
- misc.xml
- modules.xml
- vcs.xml
- batch_ingestion_dag.py
- streaming_monitoring_dag.py
- warehouse_transform_dag.py
- .gitkeep
- Dockerfile
- requirements.txt
- architecture_diagram.png
- full_flow_diagram.png
- bi_dashboard.py
- atlas_stub.py
- raw_data_validation.py
- great_expectations.yaml
- _helpers.tpl
- airflow.yaml
- configmap.yaml
- dotnet-api.yaml
- kafka-producer.yaml
- namespace.yaml
- secrets.yaml
- spark.yaml
- Chart.yaml
- values-aws.yaml
- values-azure.yaml
- values-gcp.yaml
- values-onprem.yaml
- values.yaml
- Dockerfile
- producer.py
- analysis-templates.yaml
- argo-app.yaml
- argo-rollouts-install.yaml
- deployment.yaml
- ingress.yaml
- rollout-blue-green.yaml
- rollout-canary.yaml
- servicemonitors.yaml
- services.yaml
- feature_store_stub.py
- mlflow_tracking.py
- grafana-deployment-dashboards.json
- monitoring.py
- prometheus.yml
- prometheus_grafana_setup.md
- script.js
- styles.css
- DataPipelineApi.1.0.2.nupkg
- librdkafka.so
- alpine-librdkafka.so
- centos6-librdkafka.so
- centos7-librdkafka.so
- librdkafka.so
- librdkafka.dylib
- librdkafka.dylib
- libcrypto-3-x64.dll
- libcurl.dll
- librdkafka.dll
- librdkafkacpp.dll
- libssl-3-x64.dll
- msvcp140.dll
- vcruntime140.dll
- zlib1.dll
- zstd.dll
- libcrypto-3.dll
- libcurl.dll
- librdkafka.dll
- librdkafkacpp.dll
- libssl-3.dll
- msvcp140.dll
- vcruntime140.dll
- zlib1.dll
- zstd.dll
- AWSSDK.Core.dll
- AWSSDK.S3.dll
- Confluent.Kafka.dll
- Dapper.dll
- DataPipelineApi
- DataPipelineApi.deps.json
- DataPipelineApi.dll
- DataPipelineApi.pdb
- DataPipelineApi.runtimeconfig.json
- Microsoft.AspNetCore.JsonPatch.dll
- Microsoft.AspNetCore.Mvc.NewtonsoftJson.dll
- Microsoft.Extensions.Http.Polly.dll
- Microsoft.Extensions.Logging.Abstractions.dll
- Microsoft.OpenApi.dll
- MySqlConnector.dll
- Newtonsoft.Json.Bson.dll
- Newtonsoft.Json.dll
- Npgsql.dll
- Polly.dll
- Polly.Extensions.Http.dll
- Swashbuckle.AspNetCore.Swagger.dll
- Swashbuckle.AspNetCore.SwaggerGen.dll
- Swashbuckle.AspNetCore.SwaggerUI.dll
- System.Diagnostics.DiagnosticSource.dll
- DataPipelineApi.1.0.0.nupkg
- DataPipelineApi.1.0.2.nupkg
- BatchController.cs
- CIController.cs
- GovernanceController.cs
- MLController.cs
- MonitoringController.cs
- StreamingController.cs
- WarehouseController.cs
- AirflowHealthCheck.cs
- KafkaHealthCheck.cs
- MinioHealthCheck.cs
- MLflowHealthCheck.cs
- MySqlHealthCheck.cs
- PostgresHealthCheck.cs
- BatchRequest.cs
- StreamingRequest.cs
- .NETCoreApp,Version=v8.0.AssemblyAttributes.cs
- DataPipelineApi.AssemblyInfo.cs
- DataPipelineApi.AssemblyInfoInputs.cache
- DataPipelineApi.assets.cache
- DataPipelineApi.csproj.AssemblyReference.cache
- DataPipelineApi.GeneratedMSBuildEditorConfig.editorconfig
- DataPipelineApi.GlobalUsings.g.cs
- DataPipelineApi.dll
- DataPipelineApi.dll
- .NETCoreApp,Version=v6.0.AssemblyAttributes.cs
- apphost
- DataPipelineApi.AssemblyInfo.cs
- DataPipelineApi.AssemblyInfoInputs.cache
- DataPipelineApi.assets.cache
- DataPipelineApi.csproj.AssemblyReference.cache
- DataPipelineApi.csproj.CopyComplete
- DataPipelineApi.csproj.CoreCompileInputs.cache
- DataPipelineApi.csproj.FileListAbsolute.txt
- DataPipelineApi.dll
- DataPipelineApi.GeneratedMSBuildEditorConfig.editorconfig
- DataPipelineApi.genruntimeconfig.cache
- DataPipelineApi.GlobalUsings.g.cs
- DataPipelineApi.MvcApplicationPartsAssemblyInfo.cache
- DataPipelineApi.MvcApplicationPartsAssemblyInfo.cs
- DataPipelineApi.pdb
- staticwebassets.build.json
- DataPipelineApi.1.0.0.nuspec
- DataPipelineApi.1.0.2.nuspec
- DataPipelineApi.csproj.nuget.dgspec.json
- DataPipelineApi.csproj.nuget.g.props
- DataPipelineApi.csproj.nuget.g.targets
- project.assets.json
- project.nuget.cache
- AirflowOptions.cs
- AtlasOptions.cs
- DatabaseOptions.cs
- GEOptions.cs
- GitHubOptions.cs
- KafkaOptions.cs
- MinioOptions.cs
- MLflowOptions.cs
- AtlasService.cs
- BatchService.cs
- CIService.cs
- DbService.cs
- GEValidationService.cs
- IAtlasService.cs
- IBatchService.cs
- ICIService.cs
- IDbService.cs
- IGEValidationService.cs
- IKafkaService.cs
- IMLflowService.cs
- IMonitoringService.cs
- IStorageService.cs
- IStreamingService.cs
- KafkaService.cs
- MinioService.cs
- MLflowService.cs
- MonitoringService.cs
- StreamingService.cs
- DataPipelineApi.csproj
- Program.cs
- appsettings.json
- appsettings.Production.json
- Dockerfile
- NuGet.config
- publish_nuget.sh
- README.md
- deploy-blue-green.sh
- deploy-canary.sh
- deploy.sh
- init_db.sql
- init_warehouse.sql
- setup-advanced-deployments.sh
- __init__.py
- init_warehouse.sql
- snowflake_connector.py
- Dockerfile
- spark_batch_job.py
- spark_streaming_job.py
- aws_s3_influxdb.py
- hadoop_batch_processing.py
- mongodb_streaming.py
- redis_integration.py
- argo-rollouts.tf
- ec2.tf
- eks.tf
- load-balancer-controller.tf
- networking.tf
- outputs.tf
- providers.tf
- rds.tf
- s3.tf
- security.tf
- terraform.tfvars.example
- variables.tf
- __init__.py
- test_data_validation.py
- test_docker_infrastructure.py
- test_kafka_producer.py
- test_pipeline_config.py
- test_snowflake.py
- test_warehouse_sql.py
- .dockerignore
- .editorconfig
- .env.example
- .gitignore
- .prettierignore
- .prettierrc
- api.sh
- ARCHITECTURE.md
- CITATION.cff
- data_pipeline_setup.sh
- DEPLOYMENT_STRATEGIES.md
- docker-compose.ci.yaml
- docker-compose.lite.yaml
- docker-compose.yaml
- End_to_End_Data_Pipeline.ipynb
- index.html
- LICENSE
- Makefile
- pyproject.toml
- QUICK_START.md
- README.md
- requirements.txt
- serve_wiki.py
# ์ค์น ๊ฐ์ด๋
git clone https://github.com/hoangsonww/End-to-End-Data-Pipeline
๊นํ๋ธ์์ ํ๋ก์ ํธ ์ฝ๋ ์ ์ฒด๋ฅผ ๋ด ์ปดํจํฐ๋ก ๋ด๋ ค๋ฐ์ต๋๋ค.
cd End-to-End-Data-Pipeline
๋ฐฉ๊ธ ๋ด๋ ค๋ฐ์ ํ๋ก์ ํธ ํด๋ ์์ผ๋ก ์ด๋ํฉ๋๋ค.
2. Docker
์ฌ์ ์ถ์ฒ- Git GitHub์์ ํ๋ก์ ํธ ์ฝ๋๋ฅผ ๋ด๋ ค๋ฐ์ผ๋ ค๋ฉด ํ์ํฉ๋๋ค.
- Docker Desktop ์ปจํ ์ด๋๋ฅผ ๋น๋ํ๊ณ ์คํํ๋ ค๋ฉด ํ์ํฉ๋๋ค. ์ค์น ํ ์คํํด์ ๋ฐฑ๊ทธ๋ผ์ด๋์ ์ผ๋์ธ์.
DC[Docker Compose<br/>make deploy-local]
compose ์ค์ ํ์ผ์ ์ ์๋ ์๋น์ค๋ค์ ๋์์ผ๋ก ๋ช ๋ น์ ์คํํฉ๋๋ค.
DL[Docker Compose Lite<br/>make deploy-lite]
compose ์ค์ ํ์ผ์ ์ ์๋ ์๋น์ค๋ค์ ๋์์ผ๋ก ๋ช ๋ น์ ์คํํฉ๋๋ค.
โโโ docker-compose.yaml # 20 services
compose ์ค์ ํ์ผ์ ์ ์๋ ์๋น์ค๋ค์ ๋์์ผ๋ก ๋ช ๋ น์ ์คํํฉ๋๋ค.
โโโ docker-compose.ci.yaml # CI-specific compose
compose ์ค์ ํ์ผ์ ์ ์๋ ์๋น์ค๋ค์ ๋์์ผ๋ก ๋ช ๋ น์ ์คํํฉ๋๋ค.
์ด ๋ ํฌ์ README์ ์ ํ ์ค์ ๋ช ๋ น์ด๋ฅผ ๊ทธ๋๋ก ๊ฐ์ ธ์์ต๋๋ค.
3. .NET
๋ณดํตโ โโโ templates/ # 8 templates (airflow, spark, dotnet-api, kafka-producer, configmap, secrets, namespace)
์ด ๋ช ๋ น์ด๋ฅผ ํฐ๋ฏธ๋์ ๊ทธ๋๋ก ์ ๋ ฅํด ์คํํ์ธ์.
์ด ๋ ํฌ์ README์ ์ ํ ์ค์ ๋ช ๋ น์ด๋ฅผ ๊ทธ๋๋ก ๊ฐ์ ธ์์ต๋๋ค.
4. Python
์ฌ์pip install -r airflow/requirements.txt
requirements.txt ๋ฑ์ ๋ช ์๋ ํ์ด์ฌ ๋ผ์ด๋ธ๋ฌ๋ฆฌ๋ฅผ ์ค์นํฉ๋๋ค.
jupyter notebook
๋ธ๋ผ์ฐ์ ์์ ๋ ธํธ๋ถ(.ipynb) ํ์ผ๋ค์ ์ด๋ํ๊ณ ์คํํ ์ ์๋ Jupyter ํ๋ฉด์ ์ผญ๋๋ค.
5. Make
๋ณดํต- Git GitHub์์ ํ๋ก์ ํธ ์ฝ๋๋ฅผ ๋ด๋ ค๋ฐ์ผ๋ ค๋ฉด ํ์ํฉ๋๋ค.
- Make Linux/macOS๋ ๋ณดํต ๊ธฐ๋ณธ ์ค์น๋์ด ์์ต๋๋ค. Windows๋ ๋ณ๋ ์ค์น(์: MSYS2, WSL)๊ฐ ํ์ํฉ๋๋ค.
make build
์์ฑ๋ ๋น๋ ์ค์ ์ ๋ฐํ์ผ๋ก ์ค์ ์ปดํ์ผ์ ์งํํด ์คํ ํ์ผ์ ๋ง๋ญ๋๋ค.
make up
์์ฑ๋ ๋น๋ ์ค์ ์ ๋ฐํ์ผ๋ก ์ค์ ์ปดํ์ผ์ ์งํํด ์คํ ํ์ผ์ ๋ง๋ญ๋๋ค.
make health
์์ฑ๋ ๋น๋ ์ค์ ์ ๋ฐํ์ผ๋ก ์ค์ ์ปดํ์ผ์ ์งํํด ์คํ ํ์ผ์ ๋ง๋ญ๋๋ค.
make urls
์์ฑ๋ ๋น๋ ์ค์ ์ ๋ฐํ์ผ๋ก ์ค์ ์ปดํ์ผ์ ์งํํด ์คํ ํ์ผ์ ๋ง๋ญ๋๋ค.
make trigger-batch
์์ฑ๋ ๋น๋ ์ค์ ์ ๋ฐํ์ผ๋ก ์ค์ ์ปดํ์ผ์ ์งํํด ์คํ ํ์ผ์ ๋ง๋ญ๋๋ค.
์ด ๋ ํฌ์ README์ ์ ํ ์ค์ ๋ช ๋ น์ด๋ฅผ ๊ทธ๋๋ก ๊ฐ์ ธ์์ต๋๋ค.