End-to-End-Data-Pipeline
π A scalable, production-ready data pipeline for real-time streaming & batch processing, integrating Kafka, Spark, Airflow, AWS, Kubernetes, and MLflow. Supports end-to-end data ingestion, transformation, storage, monitoring, and AI/ML serving with CI/CD automation using Terraform & GitHub Actions.
νμΌ νμκΈ°
μ΅μ’ λ²μ λ€μ΄λ‘λ (.zip)- settings.local.json
- devcontainer.json
- cicd-pipeline.yml
- CODE_OF_CONDUCT.md
- CONTRIBUTING.md
- pull_request_template.md
- SECURITY.md
- profiles_settings.xml
- Project_Default.xml
- .gitignore
- copilot.data.migration.agent.xml
- copilot.data.migration.ask.xml
- copilot.data.migration.ask2agent.xml
- copilot.data.migration.edit.xml
- E2E-Data-Pipeline.iml
- google-java-format.xml
- misc.xml
- modules.xml
- vcs.xml
- batch_ingestion_dag.py
- streaming_monitoring_dag.py
- warehouse_transform_dag.py
- .gitkeep
- Dockerfile
- requirements.txt
- architecture_diagram.png
- full_flow_diagram.png
- bi_dashboard.py
- atlas_stub.py
- raw_data_validation.py
- great_expectations.yaml
- _helpers.tpl
- airflow.yaml
- configmap.yaml
- dotnet-api.yaml
- kafka-producer.yaml
- namespace.yaml
- secrets.yaml
- spark.yaml
- Chart.yaml
- values-aws.yaml
- values-azure.yaml
- values-gcp.yaml
- values-onprem.yaml
- values.yaml
- Dockerfile
- producer.py
- analysis-templates.yaml
- argo-app.yaml
- argo-rollouts-install.yaml
- deployment.yaml
- ingress.yaml
- rollout-blue-green.yaml
- rollout-canary.yaml
- servicemonitors.yaml
- services.yaml
- feature_store_stub.py
- mlflow_tracking.py
- grafana-deployment-dashboards.json
- monitoring.py
- prometheus.yml
- prometheus_grafana_setup.md
- script.js
- styles.css
- DataPipelineApi.1.0.2.nupkg
- librdkafka.so
- alpine-librdkafka.so
- centos6-librdkafka.so
- centos7-librdkafka.so
- librdkafka.so
- librdkafka.dylib
- librdkafka.dylib
- libcrypto-3-x64.dll
- libcurl.dll
- librdkafka.dll
- librdkafkacpp.dll
- libssl-3-x64.dll
- msvcp140.dll
- vcruntime140.dll
- zlib1.dll
- zstd.dll
- libcrypto-3.dll
- libcurl.dll
- librdkafka.dll
- librdkafkacpp.dll
- libssl-3.dll
- msvcp140.dll
- vcruntime140.dll
- zlib1.dll
- zstd.dll
- AWSSDK.Core.dll
- AWSSDK.S3.dll
- Confluent.Kafka.dll
- Dapper.dll
- DataPipelineApi
- DataPipelineApi.deps.json
- DataPipelineApi.dll
- DataPipelineApi.pdb
- DataPipelineApi.runtimeconfig.json
- Microsoft.AspNetCore.JsonPatch.dll
- Microsoft.AspNetCore.Mvc.NewtonsoftJson.dll
- Microsoft.Extensions.Http.Polly.dll
- Microsoft.Extensions.Logging.Abstractions.dll
- Microsoft.OpenApi.dll
- MySqlConnector.dll
- Newtonsoft.Json.Bson.dll
- Newtonsoft.Json.dll
- Npgsql.dll
- Polly.dll
- Polly.Extensions.Http.dll
- Swashbuckle.AspNetCore.Swagger.dll
- Swashbuckle.AspNetCore.SwaggerGen.dll
- Swashbuckle.AspNetCore.SwaggerUI.dll
- System.Diagnostics.DiagnosticSource.dll
- DataPipelineApi.1.0.0.nupkg
- DataPipelineApi.1.0.2.nupkg
- BatchController.cs
- CIController.cs
- GovernanceController.cs
- MLController.cs
- MonitoringController.cs
- StreamingController.cs
- WarehouseController.cs
- AirflowHealthCheck.cs
- KafkaHealthCheck.cs
- MinioHealthCheck.cs
- MLflowHealthCheck.cs
- MySqlHealthCheck.cs
- PostgresHealthCheck.cs
- BatchRequest.cs
- StreamingRequest.cs
- .NETCoreApp,Version=v8.0.AssemblyAttributes.cs
- DataPipelineApi.AssemblyInfo.cs
- DataPipelineApi.AssemblyInfoInputs.cache
- DataPipelineApi.assets.cache
- DataPipelineApi.csproj.AssemblyReference.cache
- DataPipelineApi.GeneratedMSBuildEditorConfig.editorconfig
- DataPipelineApi.GlobalUsings.g.cs
- DataPipelineApi.dll
- DataPipelineApi.dll
- .NETCoreApp,Version=v6.0.AssemblyAttributes.cs
- apphost
- DataPipelineApi.AssemblyInfo.cs
- DataPipelineApi.AssemblyInfoInputs.cache
- DataPipelineApi.assets.cache
- DataPipelineApi.csproj.AssemblyReference.cache
- DataPipelineApi.csproj.CopyComplete
- DataPipelineApi.csproj.CoreCompileInputs.cache
- DataPipelineApi.csproj.FileListAbsolute.txt
- DataPipelineApi.dll
- DataPipelineApi.GeneratedMSBuildEditorConfig.editorconfig
- DataPipelineApi.genruntimeconfig.cache
- DataPipelineApi.GlobalUsings.g.cs
- DataPipelineApi.MvcApplicationPartsAssemblyInfo.cache
- DataPipelineApi.MvcApplicationPartsAssemblyInfo.cs
- DataPipelineApi.pdb
- staticwebassets.build.json
- DataPipelineApi.1.0.0.nuspec
- DataPipelineApi.1.0.2.nuspec
- DataPipelineApi.csproj.nuget.dgspec.json
- DataPipelineApi.csproj.nuget.g.props
- DataPipelineApi.csproj.nuget.g.targets
- project.assets.json
- project.nuget.cache
- AirflowOptions.cs
- AtlasOptions.cs
- DatabaseOptions.cs
- GEOptions.cs
- GitHubOptions.cs
- KafkaOptions.cs
- MinioOptions.cs
- MLflowOptions.cs
- AtlasService.cs
- BatchService.cs
- CIService.cs
- DbService.cs
- GEValidationService.cs
- IAtlasService.cs
- IBatchService.cs
- ICIService.cs
- IDbService.cs
- IGEValidationService.cs
- IKafkaService.cs
- IMLflowService.cs
- IMonitoringService.cs
- IStorageService.cs
- IStreamingService.cs
- KafkaService.cs
- MinioService.cs
- MLflowService.cs
- MonitoringService.cs
- StreamingService.cs
- DataPipelineApi.csproj
- Program.cs
- appsettings.json
- appsettings.Production.json
- Dockerfile
- NuGet.config
- publish_nuget.sh
- README.md
- deploy-blue-green.sh
- deploy-canary.sh
- deploy.sh
- init_db.sql
- init_warehouse.sql
- setup-advanced-deployments.sh
- __init__.py
- init_warehouse.sql
- snowflake_connector.py
- Dockerfile
- spark_batch_job.py
- spark_streaming_job.py
- aws_s3_influxdb.py
- hadoop_batch_processing.py
- mongodb_streaming.py
- redis_integration.py
- argo-rollouts.tf
- ec2.tf
- eks.tf
- load-balancer-controller.tf
- networking.tf
- outputs.tf
- providers.tf
- rds.tf
- s3.tf
- security.tf
- terraform.tfvars.example
- variables.tf
- __init__.py
- test_data_validation.py
- test_docker_infrastructure.py
- test_kafka_producer.py
- test_pipeline_config.py
- test_snowflake.py
- test_warehouse_sql.py
- .dockerignore
- .editorconfig
- .env.example
- .gitignore
- .prettierignore
- .prettierrc
- api.sh
- ARCHITECTURE.md
- CITATION.cff
- data_pipeline_setup.sh
- DEPLOYMENT_STRATEGIES.md
- docker-compose.ci.yaml
- docker-compose.lite.yaml
- docker-compose.yaml
- End_to_End_Data_Pipeline.ipynb
- index.html
- LICENSE
- Makefile
- pyproject.toml
- QUICK_START.md
- README.md
- requirements.txt
- serve_wiki.py
// repository documentation
Was this content helpful?
(0 ratings)