news-pipeline

★ 1 Open GitHub ↗

End-to-end news data pipeline using Apache Airflow, Kafka, MinIO, PostgreSQL, Docker, and Grafana. Automates web scraping, streaming ingestion, ETL, and Medallion (Bronze, Silver, Gold) storage for scalable analytics, monitoring, and reliable data engineering workflows. Built with Python and Docker Compose for scalable, reliable, automated flows!!!

Explore Similar Repositories

DocMeld : Lightweight Doc-to-agent-ready knowledge pipeline. Three-stage Bronze→Silver→Gold architecture extracts structured elements, page content, and AI-enriched metadata from research papers and books. Generate PRDs, workflows, topic clusters, and Claude Code skills from PDFs. No OCR required.
faang-real-time-data-platform : Production-grade real-time data platform on AWS — SQS → S3 Medallion Architecture (Bronze/Silver/Gold) → Athena. Built with Python, Pandas, Parquet & Terraform. 100% free tier.
modern-etl-stack : Production-ready open source ETL pipeline using Apache Kafka, Airflow, dbt, Spark, and Prometheus. Bronze-Silver-Gold architecture with real-time CDC and ML anomaly detection.
databricks-ceres-pipeline : Medallion analytics for the Ceres Open Data Index on Databricks — Lakeflow Declarative Pipelines (Bronze → Silver → Gold), runs on Free Edition.
databricks-mlops-workshop : A hands-on, one-day Databricks MLOps workshop: build an end-to-end fraud-detection pipeline (bronze → silver → gold, feature engineering, MLflow training, Unity Catalog model registry, serving, and monitoring) with Declarative Automation Bundles and CI/CD.
// repository documentation