Corporate_Data_Lakehouse_Pipeline

★ 1 Open GitHub ↗

An end-to-end data engineering project that builds a modern lakehouse pipeline integrating corporate registry and financial market data. The system ingests data from Companies House and Yahoo Finance APIs, processes it using PySpark, and organizes it into bronze, silver, and gold layers for analytics and reporting.

Explore Similar Repositories

DocMeld : Lightweight Doc-to-agent-ready knowledge pipeline. Three-stage Bronze→Silver→Gold architecture extracts structured elements, page content, and AI-enriched metadata from research papers and books. Generate PRDs, workflows, topic clusters, and Claude Code skills from PDFs. No OCR required.
faang-real-time-data-platform : Production-grade real-time data platform on AWS — SQS → S3 Medallion Architecture (Bronze/Silver/Gold) → Athena. Built with Python, Pandas, Parquet & Terraform. 100% free tier.
modern-etl-stack : Production-ready open source ETL pipeline using Apache Kafka, Airflow, dbt, Spark, and Prometheus. Bronze-Silver-Gold architecture with real-time CDC and ML anomaly detection.
databricks-ceres-pipeline : Medallion analytics for the Ceres Open Data Index on Databricks — Lakeflow Declarative Pipelines (Bronze → Silver → Gold), runs on Free Edition.
databricks-mlops-workshop : A hands-on, one-day Databricks MLOps workshop: build an end-to-end fraud-detection pipeline (bronze → silver → gold, feature engineering, MLflow training, Unity Catalog model registry, serving, and monitoring) with Declarative Automation Bundles and CI/CD.
// repository documentation