KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #codex #typescript #ai-agents #ai-agent #cli #claude-code #open-source #deepseek-harness #dsh-plugin #python #mcp

f1_adf_databricks_pipeline

★ 1 Open GitHub ↗

End-to-end Azure data engineering project implementing a production-style lakehouse architecture using Azure Data Factory, Databricks, ADLS Gen2 and Delta Lake. Covers full lifecycle from platform setup and data ingestion to transformation, modeling, and analytics-ready datasets.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1>Project Overview</h1>
<p>
  This project demonstrates the design and implementation of an end-to-end Azure data platform,
  covering the full data lifecycle from ingestion to analytics-ready datasets.
</p>

<p>
  The platform was architected and provisioned from scratch, with a focus on production-style
  data engineering practices such as orchestration, incremental processing, and clean data
  modeling.
</p>

<p>
  A lakehouse architecture is used to separate raw ingestion, transformation, and presentation
  layers, enabling scalability, maintainability, and downstream analytics consumption.
</p>

<h3>Domain Covered</h3>
<ul>
  <li>Data Engineering</li>
  <li>Analytics Engineering</li>
  <li>Lakehouse Architecture on Azure</li>
</ul>

<h2>Architecture Overview</h2>
<img width="1837" height="964" alt="Architecture" src="https://github.com/user-attachments/assets/d9399e27-4e00-4895-84b3-df77d9a0eab9" />

<p>
  The solution follows a <strong>Raw → Ingested → Presentation</strong> (medallion) architecture:
</p>

<ul>
  <li>
    <strong>Raw layer:</strong> Stores source data ingested from the Ergast Formula 1 API
  </li>
  <li>
    <strong>Ingested layer:</strong> Cleaned and standardized Delta tables with enforced schemas
  </li>
  <li>
    <strong>Presentation layer:</strong> Analytics-ready fact and dimension tables optimized for BI
    and reporting use cases
  </li>
</ul>

<p>
  Azure Data Factory handles orchestration, Databricks performs compute and transformations,
  ADLS provides storage, and Delta Lake ensures transactional reliability.
</p>

<h3>Azure Services Used</h3>

<ul>
  <li>
    <strong>Azure Data Lake Storage Gen2 (ADLS):</strong> Centralized, scalable data storage
  </li>
  <li>
    <strong>Azure Data Factory (ADF):</strong> Pipeline orchestration and dependency management
  </li>
  <li>
    <strong>Azure Databricks:</strong> PySpark-based ingestion and transformation
  </li>
  <li>
    <strong>Delta Lake:</strong> ACID-compliant tables, MERGE operations, and incremental loads
  </li>
  <li>
    <strong>Power BI:</strong> Analytics and reporting layer
  </li>
</ul>

<h3>Data Ingestion</h3>

<ul>
  <li>Source data is ingested from the Ergast Formula 1 API</li>
  <li>
    Separate ingestion pipelines are implemented for:
    <ul>
      <li>Circuits</li>
      <li>Constructors</li>
      <li>Drivers</li>
      <li>Races</li>
      <li>Results</li>
      <li>Lap Times</li>
      <li>Pit Stops</li>
      <li>Qualifying</li>
    </ul>
  </li>
  <li>
    Pipelines are designed to be <strong>idempotent and re-runnable</strong> to prevent data
    duplication
  </li>
</ul>

<h3>Data Transformation &amp; Modeling</h3>

<ul>
  <li>Transformations are implemented using <strong>PySpark</strong></li>
  <li>Business rules are applied in the <strong>Ingested layer</strong></li>
  <li>
    <strong>Delta Lake MERGE</strong> logic supports incremental updates
  </li>
  <li>
    Presentation tables are modeled for:
    <ul>
      <li>Driver standings</li>
      <li>Constructor standings</li>
      <li>Race-level analytics</li>
    </ul>
  </li>
</ul>

<h3>Orchestration</h3>

<ul>
  <li>
    Azure Data Factory orchestrates:
    <ul>
      <li>End-to-end pipeline execution</li>
      <li>Layer dependencies</li>
      <li>Notebook scheduling</li>
    </ul>
  </li>
  <li>
    This ensures a clear separation between orchestration and transformation logic
  </li>
</ul>

<h3>Analytics &amp; Consumption</h3>

<ul>
  <li>The <strong>Presentation layer</strong> is exposed to Power BI</li>
  <li>
    Data models are optimized for:
    <ul>
      <li>BI performance</li>
      <li>Reusability</li>
      <li>Clear semantic definitions</li>
    </ul>
  </li>
</ul>

<h3>Key Focus Areas</h3>

<ul>
  <li>End-to-end platform setup and ownership</li>
  <li>Production-style orchestration and data pipelines</li>
  <li>Incremental processing and data reliability</li>
  <li>Analytics-ready data modeling</li>
  <li>Scalable lakehouse design</li>
</ul>

<h3>What This Project Demonstrates</h3>

<ul>
  <li>Ability to architect and implement Azure data platforms from scratch</li>
  <li>Strong understanding of data engineering and analytics engineering patterns</li>
  <li>Practical use of ADF, Databricks, Delta Lake, and Power BI together</li>
  <li>Clean separation of storage, compute, and orchestration layers</li>
</ul>

f1_adf_databricks_pipeline

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README