Brazilian-E-Commerce-Data-Engineering-Project

KO | EN

gitlite — search

#javascript #python #rust #deep-learning #macos #java #windows #kubernetes #awesome-list #vue #pytorch #security #ios #mcp

Brazilian-E-Commerce-Data-Engineering-Project

(★ 1)

End-to-end data engineering pipeline using Azure Data Factory, ADLS Gen2, and Databricks on the Olist Brazilian E-Commerce dataset, implementing Medallion Architecture, incremental loads, SCDs, and Star Schema modeling.

Brazilian-E-Commerce-Data-Engineering-Project 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Brazilian E-Commerce Data Engineering Project

## 📌 Project Overview
This project implements an end-to-end **data engineering pipeline** using the **Olist Brazilian E-Commerce dataset**.  
The solution follows **Medallion Architecture (Bronze → Silver → Gold)** and builds an **analytics-ready Star Schema** in the Gold layer.

The pipeline supports:
- Incremental data ingestion using watermarking
- Streaming-based processing with Databricks Auto Loader
- Slowly Changing Dimensions (SCD Type 1 & Type 2)
- Late-arriving dimension handling
- Fact and dimension modeling using best practices

---

## 🏗️ Architecture Overview
**Source → Lake → Warehouse**

- **Source**: Kaggle Olist Brazilian E-Commerce dataset (CSV)
- **Ingestion**: Azure Data Factory (ADF) with Self-Hosted Integration Runtime
- **Storage**: Azure Data Lake Storage Gen2 (ADLS)
- **Processing**: Azure Databricks (PySpark & SQL)
- **Warehouse**: Delta Lake tables managed via Unity Catalog

---

## 🧰 Technology Stack
- Azure Data Factory (ADF)– Orchestration & ingestion
- Self-Hosted Integration Runtime (SHIR)– Secure on-prem data movement
- Azure Data Lake Storage Gen2 (ADLS)– Data lake storage
- Azure Databricks– Streaming ingestion & transformations
- Delta Lake– ACID transactions & incremental loads
- Unity Catalog– Centralized governance
- PySpark & SQL– Transformations
- GitHub– Version control & documentation

---

## 🥉 Bronze Layer (Raw Ingestion)
- CSV files are copied from **local PC** to **ADLS Gen2**
- Files are converted to **Parquet format**
- Stored under: abfss://brazilian-e-commerce@<storage-account>.dfs.core.windows.net/bronze/<table_name>/

---

## 🥈 Silver Layer (Staging)
- Incremental ingestion using **Databricks Auto Loader**
- Implemented using `readStream` and `writeStream`
- Data is loaded into **Unity Catalog Silver schema**
- `ingestion_timestamp` is added during ingestion
- Schema enforcement using Delta Lake

### Watermarking
An `etl_watermark` table is used to track incremental loads.
- Initial load: last_processed_ingestion_ts = '1970-01-01 00:00:00'
- Subsequent loads process only new data

---

## 📊 Gold Layer Tables

### Dimensions
- `dim_customer` (SCD Type 2)
- `dim_products` (SCD Type 2)
- `dim_sellers` (SCD Type 2)
- `dim_geolocation` (SCD Type 1)
- `dim_date` (Static)

### Facts
- `fact_order_item`
- `fact_orders`
- `fact_order_payments`

---

## 📐 Star Schema – fact_order_item
**Grain:** One record per order item

Linked Dimensions:
- Customer
- Product
- Seller
- Customer Geolocation
- Seller Geolocation
- Shipping Date

Diagram:
- ![fact_order_item Star Schema](https://raw.githubusercontent.com/jdshelke/Brazilian-E-Commerce-Data-Engineering-Project/main/diagram/star_schema_order_items.png)

---

## 📐 Star Schema – fact_orders
**Grain:** One record per order

Linked Dimensions:
- Customer
- Date (role-playing dimension)

Diagram:
- ![fact_orders_Star Schema](https://raw.githubusercontent.com/jdshelke/Brazilian-E-Commerce-Data-Engineering-Project/main/diagram/star_schema_orders.png)

---

## 📐 Star Schema – fact_order_payments
**Grain:** One record per payment per order

Linked Dimensions:
- Customer
- Date (role-playing dimension)

Diagram:
- ![fact_order_payments_Star Schema](https://raw.githubusercontent.com/jdshelke/Brazilian-E-Commerce-Data-Engineering-Project/main/diagram/star_schema_order_payments.png)

---

## 🚀 Key Highlights
- End-to-end Azure data pipeline
- Incremental & streaming ingestion
- Watermark-based processing
- SCD Type 1 & Type 2 implementation
- Late-arriving dimension handling
- Analytics-ready Star Schema

---

## 📎 Dataset Reference
- **Olist Brazilian E-Commerce Dataset** (Kaggle)
- https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce

Brazilian-E-Commerce-Data-Engineering-Project

Brazilian-E-Commerce-Data-Engineering-Project 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Brazilian-E-Commerce-Data-Engineering-Project

Brazilian-E-Commerce-Data-Engineering-Project 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README