Azure-Olist-ETL-Data-Engineering-Project

KO | EN

gitlite — search

#javascript #ai #go #golang #rust #deep-learning #linux #docker #nodejs #java #awesome #vue #security #mcp

Azure-Olist-ETL-Data-Engineering-Project

(★ 1)

This Azure project demonstrates an end-to-end ETL pipeline built on Azure, processing the Olist eCommerce dataset using Azure Data Factory, Azure Databricks (PySpark), and Azure Data Lake Storage Gen2. It follows a modern medallion architecture (Bronze, Silver, Gold) for layered data management.

Azure-Olist-ETL-Data-Engineering-Project 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Azure-Olist-Data-Engineering-Project

# 📌 Project Overview

This project demonstrates an end-to-end ETL pipeline built on Azure, processing the Olist eCommerce dataset using Azure Data Factory, Azure Databricks (PySpark), and Azure Data Lake Storage Gen2. It follows a modern medallion architecture (Bronze, Silver, Gold) for layered data management.

# 🔁 Project Architecture

# ⚙️ Tools & Technologies

Azure Data Factory (ADF)

Azure Databricks (PySpark)

Azure Data Lake Storage Gen2 (ADLS Gen2)

Python / PySpark

# 🧭 ETL Architecture

# 1️⃣ Data Extraction

Linked ADLS Gen2 to ADF using service principal authentication.

Created ADF pipelines to load .csv files from http(github) into Bronze Layer in ADLS Gen2.

![image](https://github.com/user-attachments/assets/3d3c7666-ad47-43ad-b40d-5b07757e1269)

# 2️⃣ Data Exploration / EDA

Loaded data into PySpark DataFrames.

- Constructed a basic data model for joins.

- Performed schema inference (not recommended for Real-Time Use-Case), Prefer - Manual Schema using **StructType**.

# 3️⃣ Data Validation / Quality Analysis

- Validated: Null values, Column types, Uniqueness and referential integrity

# 4️⃣ Data Curation

- Selected necessary columns from each dataset, Renamed columns and applied logical grouping, Joined key datasets.

- Joining Datasets.

# 5️⃣ Optimization

Applied .cache() on final joined DataFrame to optimize repeated usage.

# 7️⃣ Data Transformation

- Renaming Columns: Used .withColumnRenamed() for clarity.

- Structuring Columns: Grouped fields logically.

- Data Cleaning :

# 8️⃣ Data Serving to Silver Layer :

# 9️⃣ Feature Engineering

- Filling Null Columns with Respective Values

- Calculating Delivery Gap :

- Extracting Year From Pruchase Date :

# 🔟 Business Insight :

- Analysing Top 10 Customers based on Total Spent

- Top Sellers Based on Revenue Generated

- Top Sellers Based on Total Reviews Scores.

# 1️⃣1️⃣ Data Serving to Gold Layer :

- Saved the final aggregated datasets to Gold Layer in Parquet format.

- Partitioned datasets based on Customer Segmentation.

Azure-Olist-ETL-Data-Engineering-Project

Azure-Olist-ETL-Data-Engineering-Project 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Azure-Olist-ETL-Data-Engineering-Project

Azure-Olist-ETL-Data-Engineering-Project 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README