uber-data-engineering-mage-project

KO | EN

gitlite — search

#javascript #hacktoberfest #react #llm #go #linux #docker #nodejs #macos #java #awesome #awesome-list #pytorch #open-source

uber-data-engineering-mage-project

(★ 26)

Uber Data Engineering Pipeline using Mage AI and BigQuery

uber-data-engineering-mage-project 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Uber Data Analytics | Modern Data Engineering GCP Project

## Introduction
This project dives into the world of Uber data analytics using modern data engineering practices on Google Cloud Platform (GCP). We'll leverage tools like Mage.ai for building an ETL pipeline, BigQuery for data warehousing, Looker Studio for data visualization, and Cloud Storage for managing data throughout the process.

## Architecture
![Project Architecture](https://raw.githubusercontent.com/darshilparmar/uber-data-engineering-mage-project/main/architecture.jpg)

## Technology Used
1. Programming Language - Python
2. Scripting Language - SQL
3. Google Cloud Platform
   -  BigQuery
   -  Cloud Storage
   -  Looker Studio
   -  Compute Instance
4. Mage.AI (modern data pipeline tool)

**Modern data Pipeline Tool:** https://www.mage.ai/

**Contribute to this project here:** https://github.com/mage-ai/mage-ai

## Dataset Used
TLC Trip Record Data
Yellow and green taxi trip records include fields capturing pick-up and drop-off dates/times, pick-up and drop-off locations, trip distances, itemized fares, rate types, payment types, and driver-reported passenger counts.

Here is the dataset used in the video - https://github.com/darshilparmar/uber-data-engineering-mage-project/blob/main/data/uber_data.csv

### More Info About Dataset
1. Original Data Source - https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page
2. Data Dictionary - https://www.nyc.gov/assets/tlc/downloads/pdf/data_dictionary_trip_records_yellow.pdf

## Data Model
![Data model image](https://raw.githubusercontent.com/darshilparmar/uber-data-engineering-mage-project/main/data_model.jpeg)

## Scripts for project
1. [Extract Python File](mage-files/extract.py)
2. [Load Python File](mage-files/load.py)
3. [Transform Python File](mage-files/transform.py)

## Complete Video Tutorial
Video Link - https://www.youtube.com/watch?v=WpQECq5Hx9g

uber-data-engineering-mage-project

uber-data-engineering-mage-project 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

uber-data-engineering-mage-project

uber-data-engineering-mage-project 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README