Data-Engineering-Portfolio

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #cli #claude-code #codex #developer-tools #react #windows

Data-Engineering-Portfolio

★ 30 Open GitHub ↗

Data Engineering portfolio projects, resources used to study data tools...

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# 🔥Data-Engineering-Resources and Projects

- https://github.com/DataEngineer-io/data-engineer-handbook

### Data Engineering Workflow

### Big Data Architecture

## 📚Books

- [Designing Data-Intensive Applications]()
- [Fundamentals of Data Engineering]()
- [The Data Warehouse Toolkit]()
- [Cracking the Data Engineering Interview]()
- [Data Engineering with Python]()
- [Data Pipelines with Apache Airflow]()
- [The Data Warehouse Toolkit]()
- [Big Data: Principles and Best Practices of Scalable Real-Time Data Systems]()
- [Designing Data-Intensive Applications]()

## 🧰Tools for Data Engineers

- **Basic Skills**: `Linux`, `Git & GitHub`, `Computer Networking`, `Cloud Computing`, `Network & Security`, `Agile Development`
  
- **Advanced Skills (Good to Know)**: ` Data Lake & Data WareHouse Concepts`, `REST APIs`, `Databases(SQL & NoSQL)`
  
- **Programming Languages**: `Python`, `SQL`, `Java`, `Scala`
  
- **Databases**: `PostgreSQL`, `MongoDB`, `Neo4j`, `Redis`, `Cassandra`, `Apache HBase`, `Snowflake`, `InfluxDB`

- Data Ingestion: `Apache Kafka`, `Flume`, `Logstash`, `Airbyte`, `Apache Spark`, `Talend`, `Informatica`

- Data Tranformation: `Python`, `Pandas`, `SQL`, `Apache Spark`, `Hive`, `dbt`, `Matillion`, `Pig`
  
- Data Preprocessing: `Apache Spark`, `Apache Hadoop`, `Apache Flink`
  
- Data Orchestration: `Apache Airflow`, `Luigi`
      
- Data Storage: `Data Lake`: AWS S3, Azure Blob Storage, Google Cloud Storage, `Data Warehouse`: Snowflake, Google BigQuery, Amazon Redshift, Apache Hive
  
- Data Visualization: `Tableau`, `PowerBI`, `Looker`
  
- DataOps: `Docker`, `Kubernetes`, `Jenkins`

### Here's what's on the menu: 👇

- 🐍 Python,
- 📊 SQL,
- 🛠️ MySQL,
- 🌳 MongoDB,
- 🔥 PySpark,
- 🎈 Bash,
- 🌬️ Airflow,
- ☕ Apache Kafka,
- 🐙 Git,
- 🐈 GitHub,
- ⚙️ CICD basics,
- 🏬 Data Warehousing,
- 🛠️ DBT,
- 🌊 Data Lakes,
- 📘 DataBricks,
- ☁️ Azure Databricks,
- ❄️ Snowflake,
- 🌪️ Apache NiFi,
- 🌐 Debezium

1. Master Python: https://lnkd.in/d-pZPyf5

2. Learn SQL: https://lnkd.in/dzAiRF-x

3. Get hands-on with MySQL: https://lnkd.in/ddpSkUhc

4. Dive into MongoDB: https://lnkd.in/dHQ4VC2E

5. Master PySpark: https://lnkd.in/d7fgs7dE

6. Discover Bash, Airflow & Kafka: https://lnkd.in/dDhuEqQE

7. Master Git & GitHub: https://lnkd.in/dqJ7J3kN

8. Understand CICD basics: https://lnkd.in/dcfKBmCa

9. Decode Data Warehousing: https://lnkd.in/dPVRDJT5

10. Learn DBT: https://lnkd.in/eG9eaEuE

11. Understand Data Lakes: https://lnkd.in/dtZKJ4d6

12. Explore DataBricks: https://lnkd.in/dCBiQXPR

13. Learn Azure Databricks: https://lnkd.in/dzmwBs4Y

14. Master Snowflake: https://lnkd.in/dDBeddVy

15. Explore Apache NiFi: https://lnkd.in/de7bvnSt

## 📙Projects

| Sr. No. | Projects | Description | Tech Stack | Tags | Code Link |
|--------|-----------|--------------|-----------|-----|------------|
| 01. | [Build ETL Pipeline Using AWS Cloud]() | 
| 02. | [Covid Data Analysis Project]() |
| 03. | [Twitter Data Pipeline using Airflow and AWS]() |
| 04. | [YouTube Data Analysis (End-To-End Data Engineering Project)]() |
| 05. | [Olympic Data Analytics: End-To-End Azure Data Engineering Project]() |
| 06. | [Uber Data Analytics Project On GCP]() |
| 07. | [Data Ingestion and ETL Pipeline using Azure]() | 
| 08. | [Indian Stock Market Real-Time Data Processing, Analysis & Visualization using Azure Stream Analytics]() |
| 09. | [Simple Stock Market ETL Process with SQL]() |

## 🔶 Free Learning Resources

| Tools | Link| Used for | Official Docs | Youtube |
| ------|---------|---------------|----------|-----|
| DBMS | - [MySQL](https://lnkd.in/ddpSkUhc) - [MongoDB](https://lnkd.in/dHQ4VC2E)
| SQL | https://lnkd.in/dzAiRF-x
| Python | https://lnkd.in/d-pZPyf5 |
| Linux |
| Data Warehouse & Lake Concepts | - [Data Warehouse](https://lnkd.in/dPVRDJT5) - [Data Lakes](https://lnkd.in/dtZKJ4d6) 
| Data Pipelines |
| DBT | https://lnkd.in/eG9eaEuE
| PySpark | https://lnkd.in/d7fgs7dE
| Kafka |
| Apache Nifi | https://lnkd.in/de7bvnSt
| Airflow |
| Databricks | https://lnkd.in/dCBiQXPR
| Snowflake | https://lnkd.in/dDBeddVy
| Cloud Computing Concepts |
| Distributed Systems fundamentals |
| AWS |
| Azure |
| GCP |
| Git & GitHub | https://lnkd.in/dqJ7J3kN
| CI/CD | https://lnkd.in/dcfKBmCa
| Jenkins |
| Github Actions |
| Terraform |
| Sonarqube |
| Docker |
| Kubernetes |
| Power BI |
| Tableau |
| Apache Superset | 
| Prometheus |
| Graphana |
| Datadog |

## 💼 Read Real-World Case Studies -> Tech Blogs
1. Netflix - https://netflixtechblog.medium.com/
2. AWS - https://aws.amazon.com/solutions/case-studies/
3. GCP - https://cloud.google.com/customers
4. Azure - https://azure.microsoft.com/en-us/resources/customer-stories/
5. Spotify - https://engineering.atspotify.com/category/data/
6. MongoDB - https://www.mongodb.com/blog/all
7. Swiggy - https://bytes.swiggy.com/the-swiggy-delivery-challenge-part-one-6a2abb4f82f6
          - https://bytes.swiggy.com/swiggy-distance-service-9868dcf613f4
          - https://bytes.swiggy.com/the-tech-that-brings-you-your-food-1a7926229886
8. Zomato - https://blog.zomato.com/

Data-Engineering-Portfolio

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README