de-zoomcamp-2024

KO | EN

gitlite — search

#python #hacktoberfest #react #ai #go #deep-learning #docker #nodejs #java #windows #kubernetes #awesome-list #vue #openai

de-zoomcamp-2024

(★ 20)

No description available.

de-zoomcamp-2024 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DataTalks Data Engineering ZoomCamp
This repo contains my notes and homework assignments for the 2024 [Data Talks Data Engineering Zoomcamp](https://github.com/DataTalksClub/data-engineering-zoomcamp/tree/main) plus additional notes on orchestrators from other years -  Airflow (2022), Prefect (2023), and Kestra (2025) 
The final project for the 2024 course can be found in this repo [NYC Collisions Analytics](https://github.com/inner-outer-space/nyc-collisions-analytics)
<div align="center">
  <img src="https://github.com/inner-outer-space/de-zoomcamp-2024/assets/12296455/2689b1a3-60af-4b13-9ca5-dde96f28773c" 
    width="800" 
    height="auto">
</div>

## NOTES 
MODULE 1:   [Docker & SQL](https://github.com/inner-outer-space/de-zoomcamp-2024/blob/main/1a-docker_sql/readme.md) • [Terraform & GCP](https://github.com/inner-outer-space/de-zoomcamp-2024/blob/main/1b-terraform_gcp/readme.md) <br>
MODULE 2A:  [Orchestration with Mage](https://github.com/inner-outer-space/de-zoomcamp-2024/tree/main/2-workflow-orchestration) <br>
MODULE 2B:  [Orchestration with Airflow]() <br>
MODULE 2C:  [Orchestration with Prefect]() <br>
MODULE 2D:  [Orchestration with Kestra]() <br>
MODULE 3:   [Data Warehouses with BigQuery](https://github.com/inner-outer-space/de-zoomcamp-2024/blob/main/3a-data-warehouse/readme.md) <br>
MODULE 4:   [Analytics Engineering with dbt](https://github.com/inner-outer-space/de-zoomcamp-2024/tree/main/4-analytics-engineering) <br>
MODULE 5:   [Batch with Spark](https://github.com/inner-outer-space/de-zoomcamp-2024/tree/main/5-batch) <br>
MODULE 6:   [Streaming with Kafka](https://github.com/inner-outer-space/de-zoomcamp-2024/tree/main/6-streaming)
<br>
<br>
## RESOURCES

- [GIT Beginner](https://jdsalaro.com/tutorial/git/)
  - [GIT Cheatsheet](https://jdsalaro.com/cheatsheet/git/index.html)
  - [GIT Markdown Cheatsheet](https://github.com/aerabi/markdown-cheatsheet)
  - [GIT Extended](https://dangitgit.com/) Special commit cases
  - [GIT Readme](https://pragmaticpineapple.com/adding-custom-html-and-css-to-github-readme/) Adding HTML and CSS to GIT Readme
  - [GIT Merge Divergent Branches](https://stackoverflow.com/questions/71768999/how-to-merge-when-you-get-error-hint-you-have-divergent-branches-and-need-to-s)
</details>

<details>
<summary>Terraform</summary> 
  
  - [Terraform State Files](https://www.devopsschool.com/blog/what-is-terraform-tfstate-backup-file-in-terraform/)
</details>

<details>
<summary>Docker</summary>  
  
  - [Docker Commands - DockerLabs](https://dockerlabs.collabnix.com/docker/cheatsheet/)
  - [Docker Commands - Linux Handbook](https://linuxhandbook.com/essential-docker-commands/)
</details>

<details>
<summary>PostgreSQL</summary>
  
  - [Postgres](https://www.postgresql.org/docs/16/index.html) PostgreSQL official documentation  
</details>

<details>
<summary> PGAdmin </summary>
  
  - [PG Admin]()
</details>

<details>
<summary>Vizualization</summary>

- [Vizualize in Python](https://hossainlab.github.io/dataviz/notebooks/MPL01-Intro%20to%20Data%20Visualization.html)
</details>

<details>
<summary> Google Cloud Platform </summary>
  
  - [CLI Auth Login](https://mahira-technology.medium.com/gcloud-auth-login-using-cli-658ba1a0268f)
</details>

<details>
<summary> dbt - data build tool </summary>
  
  - [dbt resources](https://github.com/Hiflylabs/awesome-dbt)
</details>

<details>
<summary> dlt - data loader tool </summary>
  
  - [dlt practice api sources](https://dlthub.com/docs/blog/practice-api-sources)
  - [Building a dbt Pipeline](https://dlthub.com/docs/build-a-pipeline-tutorial)
  - [dlt Demos](https://github.com/dlt-hub/dlt_demos)
  - [dlt Blog](https://dlthub.com/docs/blog)
</details>

<details>
<summary> Duck DB </summary>
  
  - [duck DB](https://motherduck.com/blog/duckdb-tutorial-for-beginners/)
  - [Video - Why should you care about DuckDB](https://www.youtube.com/watch?v=q55UMyQapKA)
  - 
</details>

<details>
<summary> Batch Processing - Apache Spark </summary>
  
  - [Apache Spark](https://spark.apache.org/docs/latest/index.html)
  - [Spark 101](https://developer.hpe.com/blog/spark-101-what-is-it-what-it-does-and-why-it-matters/)
  - [Spark - Stand Alone Mode](https://spark.apache.org/docs/latest/spark-standalone.html)
  - [Deploy Spark Cluster on AWS](https://blog.insightdatascience.com/create-a-cluster-of-instances-on-aws-899a9dc5e4d0)
  - [MapReduce vs. Spark](https://www.knowledgehut.com/blog/big-data/apache-spark-and-mapreduce-comparison)
  - [PySpark Job in DataProc](https://cloud.yandex.com/en/docs/data-proc/tutorials/run-spark-job)
  - [Spark Context vs Spark Session](https://medium.com/@akhilasaineni7/exploring-sparkcontext-and-sparksession-8369e60f658e#:~:text=Internally%2C%20Spark%20Session%20creates%20a,worry%20about%20creating%20difference%20contexts.)
  - [Spark Pros Cons](https://www.altexsoft.com/blog/apache-spark-pros-cons/)
  - [Cluster vs Client Mode](https://medium.com/@sephinreji98/understanding-spark-cluster-modes-client-vs-cluster-vs-local-d3c41ea96073)
  - [Stand Alone Internals](https://books.japila.pl/spark-standalone-internals/overview/#scheduleexecutorsonworkers)
</details>

<details>
<summary> Streaming - Kafka </summary>
  
  - [Kafka]()
  - 
</details>

<details>
<summary> Potential Project Data Sources </summary>
  
  - [US Energy Use](https://www.eia.gov/opendata/)
  - 
</details>

de-zoomcamp-2024

de-zoomcamp-2024 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

de-zoomcamp-2024

de-zoomcamp-2024 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README