modern-data-eng-dbt-databricks-azure

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

modern-data-eng-dbt-databricks-azure

(★ 39)

In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our cloud provider.

modern-data-eng-dbt-databricks-azure 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Modern Data Engineering with Medallion Architecture using DBT, Databricks, Spark and Azure Cloud 
In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our cloud provider. This project illustrate the process of data ingestion to the lakehouse, data integration with ADF and data transformation with Databricks, and DBT.

## System Architecture
![System Architecture.jpeg](https://raw.githubusercontent.com/airscholar/modern-data-eng-dbt-databricks-azure/main/System%20Architecture.jpeg)

## Commands
Try running the following commands:
- dbt run # for running models
- dbt test # for tests
- dbt snapshot # for snapshotting and slowly changing dimensions
- dbt docs generate # for documentation
- dbt docs serve # for documentation preview

## Resources:
* [Medium Article](https://medium.com/@yusuf.ganiyu/robust-data-pipelines-with-databricks-spark-dbt-and-azure-data-engineering-project-e5780fbc07a6)
* [DBT](https://docs.getdbt.com/guides)
* [Databricks](https://docs.databricks.com/)
* [Azure](https://docs.microsoft.com/en-us/azure/?product=featured)
* [Azure Data Factory](https://docs.microsoft.com/en-us/azure/data-factory/)
* [Azure Data Lake Storage Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction)

### Youtube Video
[![Modern Data Engienering](https://img.youtube.com/vi/divjURi-low/0.jpg)](https://youtu.be/divjURi-low)

modern-data-eng-dbt-databricks-azure

modern-data-eng-dbt-databricks-azure 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

modern-data-eng-dbt-databricks-azure

modern-data-eng-dbt-databricks-azure 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README