ApacheSpark - GitLite

KO | EN

gitlite — search

#javascript #python #hacktoberfest #ai #android #machine-learning #linux #macos #windows #cli #kubernetes #awesome-list #openai #open-source

ApacheSpark

(★ 105)

This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

파일 탐색기

최종 버전 다운로드 (.zip)

- PS00-Introduction.py
- PS01-Read Files.py
- PS02-Schema Handling.py
- PS03-Creating Dataframes.py
- PS04-Basic Transformation.py
- PS05-Handling JSON.py
- PS06-JOINS.py
- PS07-Grouping & Aggregation.py
- PS08-Ordering Data.py
- PS09-String Functions.py
- PS10-Date & Time Functions.py
- PS11-Partitioning & Repartitioning.py
- PS12-Missing Value Handling.py
- PS13-Deduplication.py
- PS14-Data Profiling using PySpark.py
- PS15-Data Caching.py
- PS16-User Defined Functions.py
- PS17-Write Data.py
- Z01- Case Study Sales Order Analysis.py
- PYT01 - Unit Testing.py
- _clean_up.py
- _initial_setup.py
- _pyspark_clean_up.py
- _pyspark_init_setup.py
- _pyspark_setup_files.py
- _setup_database.py
- _setup_demo_table.py
- PS000-INTRODUCTION.py
- SR000-Introduction.py
- SR001-Basic CRUD.py
- SR002-Select & Filtering.py
- SR003-JOINS.py
- SR004-Order & Grouping.py
- SR005-Sub Queries.py
- SR006-Views & Temp Views.py
- SR007-Common Table Expressions.py
- SR008 - EXCEPT, UNION, UNION ALL, INTERSECTION.py
- SR009-External Tables.py
- SR010-Drop database & tables.py
- SR011-Check Table & Database Details.py
- SR012-Versioning, Time Travel & Optimization.py
LICENSE
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Data Engineering Using Azure Databricks

### Introduction

This course include multiple sections. We are mainly focusing on Databricks Data Engineer certification exam. We have following tutorials:
1. Spark SQL ETL
2. Pyspark ETL

### DATASETS
All the datasets used in the tutorials are available at: https://github.com/martandsingh/datasets

### HOW TO USE?
follow below article to learn how to clone this repository to your databricks workspace.

https://www.linkedin.com/pulse/databricks-clone-github-repo-martand-singh/

### Spark SQL
This course is the first installment of databricks data engineering course. In this course you will learn basic SQL concept which include:
1. Create, Select, Update, Delete tables
1. Create database
1. Filtering data
1. Group by & aggregation
1. Ordering
1. [SQL joins](https://www.scaler.com/topics/sql/joins-in-sql/)
1. Common table expression (CTE)
1. External tables
1. [Sub queries](https://www.geeksforgeeks.org/sql-subquery/)
1. Views & temp views
1. UNION, INTERSECT, EXCEPT keywords
1. Versioning, time travel & optimization

### PySpark ETL
This course will teach you how to perform ETL pipelines using pyspark. ETL stands for Extract, Load & Transformation. We will see how to load data from various sources & process it and finally will load the process data to our destination.

This course includes:
1. Read files
2. Schema handling
3. Handling JSON files
4. Write files
5. Basic transformations
6. partitioning
7. caching
8. joins
9. missing value handling
10. Data profiling
11. date time functions
12. string function
13. deduplication 
14. grouping & aggregation
15. User defined functions
16. Ordering data
17. Case study - sales order analysis

you can download all the notebook from our

github repo: https://github.com/martandsingh/ApacheSpark

facebook: https://www.facebook.com/codemakerz

email: martandsays@gmail.com

### SETUP folder
you will see initial_setup & clean_up notebooks called in every notebooks. It is mandatory to run both the scripts in defined order. initial script will create all the mandatory tables & database for the demo. After you finish your notebook, execute clean up notebook, it will clean all the db objects.

pyspark_init_setup - this notebook will copy dataset from my github repo to dbfs. It will also generate used car parquet dataset. All the datasets will be avalable at

**/FileStore/datasets**

![d5859667-databricks-logo](https://user-images.githubusercontent.com/32331579/174993501-dc93102a-ec36-4607-a3dc-ab67a54a341b.png)