learn-pyspark

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #claude-code #codex #cli #developer-tools #react #windows

★ 35 Open GitHub ↗

Beginner Friendly PySpark Tutorials hosted at Spark Playground

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Learn PySpark - Beginner Friendly Tutorials

This repository contains a curated set of hands-on **PySpark tutorials** designed to help data engineers, data scientists, and analysts get comfortable with PySpark through bite-sized, practical tutorials.

Each tutorial is structured in a Markdown file and focuses on a single topic, from basics like reading CSVs to advanced topics like window functions.

---

## Features

- Markdown-based tutorials
- Covers beginner to advanced PySpark concepts
- Focus on **real-world data engineering tasks**
- Advanced window functions and transformations

---

### Spark Theory

| Title | Description |
|-------|-------------|
| [Introduction to Spark](./spark-theory/introduction-to-spark.md) | Learn Spark’s origins, motivations, and where it fits in the big data ecosystem. |
| [Spark Architecture](./spark-theory/spark-architecture.md) | Understand the high-level architecture of Spark: driver, executors, cluster manager. |
| [RDDs in Spark](./spark-theory/rdds-in-spark.md) | Explore Resilient Distributed Datasets and their core properties. |
| [DataFrames & Datasets](./spark-theory/dataframes-datasets.md) | Learn how Spark SQL builds on RDDs with optimized, structured APIs. |
| [Transformations & Actions](./spark-theory/transformations-actions.md) | Distinguish between lazy transformations and actions in Spark. |
| [Lazy Evaluation & DAG](./spark-theory/lazy-evaluation-dag.md) | See how Spark builds and optimizes the DAG before execution. |
| [Jobs, Stages & Tasks](./spark-theory/jobs-stages-tasks.md) | Break down how Spark splits work into jobs, stages, and tasks. |
| [Join Strategies](./spark-theory/join-strategies.md) | Understand broadcast, shuffle, and other join strategies in Spark. |
| [Catalyst Optimizer](./spark-theory/catalyst-optimizer.md) | Discover how Spark’s Catalyst optimizer rewrites and optimizes queries. |
| [Adaptive Query Execution](./spark-theory/adaptive-query-execution.md) | See how Spark adapts query plans at runtime for better performance. |
| [Common File Formats](./spark-theory/common-file-formats.md) | Compare CSV, JSON, Parquet, and other formats in the context of Spark. |
| [Partitioning & Bucketing](./spark-theory/partitioning-and-bucketing.md) | Learn how data partitioning and bucketing affect performance. |
| [Repartition & Coalesce](./spark-theory/repartition-and-coalesce.md) | Control the number of partitions efficiently using repartition and coalesce. |
| [Executor Memory](./spark-theory/executor-memory.md) | Dive into executor memory layout and how it impacts jobs. |

---

### PySpark Beginner Tutorials

| Title | Description |
|-------|-------------|
| [Introduction to PySpark](./tutorials/introduction-to-pyspark.md) | Learn what PySpark is and how it fits into the big data ecosystem. |
| [Setting Up Spark Session](./tutorials/setting-up-spark-session.md) | Learn how to create and configure a SparkSession in PySpark. |
| [Reading & Writing Files](./tutorials/reading-csv-files.md) | Read and write files in PySpark. |
| [Working with CSV Files](./tutorials/reading-csv-files.md) | Read and write CSVs with PySpark DataFrames. |
| [Working with JSON Files](./tutorials/reading-json-files.md) | Read and write JSON data in PySpark. |
| [Referring to Columns](./tutorials/referring-to-columns.md) | Access columns using different PySpark syntax styles. |
| [Selecting Columns](./tutorials/selecting-columns.md) | Various ways to select columns from a DataFrame. |
| [Filtering Data](./tutorials/filtering-data.md) | Apply filters to refine datasets in PySpark. |
| [Grouping Data](./tutorials/grouping-data.md) | Perform aggregations using `groupBy`. |
| [Joining Data](./tutorials/joining-data.md) | Combine multiple datasets using joins. |
| [Handling Nulls & Missing Data](./tutorials/handling-nulls.md) | Clean and transform missing data. |

### PySpark Advanced Tutorials

| Title | Description |
|-------|-------------|
| [Date & Time Functions](./tutorials/date-functions.md) | Work with timestamps and date functions. |
| [Math Functions](./tutorials/math-functions.md) | Use arithmetic and math functions efficiently. |
| [String Functions](./tutorials/string-functions.md) | Handle string operations in PySpark. |
| [Window Functions #1](./tutorials/window-functions.md) | Rank, aggregate, and apply window-based logic. |
| [Lead and Lag #2](./tutorials/window-functions-lead-lag.md) | Use `lead` and `lag` for time-based comparisons. |
| [Rows Between #3](./tutorials/window-functions-rows-between.md) | Work with the `rowsBetween` clause for rolling stats. |

---

## Contributions

Pull requests are welcome! If you’d like to add:

* A new tutorial
* Code examples
* Corrections or formatting fixes

### Guidelines

* Keep the format consistent with existing tutorials.
* Use **kebab-case** for file names. Example: `handling-nulls.md`, `window-functions-lead-lag.md`
* No title is required in the beginning of the tutorial md. The page title is handled by the website.
* Do not use emojis in the content (only in some places where it may be relevant).
* Use clear section headers (`##`) and code blocks where needed.
* Prefer concise explanations over long paragraphs.
* Ensure code examples are runnable and formatted properly.

### Before submitting a PR

* Ensure markdown renders correctly.
* Check for consistent formatting with existing tutorials.
* If it contains a new file, add the details in the `README.md` as well.
* Run and validate all code snippets.

---

Please open an issue or PR with your suggestion 🙌

---

## 🌐 Related Project

* [Spark Playground](https://www.sparkplayground.com) - an interactive platform where data engineers can learn, practice, and run PySpark code directly in the browser

---

learn-pyspark

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README