spark-rdd-workshop

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #cli #claude-code #codex #developer-tools #react #windows

spark-rdd-workshop

★ 11 Open GitHub ↗

🔥 Comprehensive Apache Spark RDD workshop with 3 hands-on exercises: log analysis, sales aggregation, and distributed word counting. Learn data filtering, partitioning, map-reduce, joins & caching. Perfect for data engineering students & big data enthusiasts. Jupyter notebooks included.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# 🔥 Apache Spark RDD Workshop

[![Python](https://img.shields.io/badge/Python-3.7+-blue.svg)](https://www.python.org/downloads/)
[![PySpark](https://img.shields.io/badge/PySpark-3.0+-orange.svg)](https://spark.apache.org/)
[![License](https://img.shields.io/badge/License-MIT-green.svg)](https://raw.githubusercontent.com/Dansoko22md/spark-rdd-workshop/main/LICENSE)
[![Jupyter](https://img.shields.io/badge/Jupyter-Notebook-orange.svg)](https://jupyter.org/)

A comprehensive hands-on workshop for learning Apache Spark RDD (Resilient Distributed Dataset) operations through practical exercises. This project contains three complete exercises demonstrating core Spark concepts including data filtering, aggregation, partitioning, and distributed word counting.

## 📋 Table of Contents

- [Overview](#overview)
- [Features](#features)
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Project Structure](#project-structure)
- [Exercises](#exercises)
- [Usage](#usage)
- [Technologies](#technologies)
- [Results](#results)
- [Contributing](#contributing)
- [License](#license)

## 🎯 Overview

This workshop provides hands-on experience with Apache Spark's foundational data structure: RDDs. Through three progressive exercises, you'll learn:

1. **Log Analysis**: Filter and count ERROR entries in log files
2. **Sales Analytics**: Aggregate purchase data across multiple stores with partitioning
3. **Text Processing**: Perform distributed word counting with RDD joins and caching

Perfect for data engineering students, big data enthusiasts, or anyone looking to master Spark fundamentals.

## ✨ Features

- **Exercise 1: Error Log Analysis**
  - RDD creation from text files
  - Data filtering with lambda functions
  - Action operations (count, take)
  
- **Exercise 2: Sales Data Processing**
  - Tab-separated value (TSV) parsing
  - Dynamic repartitioning
  - Map-Reduce aggregation
  - Group-by operations
  
- **Exercise 3: Advanced Word Count**
  - Multi-file RDD processing
  - Full outer joins
  - RDD caching for performance
  - Regular expression text cleaning
  - Sorted result generation

## 🔧 Prerequisites

- Python 3.7 or higher
- Java JDK 8 or 17 (required for Spark)
- Jupyter Notebook or JupyterLab
- Basic understanding of Python and distributed computing concepts

## 📦 Installation

### 1. Clone the Repository

```bash
git clone https://github.com/Dansoko22md/spark-rdd-workshop.git
cd spark-rdd-workshop
```

### 2. Set Up Python Environment

```bash
# Create virtual environment
python -m venv venv

# Activate virtual environment
# On Windows:
venv\Scripts\activate
# On macOS/Linux:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt
```

### 3. Install Apache Spark

#### On Windows:
```bash
# Download Spark from https://spark.apache.org/downloads.html
# Extract to C:\spark
# Set environment variables as shown in the notebook
```

#### On macOS/Linux:
```bash
# Using Homebrew (macOS)
brew install apache-spark

# Or download manually
wget https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz
tar -xvf spark-3.5.0-bin-hadoop3.tgz
sudo mv spark-3.5.0-bin-hadoop3 /opt/spark
```

### 4. Configure Java

Ensure JAVA_HOME is set:

```bash
# Windows
setx JAVA_HOME "C:\Program Files\Java\jdk-17"

# macOS/Linux
export JAVA_HOME=$(/usr/libexec/java_home)  # macOS
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk  # Linux
```

## 📁 Project Structure

```
spark-rdd-workshop/
│
├── atelier1.ipynb              # Main workshop notebook
├── requirements.txt            # Python dependencies
├── README.md                   # This file
├── LICENSE                     # MIT License
│
├── data/                       # Data files
│   ├── error_log.txt          # Log file for Exercise 1
│   ├── purchases.txt          # Sales data for Exercise 2
│   ├── README.md              # Spark documentation
│   └── CHANGES.txt            # Spark changelog
│
├── results/                    # Output directory
│   └── .gitkeep
│
└── screenshots/               # Visual documentation
    └── .gitkeep
```

## 📚 Exercises

### Exercise 1: Error Log Analysis

**Objective**: Filter and analyze error entries in server log files.

**Key Concepts**:
- Creating RDDs from text files
- Filtering operations
- Count and collect actions

**Code Snippet**:
```python
errors_rdd = rdd.filter(lambda line: "ERROR" in line)
error_count = errors_rdd.count()
```

**Expected Output**:
```
Nombre de lignes contenant 'ERROR' : 13
```

---

### Exercise 2: Sales Data Aggregation

**Objective**: Calculate total sales per store from transaction data.

**Key Concepts**:
- RDD partitioning
- Map-Reduce pattern
- Key-value pair operations
- Aggregation with reduceByKey

**Code Snippet**:
```python
total_par_magasin = rdd.map(lambda line: line.split('\t')) \
                       .map(lambda parts: (parts[2], float(parts[4]))) \
                       .reduceByKey(lambda a, b: a + b)
```

**Sample Output**:
```
Albuquerque: 3028079.62
Anaheim: 3085295.94
Cleveland: 3121222.68
Total général: 313270384.87
```

---

### Exercise 3: Distributed Word Count with Joins

**Objective**: Count word frequencies across multiple documents using RDD joins.

**Key Concepts**:
- Multiple RDD processing
- Text tokenization with regex
- Full outer joins
- RDD caching
- Sorting and ranking

**Code Snippet**:
```python
rdd_joint = word_count_readme.fullOuterJoin(word_count_changes)
rdd_joint.cache()
rdd_final = rdd_joint.mapValues(combiner_counts)
```

**Top Results**:
```
1. spark: 3918 occurrences
2. com: 3878 occurrences
3. pull: 2833 occurrences
```

## 🚀 Usage

### Running in Jupyter Notebook

```bash
# Start Jupyter
jupyter notebook

# Open atelier1.ipynb
# Run cells sequentially
```

### Running in Google Colab

1. Upload the notebook to Google Colab
2. Upload data files to Colab session
3. Install PySpark: `!pip install pyspark`
4. Run cells sequentially

### Running as Python Script

```bash
# Convert notebook to script
jupyter nbconvert --to script atelier1.ipynb

# Run the script
python atelier1.py
```

## 🛠️ Technologies

- **Apache Spark 3.5+**: Distributed computing framework
- **PySpark**: Python API for Spark
- **Jupyter Notebook**: Interactive development environment
- **Python 3.7+**: Programming language
- **Pandas**: Data manipulation (optional)
- **Matplotlib/Seaborn**: Visualization (optional)

## 📊 Results

### Performance Metrics

| Exercise | Dataset Size | Processing Time | RDD Partitions |
|----------|-------------|-----------------|----------------|
| Exercise 1 | ~100 lines | < 1 second | 2 |
| Exercise 2 | 4,139 records | ~2 seconds | 4 |
| Exercise 3 | 14,718 lines | ~3 seconds | Default |

### Key Findings

- **Exercise 2**: Total sales across 107 stores = $313,270,384.87
- **Exercise 3**: 4,090 unique words processed from documentation files
- **Caching Impact**: ~30% performance improvement on repeated operations

## 🤝 Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/AmazingFeature`)
3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request

### Guidelines

- Follow PEP 8 style guide for Python code
- Add comments for complex operations
- Update documentation for new features
- Include test cases when applicable

## 📝 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 👥 Authors

- **Moussa Dansoko** - *Initial work* - [Dansoko22md](https://github.com/Dansoko22md)

## 🙏 Acknowledgments

- Apache Spark community for excellent documentation
- Databricks for educational resources
- All contributors and workshop participants

## 📞 Contact

- GitHub: [@Dansoko22md](https://github.com/Dansoko22md)
- Email: dansokomoussa42@gmail.com
- LinkedIn: [Moussa Dansoko](https://www.linkedin.com/in/moussa-dansoko-a6a357172/)

## 🔗 Useful Links

- [Apache Spark Documentation](https://spark.apache.org/docs/latest/)
- [PySpark API Reference](https://spark.apache.org/docs/latest/api/python/)
- [RDD Programming Guide](https://spark.apache.org/docs/latest/rdd-programming-guide.html)
- [Spark Examples](https://github.com/apache/spark/tree/master/examples)

## 📈 Roadmap

- [ ] Add DataFrame API exercises
- [ ] Include Spark SQL examples
- [ ] Add streaming data processing
- [ ] Create visualization dashboard
- [ ] Add performance tuning section
- [ ] Include MLlib machine learning examples

---

⭐ **Star this repository if you find it helpful!** ⭐

**Happy Spark Learning! 🎓**

spark-rdd-workshop

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README