Spark-And-MLlib-Projects

KO | EN

gitlite — search

#javascript #golang #android #machine-learning #deep-learning #linux #macos #java #nodejs #kubernetes #vue #pytorch #security #mcp

Spark-And-MLlib-Projects

(★ 49)

This repository contains Spark, MLlib, PySpark and Dataframes projects

Spark-And-MLlib-Projects Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Spark-Projects
Apache Spark is a fast and general-purpose cluster computing system. Designed for large-scale data processing, it run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk. Spark can run on Hadoop, Mesos, standalone, or in the cloud. It can access diverse data sources including HDFS, Cassandra, HBase, and S3. Spark has an advanced DAG execution engine that supports acyclic data flow and in-memory computing. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. You can learn more about spark at their quick start guide [here.](http://spark.apache.org/docs/latest/quick-start.html)

## About
This repository contains the projects and exercises done by me using Spark on Python. Each folder has the code as well as data associated with that project. All the code should be executable as long as the computer meets the requirements mentioned in the dependencies section.

## Dependencies
To execute my projects you will need a system that satisfies below dependencies. These projects were done on Linux machine so you can use Linux Ubuntu, AWS EC2, AWS EMR (Elastic MapReduce) or any cluster distributed computing environment that has spark.
- Python 3.5
- Spark 2.1
- Scala
- Java
- Linux(Ubuntu, AWS EC2, AWS EMR, Databricks Notebook)

## Projects
### Spark DataFrame API
- The code for spark DataFrameAPI can be found [here.](https://github.com/jubins/Spark-Projects/blob/master/Spark_DataFrame_API_Project/Spark_DataFrames_API.ipynb)

### Walmart Data Analysis on Spark
- The code and project for Walmart data analysis can be found [here.](https://github.com/jubins/Spark-Projects/blob/master/Spark_Walmart_Data_Analysis_Project/Spark%20Walmart%20Data%20Analysis%20Project.ipynb)

## More Information 
- More information about PySpark and programming Spark using Python can be found [here.](https://spark.apache.org/docs/0.9.1/python-programming-guide.html)
- More information about Spark can be found [here.](http://spark.apache.org/)

### Spark Cluster Overview
![Alt](https://spark.apache.org/docs/latest/img/cluster-overview.png)

### Spark Architechture
![Alt](https://media.licdn.com/mpr/mpr/shrinknp_800_800/AAEAAQAAAAAAAAb5AAAAJDUwZGZmZWEwLWNhZGEtNDc4NC1hOTkyLTVjMTNiYmUzNjVmNw.png)

## References
- [Apache Spark](https://spark.apache.org/)
- [Media Licdn](https://media.licdn.com/)
- [Pieraian Data](http://www.pieriandata.com/)
- [SparkSQL Tutorial](https://www.tutorialspoint.com/spark_sql/spark_sql_quick_guide.htm)

Spark-And-MLlib-Projects

Spark-And-MLlib-Projects Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Spark-And-MLlib-Projects

Spark-And-MLlib-Projects Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README