SparkJavaExamples

KO | EN

gitlite — search

#python #react #ai #golang #rust #deep-learning #linux #java #kubernetes #awesome-list #vue #security #openai #open-source

SparkJavaExamples

(★ 9)

Apache Spark Basics - Java Examples

SparkJavaExamples Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Spark Java Examples

This project is created to learn Apache Spark Programming using Java. This project consists of the following examples:

* How to create SparkContext and SparkSession. 
  * Taking data from arrays and external file source. 
  * Spark Map Transformation. 
  * Spark Filter Transformation. 
  * Spark FlatMap Transformation. 
  * Compare Map and FlatMap. 
  * Set Operations. 
  * Spark Reduce Transformation. 
  * Spark Aggregate Transformation. 
  * Using Functions in Spark Transformation. 
  * Key Value RDD.
  * Using HDFS
 
### Data Sets
 * cars.csv - A data set with many attributes of various car models.	
 * Some random array data.

### Getting Started

These instructions will get you a brief idea on setting up the environment and running on your local machine for development and testing purposes.

**Prerequisities**

- Java
- Apache Spark 
- Hadoop

**Setup and running tests**

1. Run `javac` and `java -version` to check the installation
   
2. Run `spark-shell` and check if Spark is installed properly. 
 
3. Go to Hadoop user (If  installed on different user) and run the following (On Ubuntu Systems):

`sudo su hadoopuser`
          
      `start-all.sh`   
           
4. Execute the following commands from terminal to run the tests:

`javac -classpath "Path to required jar files(spark, hadoop, scala)" Main.java`

###Classes
Please start exploring from Main.java

All classes in this project are listed below:

* **CreateSpark.java** - To create SparkContext and SparkSession. Contains the following methods:
	
      	  `public JavaSparkContext context(String appName, String master)`
      	  `public SparkSession session(String appName, String master)`
	
* **ArrayData.java** - Using array data to create JavaRDD and performs spark actions on it. Contains the following method:
	
      	  `public void callArrayData()`
	
* **ExternalFileData.java** - Using external file source to create JavaRDD and performs spark actions on it. Contains the following method:

`public void callFileData(String filePath)`
	
* **SparkMap.java** - Example code on using Spark Map Transformation, contains the following method:
	 
      	  `public void mapReplace(String arg0, String arg1)`
	 
* **SparkFilter.java** - Example code on using Spark Filter Transformation, contains the following method:
	
	  `public void callFilter(String str)`
		
* **SparkFlatMap.java** - Example code on using Spark FlatMap Transformation, contains the following method:
	
	  `public void callFlatMap()`
	
* **CompareMapAndFlatMap.java** - To compare and understand Map and FlapMap Transformations. Contains the following method:
	
	  `public void compare()`
	
* **SetOperations.java** - Performing set operations on JavaRDD. Contains the following method: 
	
	  `public void callSetOp()`
	 
* **Reduce.java** - Examples on Spark Reduce Transformation. Contains the following methods:
	
	  `public void sum()`	
	  `public void shortestLine()`

* **Aggregation.java** - Uses two different use cases of Spark Aggregate Transformation. Contains the following methods:
	
	  `public void sum()`	
	  `public void sumAndProduct()` 
	
* **Functions.java** - Using Functions in Spark Transformation. Contains the following methods:
	
	  `public static void example1(JavaSparkContext sparkContext)`
	  `public static void example2(JavaSparkContext sparkContext)`

* **KeyValueRDD.java** - Examples on using Key Value RDD. Contains the following method:
	
	  `public void callKVRDD()`

* **UsingHDFS.java** - Example on using HDFS in Spark Programming. Contains the following methods:
	
	  `public <T> void saveToHDFS(JavaRDD<T> hdfsData, String path)` 
	  `public JavaRDD<String> readHDFS(String filePath)`
	
* **Main.java** - Main class to test and run the classes in this project.

SparkJavaExamples

SparkJavaExamples Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

SparkJavaExamples

SparkJavaExamples Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README