h2o-sparkling

KO | EN

gitlite — search

#javascript #python #typescript #llm #golang #go #linux #java #nodejs #awesome #vue #ios #mcp #openai

(★ 44)

DEPRECATED! Use https://github.com/h2oai/sparkling-water repository! H2O and Spark interoperability based on Tachyon.

h2o-sparkling Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# This repository is DEPRECATED! Please use the new Sparkling Water repository https://github.com/h2oai/sparkling-water!
-------------------

h2o-sparkling
=============

Makes interoperability between H2O and Spark trivial.

## Requirements
  - Spark 1.0.0 (SQL component required)
  - Tachyon 0.4.1
  - Java 1.6+

## Installation

- First compile latest version of spark with SQL component
  ```
  git clone spark
  cd spark
  sbt/sbt assembly publish-local
  ```
    
  - For Tachyon support please download Tachyon 0.4.1 from https://github.com/amplab/tachyon/releases/tag/v0.4.1
  
  - Compile sparkling demo
  ```
  cd h2o-sparkling-demo
  sbt assembly
  ```

> Note: The assembly stage is important, since the demo is a Spark driver sending a jar-file containing implementation of a working job.

## Run demo

### Run local version
For this run no Spark cloud is required:
  - Execute an instance of H2O embedding Spark driver
  ```
  cd h2o-sparkling-demo
  sbt "run --local"
  ```

### Run distributed version
For this run a Spark cloud is required:
  - Run master and one worker on local node
  ```
  cd spark/sbin
  ./start-master.sh
  ./start-slave.sh 1 "spark://localhost:7077"
  ```

- Assembly h2o-sparkling-demo jar file which can be sent by the driver to Spark cloud
  ```
  cd h2o-sparkling-demo
  sbt assembly
  sbt "run --remote"
  ```

### Run additional H2O node
```
cd h2o-sparkling-demo
sbt runH2O
```

### Select different RDD2Frame extractor

Currently demo supports three extractors:

- _dummy_ - pull all data into driver and create a frame
   - _file_ - ask Spark to save RDD as a file on local filesystem and then parse a stored file
   - _tachyon_ - ask Spark to save RDD to tachyon filesystem, then H2O load a file from tachyon FS

The extractor can be selected via `--extractor` command line parameter, e.g., `--extractor==tachyon`

## Running with Tachyon
  - Start Tachyon
  ```
  cd tachyon/bin
  ./tachyon-start.sh
  ```

- Look at http://localhost:19999/ to see list of files stored on the storage or type _tfs_ command `tachyon tfs ls /`

- For more info details discuss instructions instructions on http://tachyon-project.org/Running-Spark-on-Tachyon.html

## Example

Run a demo with Tachyon-based extractor againts remote Spark cloud:
 ```
 cd h2o-sparkling-demo
 sbt assembly
 sbt "run --remote --extractor=tachyon"
 ```
 
 
 Run airlines demo with file-based extractor againts remote Spark cloud running on non-default location:
 ```
 sbt "run --remote --sparkMaster=spark://localhost:17077 --noshutdown --demo=airlines --extractor=file"
 ```

## Doc

- Matei Zaharia' slides about Spark
    - http://ampcamp.berkeley.edu/wp-content/uploads/2012/06/matei-zaharia-part-2-amp-camp-2012-standalone-programs.pdf

h2o-sparkling

h2o-sparkling Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

h2o-sparkling

h2o-sparkling Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README