pyspark-project-example

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #rust #deep-learning #nodejs #java #cli #vue #awesome-list #ios #openai #open-source

pyspark-project-example

(★ 11)

A simple example for PySpark based project.

pyspark-project-example 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# pyspark-project-example [![Build Status](https://travis-ci.org/HyukjinKwon/pyspark-project-example.svg?branch=master)](https://travis-ci.org/HyukjinKwon/pyspark-project-example) [![codecov.io](https://codecov.io/github/HyukjinKwon/pyspark-project-example/coverage.svg?branch=master)](https://codecov.io/github/HyukjinKwon/pyspark-project-example?branch=master)

A simple example for PySpark based project.

## Prerequisites

- JDK (>=7) (as the local test downloads Spark 1.6.0, which dropped JDK 6 support)

- Python PEP 8 (for checking code-style in local test)

```bash
  pip install pep8
  ```
- Python Py4J == 0.9.0 (for initiating Spark in local test)

```bash
  pip install -r requirements.txt
  ```

## Test

- For test in your local, just run below:

```
    ./dev/run_tests
    ```

This contains some codes to download Spark 1.6.0 to run tests. So, it is okay to run the script without some specific modifications.

## Warnings

- As currently PySpark cannot be installed via `pip install` due to [this issue](https://issues.apache.org/jira/browse/SPARK-1267),
Spark has to be installed and `SPARK_HOME="your Spark path"` and `PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH` should be set in order to use this library outside after building.

pyspark-project-example

pyspark-project-example 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

pyspark-project-example

pyspark-project-example 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README