spark-docker

KO | EN

gitlite — search

#javascript #ai #typescript #llm #golang #android #machine-learning #deep-learning #linux #docker #java #kubernetes #mcp #openai

(★ 17)

Esay-to-use spark-docker

spark-docker 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# spark-docker
Esay-to-use spark-docker(Spark 3.5.7), better deployed using docker-compose.yaml, good practices for **local development**, **testing** and **learning**.

I made this cause bitnami **no longer** provides free Spark docker image on docker hub.

**Three** modes provided:
- master
- worker
- thrift(Specially for kafka, dbt and Delta Lake)

# Reminder
- Use `user: "${UID}:${GID}"` in `docker-compose.yml` to avoid permission problem.

### Read-only environment variables

# Usage

### docker
```bash
docker pull ghcr.io/datu1213/spark-docker:3.5.7
```

### docker-compose.yaml example
```yaml
spark-master:
    image: ghcr.io/datu1213/spark-docker:3.5.7
    environment:
      SPARK_MODE: master
    ports:
      - "7077:7077"
      - "8080:8080"
    volumes:
      - ./conf/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf

spark-worker:
  image: ghcr.io/datu1213/spark-docker:3.5.7
  depends_on:
    - spark-master
  environment:
    SPARK_MODE: worker
    # Set SPARK_MASTER_URL if you have customized url.
    SPARK_MASTER_URL: spark://spark-master:7077
    SPARK_WORKER_CORES: 4
    SPARK_WORKER_MEMORY: 8g
    SPARK_WORKER_WEBUI_PORT: 8082
  ports:
    # Should be as same as env $SPARK_WORKER_WEBUI_PORT
    - "8082:8082"
  volumes:
    - ./ivy2:/opt/spark/.ivy2
    - ./spark-data:/opt/spark/data
    # Defaul offical Apache Spark work-dir is /opt/spark/work-dir
    - ./spark-warehouse:/opt/spark/work-dir/warehouse
    - ./metastore_db:/opt/spark/work-dir/metastore_db
    - ./conf/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf

spark-thrift:
  image: ghcr.io/datu1213/spark-docker:3.5.7
  depends_on:
    - spark-master
    - spark-worker
  environment:
    SPARK_MODE: thrift
    SPARK_MASTER_URL: spark://spark-master:7077
  volumes:
    - ./ivy2:/opt/spark/.ivy2
    - ./conf/spark-defaults.conf:/opt/spark/conf/spark-defaults.conf
```

### spark-defaults.conf
An spark-defaults.conf file s provided for kafka, dbt, and delta lake.

### More

It is recommended to use an indepentend jupyter-pyspark server or dbt server.

spark-docker

spark-docker 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

spark-docker

spark-docker 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README