datafusion-ray - GitLite

KO | EN

gitlite — search

#javascript #python #hacktoberfest #typescript #android #machine-learning #docker #macos #awesome #cli #pytorch #mcp #openai #open-source

datafusion-ray

(★ 230)

Apache DataFusion Ray

File Explorer

Download Latest Version (.zip)

- config.toml
- - build.yml
  - k8s.yml
  - main.yml
- __init__.py
- core.py
- friendly.py
- util.py
- - check-rat-report.py
  - create-tarball.sh
  - generate-changelog.py
  - rat_exclude_files.txt
  - README.md
  - release-tarball.sh
  - run-rat.sh
  - verify-release-candidate.sh
- create_license.py
- contributing.md
- README.md
- http_csv.py
- tips.py
- bench_toolbox.py
- cmds.py
- machine_prep.sh
- pricing.py
- pvcs.yaml.template
- ray_cluster.yaml.template
- ray_job.sh.template
- README.md
- requirements.txt.template
- spark_job.yaml.template
- spark_tpcbench.py
- - datafusion.proto
  - datafusion_common.proto
  - datafusion_ray.proto
  - mod.rs
- codec.rs
- context.rs
- dataframe.rs
- flight.rs
- isolator.rs
- lib.rs
- max_rows.rs
- physical.rs
- pre_fetch.rs
- processor_service.rs
- stage.rs
- stage_reader.rs
- util.rs
- - tips.parquet
- - .gitignore
- - q1.sql
  - q10.sql
  - q11.sql
  - q12.sql
  - q13.sql
  - q14.sql
  - q15.sql
  - q16.sql
  - q17.sql
  - q18.sql
  - q19.sql
  - q2.sql
  - q20.sql
  - q21.sql
  - q22.sql
  - q3.sql
  - q4.sql
  - q5.sql
  - q6.sql
  - q7.sql
  - q8.sql
  - q9.sql
- make_data.py
- requirements.txt
- tpcbench.py
.asf.yaml
.dockerignore
.gitignore
.pre-commit-config.yaml
build.rs
Cargo.lock
Cargo.toml
LICENSE.txt
NOTICE
pyproject.toml
README.md

# Installation Guide

1. Get the code

git clone https://github.com/apache/datafusion-ray

Downloads the entire project code from GitHub to your computer.

cd datafusion-ray

Moves into the project folder you just downloaded.

2. Python

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r tpch/requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

Runs the Python script (or module).

✅ If it runs without errors and prints output in the terminal, it worked.

3. Rust

Medium

Prerequisites

Git Needed to download the project code from GitHub.
Rust (rustup) Installing via rustup also installs cargo.

cargo build --release

Compiles the Rust project.

cargo run

Builds and then immediately runs the program.

✅ If cargo build finishes without errors, it worked. The executable is created under target/.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

# DataFusion for Ray

[![Apache licensed][license-badge]][license-url]
[![Python Tests][actions-badge]][actions-url]
[![Discord chat][discord-badge]][discord-url]

[license-badge]: https://img.shields.io/badge/license-Apache%20v2-blue.svg
[license-url]: https://github.com/apache/datafusion-ray/blob/main/LICENSE.txt
[actions-badge]: https://github.com/apache/datafusion-ray/actions/workflows/main.yml/badge.svg
[actions-url]: https://github.com/apache/datafusion-ray/actions?query=branch%3Amain
[discord-badge]: https://img.shields.io/badge/Chat-Discord-purple
[discord-url]: https://discord.com/invite/Qw5gKqHxUM

> ⚠️ **WARNING:** This project is no longer maintained.

## Overview

DataFusion for Ray is a distributed execution framework that enables DataFusion DataFrame and SQL queries to run on a
Ray cluster. This integration allows users to leverage Ray's dynamic scheduling capabilities while executing
queries in a distributed fashion.

## Execution Modes

DataFusion for Ray supports two execution modes:

### Streaming Execution

This mode mimics the default execution strategy of DataFusion. Each operator in the query plan starts executing
as soon as its inputs are available, leading to a more pipelined execution model.

### Batch Execution

_Note: Batch Execution is not implemented yet. Tracking issue: <https://github.com/apache/datafusion-ray/issues/69>_

In this mode, execution follows a staged model similar to Apache Spark. Each query stage runs to completion, producing
intermediate shuffle files that are persisted and used as input for the next stage.

## Getting Started

See the [contributor guide] for instructions on building DataFusion for Ray.

Once installed, you can run queries using DataFusion's familiar API while leveraging the distributed execution
capabilities of Ray.

```python
# from example in ./examples/http_csv.py
import ray
from datafusion_ray import DFRayContext, df_ray_runtime_env

ray.init(runtime_env=df_ray_runtime_env)

ctx = DFRayContext()
ctx.register_csv(
    "aggregate_test_100",
    "https://github.com/apache/arrow-testing/raw/master/data/csv/aggregate_test_100.csv",
)

df = ctx.sql("SELECT c1,c2,c3 FROM aggregate_test_100 LIMIT 5")

df.show()
```

## Contributing

Contributions are welcome! Please open an issue or submit a pull request if you would like to contribute. See the
[contributor guide] for more information.

## License

DataFusion for Ray is licensed under Apache 2.0.

[contributor guide]: docs/contributing.md