datafusion-ray - GitLite

KO | EN

gitlite — search

#javascript #python #hacktoberfest #go #android #machine-learning #rust #docker #windows #awesome #cli #kubernetes #ios #open-source

datafusion-ray

(★ 230)

Apache DataFusion Ray

파일 탐색기

최종 버전 다운로드 (.zip)

- config.toml
- - build.yml
  - k8s.yml
  - main.yml
- __init__.py
- core.py
- friendly.py
- util.py
- - check-rat-report.py
  - create-tarball.sh
  - generate-changelog.py
  - rat_exclude_files.txt
  - README.md
  - release-tarball.sh
  - run-rat.sh
  - verify-release-candidate.sh
- create_license.py
- contributing.md
- README.md
- http_csv.py
- tips.py
- bench_toolbox.py
- cmds.py
- machine_prep.sh
- pricing.py
- pvcs.yaml.template
- ray_cluster.yaml.template
- ray_job.sh.template
- README.md
- requirements.txt.template
- spark_job.yaml.template
- spark_tpcbench.py
- - datafusion.proto
  - datafusion_common.proto
  - datafusion_ray.proto
  - mod.rs
- codec.rs
- context.rs
- dataframe.rs
- flight.rs
- isolator.rs
- lib.rs
- max_rows.rs
- physical.rs
- pre_fetch.rs
- processor_service.rs
- stage.rs
- stage_reader.rs
- util.rs
- - tips.parquet
- - .gitignore
- - q1.sql
  - q10.sql
  - q11.sql
  - q12.sql
  - q13.sql
  - q14.sql
  - q15.sql
  - q16.sql
  - q17.sql
  - q18.sql
  - q19.sql
  - q2.sql
  - q20.sql
  - q21.sql
  - q22.sql
  - q3.sql
  - q4.sql
  - q5.sql
  - q6.sql
  - q7.sql
  - q8.sql
  - q9.sql
- make_data.py
- requirements.txt
- tpcbench.py
.asf.yaml
.dockerignore
.gitignore
.pre-commit-config.yaml
build.rs
Cargo.lock
Cargo.toml
LICENSE.txt
NOTICE
pyproject.toml
README.md

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/apache/datafusion-ray

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd datafusion-ray

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r tpch/requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

3. Rust

보통

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Rust (rustup) rustup으로 설치하면 cargo도 함께 설치됩니다.

cargo build --release

Rust 프로젝트를 컴파일합니다.

cargo run

빌드 후 바로 실행까지 진행합니다.

✅ cargo build가 에러 없이 끝나면 성공입니다. target/ 폴더에 실행 파일이 생성됩니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

# DataFusion for Ray

[![Apache licensed][license-badge]][license-url]
[![Python Tests][actions-badge]][actions-url]
[![Discord chat][discord-badge]][discord-url]

[license-badge]: https://img.shields.io/badge/license-Apache%20v2-blue.svg
[license-url]: https://github.com/apache/datafusion-ray/blob/main/LICENSE.txt
[actions-badge]: https://github.com/apache/datafusion-ray/actions/workflows/main.yml/badge.svg
[actions-url]: https://github.com/apache/datafusion-ray/actions?query=branch%3Amain
[discord-badge]: https://img.shields.io/badge/Chat-Discord-purple
[discord-url]: https://discord.com/invite/Qw5gKqHxUM

> ⚠️ **WARNING:** This project is no longer maintained.

## Overview

DataFusion for Ray is a distributed execution framework that enables DataFusion DataFrame and SQL queries to run on a
Ray cluster. This integration allows users to leverage Ray's dynamic scheduling capabilities while executing
queries in a distributed fashion.

## Execution Modes

DataFusion for Ray supports two execution modes:

### Streaming Execution

This mode mimics the default execution strategy of DataFusion. Each operator in the query plan starts executing
as soon as its inputs are available, leading to a more pipelined execution model.

### Batch Execution

_Note: Batch Execution is not implemented yet. Tracking issue: <https://github.com/apache/datafusion-ray/issues/69>_

In this mode, execution follows a staged model similar to Apache Spark. Each query stage runs to completion, producing
intermediate shuffle files that are persisted and used as input for the next stage.

## Getting Started

See the [contributor guide] for instructions on building DataFusion for Ray.

Once installed, you can run queries using DataFusion's familiar API while leveraging the distributed execution
capabilities of Ray.

```python
# from example in ./examples/http_csv.py
import ray
from datafusion_ray import DFRayContext, df_ray_runtime_env

ray.init(runtime_env=df_ray_runtime_env)

ctx = DFRayContext()
ctx.register_csv(
    "aggregate_test_100",
    "https://github.com/apache/arrow-testing/raw/master/data/csv/aggregate_test_100.csv",
)

df = ctx.sql("SELECT c1,c2,c3 FROM aggregate_test_100 LIMIT 5")

df.show()
```

## Contributing

Contributions are welcome! Please open an issue or submit a pull request if you would like to contribute. See the
[contributor guide] for more information.

## License

DataFusion for Ray is licensed under Apache 2.0.

[contributor guide]: docs/contributing.md