datafusion-federation - GitLite

KO | EN

gitlite — search

#react #ai #llm #android #docker #macos #nodejs #windows #awesome #kubernetes #vue #security #ios #open-source

datafusion-federation

(★ 193)

Allow DataFusion to resolve queries across remote query engines while pushing down as much compute as possible down.

파일 탐색기

최종 버전 다운로드 (.zip)

- - docs.yml
  - release_plz.yml
  - test.yml
- dependabot.yml
- - df-csv-advanced.rs
  - df-csv.rs
- - lib.rs
  - plan_node.rs
  - table_provider.rs
- - federation_pushdown.rs
- Cargo.toml
- CHANGELOG.md
.gitignore
.release-plz.toml
Cargo.toml
LICENSE
README.md
rust-toolchain.toml

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/datafusion-contrib/datafusion-federation

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd datafusion-federation

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Rust

보통 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Rust (rustup) rustup으로 설치하면 cargo도 함께 설치됩니다.

cargo build --release

Rust 프로젝트를 컴파일합니다.

cargo run

빌드 후 바로 실행까지 진행합니다.

✅ cargo build가 에러 없이 끝나면 성공입니다. target/ 폴더에 실행 파일이 생성됩니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DataFusion Federation

[![crates.io](https://img.shields.io/crates/v/datafusion-federation.svg)](https://crates.io/crates/datafusion-federation)
[![docs.rs](https://docs.rs/datafusion-federation/badge.svg)](https://docs.rs/datafusion-federation)

DataFusion Federation allows
[DataFusion](https://github.com/apache/arrow-datafusion) to execute (part of) a
query plan by a remote execution engine.

┌────────────────┐
                   ┌────────────┐       │ Remote DBMS(s) │
    SQL Query ───> │ DataFusion │  ───> │  ( execution   │
                   └────────────┘       │ happens here ) │
                                        └────────────────┘

The goal is to allow resolving queries across remote query engines while
pushing down as much compute as possible to the remote database(s). This allows
execution to happen as close to the storage as possible. This concept is
referred to as 'query federation'.

> [!TIP]
> This repository implements the federation framework itself. If you want to
> connect to a specific database, check out the compatible providers available
> in
> [datafusion-contrib/datafusion-table-providers](https://github.com/datafusion-contrib/datafusion-table-providers/).

## Usage

Check out the [examples](./datafusion-federation/examples/) to get a feel for
how it works.

For a complete step-by-step example of how federation works, you can check the
example [here](./datafusion-federation/examples/df-csv-advanced.rs).

## Potential use-cases:

- Querying across SQLite, MySQL, PostgreSQL, ...
- Pushing down SQL or [Substrait](https://substrait.io/) plans.
- DataFusion -> Flight SQL -> DataFusion
- ..

## Design concept

Say you have a query plan as follows:

┌────────────┐
                   │    Join    │
                   └────────────┘
                          ▲
                  ┌───────┴────────┐
           ┌────────────┐   ┌────────────┐
           │   Scan A   │   │    Join    │
           └────────────┘   └────────────┘
                                   ▲
                           ┌───────┴────────┐
                    ┌────────────┐   ┌────────────┐
                    │   Scan B   │   │   Scan C   │
                    └────────────┘   └────────────┘

DataFusion Federation will identify the largest possible sub-plans that
can be executed by an external database:

┌────────────┐      Optimizer recognizes
                   │    Join    │      that B and C are
                   └────────────┘      available in an
                          ▲            external database
           ┌──────────────┴────────┐
           │       ┌ ─  ─ ─ ─  ─ ─ ┴ ─ ── ─ ─ ─  ─ ─┐
    ┌────────────┐          ┌────────────┐          │
    │   Scan A   │ │        │    Join    │
    └────────────┘          └────────────┘          │
                   │               ▲
                           ┌───────┴────────┐       │
                    ┌────────────┐   ┌────────────┐ │
                   ││   Scan B   │   │   Scan C   │
                    └────────────┘   └────────────┘ │
                    ─ ── ─ ─ ── ─ ─ ─ ─  ─ ─ ─ ── ─ ┘

The sub-plans are cut out and replaced by an opaque federation node in the plan:

┌────────────┐
                   │    Join    │
                   └────────────┘    Rewritten Plan
                          ▲
                 ┌────────┴───────────┐
                 │                    │
          ┌────────────┐    ┏━━━━━━━━━━━━━━━━━━┓
          │   Scan A   │    ┃     Scan B+C     ┃
          └────────────┘    ┃  (TableProvider  ┃
                            ┃ that can execute ┃
                            ┃ sub-plan in an   ┃
                            ┃external database)┃
                            ┗━━━━━━━━━━━━━━━━━━┛

Different databases may have different query languages and execution
capabilities. To accommodate for this, we allow each 'federation provider' to
self-determine what part of a sub-plan it will actually federate. This is done
by letting each federation provider define its own optimizer rule. When a
sub-plan is 'cut out' of the overall plan, it is first passed the federation
provider's optimizer rule. This optimizer rule determines the part of the plan
that is cut out, based on the execution capabilities of the database it
represents.

## Implementation

A remote database is represented by the `FederationProvider` trait. To identify
table scans that are available in the same database, they implement
`FederatedTableSource` trait. This trait allows lookup of the corresponding
`FederationProvider`.

Identifying sub-plans to federate is done by the `FederationOptimizerRule`.
This rule needs to be registered in your DataFusion SessionState. One easy way
to do this is using `default_session_state`. To do its job, the
`FederationOptimizerRule` currently requires that all TableProviders that need
to be federated are `FederatedTableProviderAdaptor`s. The
`FederatedTableProviderAdaptor` also has a fallback mechanism that allows
implementations to fallback to a 'vanilla' TableProvider in case the
`FederationOptimizerRule` isn't registered.

The `FederationProvider` can provide a `compute_context`. This allows it to
differentiate between multiple remote execution context of the same type. For
example two different mysql instances, database schemas, access level, etc. The
`FederationProvider` also returns the `Optimizer` that is allows it to
self-determine what part of a sub-plan it can federate.

The `sql` module implements a generic `FederationProvider` for SQL execution
engines. A specific SQL engine implements the `SQLExecutor` trait for its
engine specific execution. There are a number of compatible providers available
in
[datafusion-contrib/datafusion-table-providers](https://github.com/datafusion-contrib/datafusion-table-providers/).

## Status

The project is in alpha status. Contributions welcome; land a PR = commit
access.

- [Docs (release)](https://docs.rs/datafusion-federation)
- [Docs (main)](https://datafusion-contrib.github.io/datafusion-federation/)