Flowfile

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #claude-code #codex #cli #developer-tools #react #windows

★ 341 Open GitHub ↗

Flowfile is a visual ETL tool and Python library combining drag-and-drop workflows with Polars dataframes. Build data pipelines visually, define flows programmatically with a Polars-like API, and export to standalone Python code. Perfect for fast, intuitive data processing from development to production.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">
  <img src=".github/images/logo.png" alt="Flowfile Logo" width="100">
  <br>
  Flowfile
</h1>

<p align="center">
  <b>Visual ETL that compiles to Polars.</b>
  <br>
  <sub>Build pipelines on a canvas, run them locally or in the browser, export them as standalone Python.<br>No platform lock-in. No install required to try it.</sub>
</p>

<p align="center">
  <a href="https://demo.flowfile.org"><b>▶&nbsp;&nbsp;Try it in your browser&nbsp;&nbsp;→</b></a>
  <br>
  <sub>No install. No signup. Polars in the browser via Pyodide.</sub>
</p>

<p align="center">
  <a href="https://edwardvaneechoud.github.io/Flowfile/">Docs</a> ·
  <a href="https://github.com/Edwardvaneechoud/Flowfile/releases">Releases</a> ·
  <a href="https://github.com/Edwardvaneechoud/Flowfile/discussions">Discussions</a> ·
  <a href="https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c">Architecture deep-dive</a>
</p>

---

Build pipelines on a visual canvas, run them, and export the graph as plain Python code that runs anywhere. Code and visual are two views of the same graph: drag nodes or write Python with a Polars-like API, your choice.

Beyond the canvas: a Delta-backed catalog with time-travel and virtual tables, a SQL editor with embedded viz, flow parameters, sandboxed Python kernels, and a built-in scheduler.

<div align="center">
  <img src="docs/assets/images/ai/ai-overview.gif" alt="Flowfile AI assistant building a pipeline on the canvas" width="800"/>
  <br>
  <sub>The AI assistant building a pipeline on the canvas — describe what you want, get a runnable flow.</sub>
</div>

&nbsp;

---

## What's in Flowfile

**A visual canvas** with 40+ node types — joins, fuzzy matching, filters, pivots, aggregations, text-to-rows, and more. Read from local files, databases (PostgreSQL, MySQL, SQL Server, Oracle), cloud storage (S3, ADLS, GCS), or Kafka. Write the result wherever you want. Beyond the nodes, the formula editor brings 100+ transformation functions, and a Polars code node gives you full Polars support — all running in-process, no external engine.

<div align="center">
  <img src=".github/images/superstore_demo.gif" alt="Building a Superstore pipeline on the canvas — filter, join, pivot, and aggregate" width="800"/>
  <br>
  <sub>Building a Superstore pipeline on the canvas — filters, a join, a pivot, and aggregations, with a live data preview updating at every step.</sub>
</div>

&nbsp;

**A Python API** with Polars-like syntax. Code and visual are two ways to build the same object graph — write a pipeline, call `open_graph_in_editor()`, and see it visually without re-building anything.

<div align="center">
  <img src="docs/assets/images/quickstart/python_example.png" alt="A pipeline built with the Flowfile Python API, opened in the visual editor" width="800"/>
  <br>
  <sub>A pipeline written with the Python API and opened in the visual editor — code and canvas are two views of one graph.</sub>
</div>

&nbsp;

**Code generation.** Export any visual flow as Python code. For pipelines built from standard transformations (joins, filters, aggregations, formulas, etc.), you get pure Polars code with no Flowfile dependency. For flows using Flowfile-specific nodes — the catalog, Kafka sources, virtual table reads — the export uses Flowfile's Python API instead, since there's no direct Polars equivalent. Flows also save as human-readable YAML, so version control works.

<div align="center">
  <img src=".github/images/generated_code.png" alt="Export visual flows as standalone Polars code" width="800"/>
  <br>
  <sub>Every visual flow exports as a standalone Python script — toggle between Polars and FlowFrame output.</sub>
</div>

&nbsp;

**A data catalog.** Unity-style hierarchy (catalog > schema > table), Delta Lake-backed with version history and time travel. Flows register into namespaces and write output through a Catalog Writer node.

**Virtual flow tables.** Flow outputs can live in the catalog without being materialized. If the producer graph is lazy-safe, Flowfile serializes the Polars LazyFrame and filter/projection pushdown crosses the flow boundary. Upstream Delta versions are tracked per read, so stale data doesn't ship.

**A SQL editor** on top of the catalog (Polars SQLContext). Query any registered table, visualize the result in an embedded Graphic Walker, save any ad-hoc query as a reusable flow in one click.

<div align="center">
  <img src=".github/images/sql_editor.png" alt="SQL editor with Graphic Walker visualization" width="800"/>
  <br>
  <sub>SQL queries run against catalog tables, with results feeding into Graphic Walker for visual exploration.</sub>
</div>

&nbsp;

**A scheduler.** Run flows on an interval, trigger when a catalog table updates, or fire when a set of tables has all refreshed. Run history, logs, and cancellation live in the UI. Runs embedded, standalone, or in Docker.

**Flow parameters.** Parameterize any node setting using `${variable}` syntax — file paths, SQL queries, formulas. Manage defaults from a Designer panel, override at runtime via CLI with `--param`.

**Python Kernels.** Run user code in isolated Docker containers with their own package environments, keeping the host process safe. Jupyter-style notebook editor with cell execution, autocompletions, and rich display output (matplotlib, plotly, PIL, HTML).

**Custom nodes and a community registry.** Build your own nodes in the visual Node Designer — a typed settings form, live preview, and a single-file `.py` output — then share them through the [community registry](https://github.com/edwardvaneechoud/flowfile-community-nodes). Publishing is a pull request opened straight from the app; installing is one click from Catalog → Community Nodes, with sha256-pinned downloads and a capability consent dialog.

**Templates and clipboard import.** Get started with built-in flow templates, or paste tabular data from Excel / Google Sheets directly onto the canvas to create a pre-filled input node.

**See it all together.** A single flow can filter, join, pivot, and aggregate, then branch into as many outputs as you need.

<div align="center">
  <img src="docs/assets/images/guides/sales_dashboard/dashboard_overview.png" alt="A complete Flowfile pipeline with joins, pivots, and aggregations feeding multiple outputs" width="800"/>
  <br>
  <sub>One flow, many outputs — joins, pivots, and aggregations feeding a product leaderboard, a monthly trend, and a city matrix.</sub>
</div>

---

## Quick Start

**Try it in your browser** (no install, 20+ nodes, runs entirely on Pyodide): [demo.flowfile.org](https://demo.flowfile.org)

**Embed the editor in your own app** — the browser editor also ships as a standalone Vue component, [`flowfile-editor`](https://www.npmjs.com/package/flowfile-editor), so you can drop a Polars-powered visual ETL canvas into any web app with zero backend:

```bash
npm install flowfile-editor
```

**Python package** — the fastest way to run the full thing locally:

```bash
pip install Flowfile
flowfile run ui
```

**Use the Python API:**

```python
import flowfile as ff
from flowfile import col, open_graph_in_editor

df = ff.from_dict({
    "id": [1, 2, 3, 4, 5],
    "category": ["A", "B", "A", "C", "B"],
    "value": [100, 200, 150, 300, 250]
})

result = (
    df.filter(col("value") > 150)
      .with_columns((col("value") * 2).alias("double_value"))
      .group_by("category")
      .agg(col("value").sum().alias("total"))
)

open_graph_in_editor(result.flow_graph)
```

---

## Other Ways to Run It

**Desktop app** — Windows, macOS, or Linux. Download from [Releases](https://github.com/Edwardvaneechoud/Flowfile/releases).

<details>
<summary><b>Docker</b> — full stack via Docker Compose</summary>

```bash
git clone https://github.com/edwardvaneechoud/Flowfile.git
cd Flowfile
docker compose up -d
```

Access at http://localhost:8080.

</details>

<details>
<summary><b>From source</b> — for contributors (Python 3.10+, Node.js 20+)</summary>

```bash
git clone https://github.com/edwardvaneechoud/Flowfile.git
cd Flowfile
poetry install

# Backend (two separate terminals)
poetry run flowfile_worker  # :63579
poetry run flowfile_core    # :63578

# Frontend
cd flowfile_frontend
npm install && npm run dev:web  # :8080
```

</details>

> **Note:** Desktop installers aren't code-signed yet. On Windows click "More info" → "Run anyway". On macOS, if the app shows as damaged: `find /Applications/Flowfile.app -exec xattr -c {} \;`

---

## Architecture

Three interconnected services:

- **Designer** (Tauri + Vue) — visual interface
- **Core** (FastAPI) — ETL engine running Polars (`:63578`)
- **Worker** (FastAPI) — computation and caching (`:63579`)

Plus an embedded **scheduler** and a sandboxed **kernel runtime** for Python Script nodes.

Each flow is a directed acyclic graph where nodes are data operations and edges are data flow. Every visual flow exports to standalone Python/Polars code for production use.

Deeper dive: [Architecting a Visual ETL Tool with Polars](https://dev.to/edwardvaneechoud/building-flowfile-architecting-a-visual-etl-tool-with-polars-576c).

---

## TODO

- [x] Cloud storage support (S3, ADLS, GCS)
- [x] Code generation from visual flows and reverse engineering from Polars scripts
- [x] Data catalog with Delta Lake storage
- [x] Virtual flow tables with lazy optimization
- [x] SQL editor and SQL query node
- [x] Flow scheduling (interval and table-trigger based)
- [x] Kafka / Redpanda ingestion
- [x] Sandboxed Python execution (Docker-based kernels)
- [x] Flow parameters with `${variable}` substitution
- [x] Built-in templates
- [x] Database migrations (Alembic)
- [x] Comprehensive docs site
- [ ] Comprehensive test coverage
- [ ] Multi-user collaboration
- [ ] Role-based access control

## License

[MIT](LICENSE)

---

## Acknowledgments

Built on [Polars](https://pola.rs/), [Vue.js](https://vuejs.org/), [FastAPI](https://fastapi.tiangolo.com/), [VueFlow](https://vueflow.dev/), [Delta Lake](https://delta.io/), [Graphic Walker](https://github.com/Kanaries/graphic-walker), and [Tauri](https://tauri.app/).

Flowfile

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README