kenze

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

(★ 9)

Big-file data prep that never runs out of memory - an interactive shell and CLI, powered by DuckDB.

kenze Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# kenze

[![PyPI version](https://img.shields.io/pypi/v/kenze.svg)](https://pypi.org/project/kenze/)
[![Python versions](https://img.shields.io/pypi/pyversions/kenze.svg)](https://pypi.org/project/kenze/)
[![License](https://img.shields.io/pypi/l/kenze.svg)](https://github.com/Kenzy-Zero/kenze/blob/main/LICENSE)
[![Downloads](https://static.pepy.tech/badge/kenze)](https://pepy.tech/project/kenze)
[![CI](https://github.com/Kenzy-Zero/kenze/actions/workflows/ci.yml/badge.svg)](https://github.com/Kenzy-Zero/kenze/actions/workflows/ci.yml)

**Big-file data prep that never runs out of memory — an interactive shell *and* a one-line CLI.**

`kenze` cleans and reshapes data files (CSV, Parquet, JSON) that are too big for
pandas. It's a friendly front-end over [DuckDB](https://duckdb.org): DuckDB does
the heavy lifting (streaming, disk-spill, all your CPU cores), `kenze` makes it
effortless — and auto-configures memory so your job doesn't crash.

No more `MemoryError` on a file larger than RAM, and no SQL to learn: `kenze`
processes everything **out-of-core**, streaming through DuckDB, so you can read,
clean, and reshape a CSV or Parquet file that's bigger than memory on a laptop.

![kenze in action — load, filter, plot and run a 60M-row file without ever running out of memory](https://raw.githubusercontent.com/Kenzy-Zero/kenze/main/demo.gif)

```bash
pip install kenze
```

One name for everything: `pip install kenze` → the `kenze` command → `import kenze`.

## The interactive shell

Just run `kenze`. You land in a live session: load a file once, stack simple
steps (each **previews as you go**), then run the pipeline to a file or save it
as a reusable recipe. Type `/` for a live command menu; **TAB** autocompletes
your file's real column names — and, inside a quoted condition, its real
**values**:

```
kenze > f                       <- ghost text finishes it: f`ilter`
kenze > filter ci               <- ...and your column names: filter ci`ty`
kenze > filter city = 'L        <- ...and your actual data: 'L`ondon`
                                   right-arrow takes it

```
kenze > load sales.parquet          # 60M rows, opens instantly
kenze > filter amount > 0           # each step previews live
kenze > plot amount --by city       # ascii bar chart of the live data
kenze > keep id, city, amount
kenze > dedup id
kenze > assert-unique id            # a data-quality guard, checked before writing
kenze > run clean.csv               # streamed through DuckDB, no OOM
```

It never loads more than it needs, so counts and previews on a 60-million-row file
come back in well under a second, and full writes stream through DuckDB with a
progress bar. Everything the CLI can do is in the shell — see **[SHELL.md](https://github.com/Kenzy-Zero/kenze/blob/main/SHELL.md)**.

## Feature highlights

- **An interactive shell** (`kenze`) with a `/` command menu, live previews, schema-aware autocomplete, and data-quality guards — plus the same as a one-line CLI for scripts and cron.
- **Autocomplete that knows your data, not just your columns** — ghost text finishes what you type from the first keystroke: commands, column names, file paths, and **the real values in your file**. TAB shows every value with its exact row count. It follows the pipeline you have built so far, and refuses to list a column with millions of distinct values, telling you what it measured instead.
- **Process files bigger than your RAM** without crashing — memory is auto-capped and DuckDB spills to disk.
- **40 CLI commands** for the everyday work: `keep`, `drop`, `filter`, `rename`, `cast`, `fillna`, `dedup`, `sample`, `count`, `sort`, `join`, `diff`, `pivot`, `split`, `partition`, and more — no SQL needed.
- **Count & sort with zero SQL** — `kenze count sales.csv city --top 10` is a value-counts / group-by (`--distinct user` for unique users per group); `kenze sort sales.csv --by revenue --desc --top 10` orders and keeps the top N. Both chain in the shell pipeline.
- **Model-ready in one step** — `scale`, `bin`, `encode`, `onehot`, `clip-outliers`, and a reproducible `traintest` split turn a clean file into a model-ready dataset you hand straight to scikit-learn / XGBoost.
- **See your data** — `plot amount --by city` draws an ASCII bar chart or histogram right in the terminal, so you spot skew and dirty data instantly.
- **Excel in and out** — read and write `.xlsx` workbooks natively (`convert big.parquet -o report.xlsx`), no extra dependency.
- **GeoJSON in and out** — `convert data.csv -o map.geojson` builds point geometry from lat/lon columns (auto-detected, or `--lat`/`--lon`/`--geom`); `convert map.geojson -o map.csv` flattens geometry back to WKT. Handy for prepping map layers.
- **Client-ready reports** — `kenze report data.csv -o out.pdf` turns a data file into a styled PDF/HTML report (KPI tiles + a ranked table, auto-fit to your columns); `--per-row` makes one document per row (batch / mail-merge), and `--template`/`--scaffold` let you bring your own HTML. Needs `pip install "kenze[report]"`; PDF renders with your system browser — no heavy install.
- **Messy CSVs, handled** — `--skip N` drops junk preamble rows (the shell even auto-detects them), `--skip-bad-lines` drops malformed rows, and `--no-strict-csv` opens a file that breaks the CSV standard outright — mixed line endings or a stray quote, which is ordinary Spark output.
- **Readable recipes** (`.dq` files) that chain steps into one streaming pass, with `${VAR}` templating for scheduled jobs.
- **Read and write the cloud directly** — `s3://`, `gs://`, `https://` — nothing to download first.
- **A run ledger** — `history` shows your recent runs (input → output, rows, time).
- **Data-quality guards** (`assert`, `assert_unique`, `assert_not_null`), **PII masking** (`mask`), **schema validation** (`validate`) — a failed check aborts before anything is written. `kenze validate data.csv --scaffold schema.json` writes the contract for you from a file you already trust, so the gate takes a minute to set up rather than an afternoon of hand-written JSON.
- **No lock-in** — `eject` any recipe to raw DuckDB SQL or Python.
- **Use it from Python too** — `import kenze` and call `kenze.sift(...)`, `kenze.sql(...)`.
- Atomic writes and clean, cross-platform output on any terminal.

## Why

- **It doesn't OOM.** Memory is capped to a fraction of *free* RAM and DuckDB
  spills to disk instead of dying. Point it at a file bigger than your RAM; it's fine.
- **No SQL, no pandas.** Simple verbs, or a readable recipe file.
- **One streaming pass.** A whole recipe compiles to a single query — no
  intermediate files, so it's fast and light.
- **Any format, local or cloud.** CSV / Parquet / JSON, plain or `.gz`,
  on disk or on `s3://` / `gs://` / `https://` — auto-detected.

## One-liners

```bash
kenze profile  sales.parquet                          # schema + row count, instantly
kenze peek     sales.parquet                           # first rows + types + null counts
kenze stats    sales.parquet                           # per-column min/max/nulls/unique
kenze plot     sales.parquet amount --by city          # ascii bar chart in the terminal
kenze plot     sales.parquet amount                    # ascii histogram of a numeric column
kenze check    sales.csv                               # is the file valid? any bad rows?

kenze keep     sales.parquet --cols id,city,amount -o small.csv
kenze drop     users.csv     --cols email,phone    -o clean.parquet
kenze filter   sales.parquet --where "amount > 100" -o big.csv
kenze rename   sales.csv     --map "amount:total"   -o out.csv
kenze cast     users.csv     --types "zip:VARCHAR"  -o out.parquet   # keep leading zeros
kenze fillna   users.csv     --with "city:Unknown"  -o out.csv
kenze mask     users.csv     --cols email,ssn --method hash -o safe.csv
kenze dedup    users.csv     --on id               -o unique.parquet
kenze sample   sales.parquet --n 50000             -o sample.csv
kenze clip     points.parquet --bbox -10,35,5,45    -o region.parquet

kenze join     orders.csv users.parquet --on user_id -o joined.parquet
kenze diff     old.csv new.csv --on id                # added / removed / changed
kenze pivot    sales.csv --on city --values amount --agg sum --group region -o wide.csv
kenze unpivot  wide.csv  --cols jan,feb,mar --name month --value sales -o long.csv
kenze filter   "sales_*.csv" --where "amount>0" -o all.csv   # globs unify schemas
kenze split    sales.parquet --by city -o by_city/    # one file per value
kenze partition sales.parquet --by year -o lake/      # hive year=2026/ folders
kenze convert  sales.parquet -o report.xlsx          # write a real Excel workbook
kenze keep     messy.csv --cols id,amount --skip 3 -o clean.csv  # drop junk preamble

kenze count    sales.parquet city --top 10 -o city_counts.csv        # value-counts, no SQL
kenze count    sales.parquet city --distinct user_id                 # unique users per city
kenze sort     sales.csv --by revenue --desc --top 10 -o top.csv     # order + keep top N

kenze report   summary.csv -o report.pdf --set title="Q1 Review"     # styled PDF report
kenze report   invoices.csv --per-row --format pdf -o docs/          # one PDF per row (batch)

kenze sql  "SELECT *, lag(amount) OVER (ORDER BY ts) FROM 'sales.parquet'" -o out.csv
kenze history                                         # your recent runs
```

Read or write the cloud directly (nothing to download first):

```bash
kenze filter s3://bucket/huge.parquet --where "amount > 0" -o local.csv
```

Pipe like any Unix tool (use `-` for stdin/stdout):

```bash
cat data.csv | kenze filter - --where "x > 1" -o - | gzip > out.csv.gz
```

## Recipes

Chain steps in a readable `.dq` file — they run as one streaming pass:

```yaml
# clean.dq
input:  data/sales_${DAY}.parquet     # ${DAY} filled from --set or the environment
keep:   [id, city, amount]
types:  zip:VARCHAR
filter: amount > 0
fillna: city:Unknown
dedup:  id
sample: 50000
output: out/clean.csv
```

```bash
kenze run clean.dq --set DAY=2026-07-14
kenze recipe                 # show every valid recipe step
kenze eject clean.dq --to sql    # print the raw DuckDB SQL (no lock-in)
```

Bake **data-quality tests** right into a recipe — they run before anything is written,
so a failed check aborts with no output:

```yaml
assert:          row_count > 0
assert_unique:   id
assert_not_null: id, email
```

## From Python

```python
import kenze
kenze.sift("big.parquet", "clean.csv", keep=["id", "city"], filter="amount > 0", sample=50000)
rows = kenze.sql("SELECT city, count(*) FROM 'big.parquet' GROUP BY 1")
kenze.profile("big.parquet")
```

## Handy flags

- `--dry-run` — show the compiled query + output schema *without* running it.
- `--errors bad.csv` — quarantine malformed CSV rows to a file (with line/column diagnostics) and keep going.
- `--append` — add to an existing csv/json output instead of overwriting.
- `--source-format delta|iceberg` — read a Delta Lake or Apache Iceberg table.
- `--memory-limit 8` — pin the RAM budget (GB) for reproducible / SLA runs (great for shared CI/Airflow nodes).
- `--temp-dir D:/spill` — put disk-spill where there's room.
- `--threads N` — cap how many CPU threads DuckDB uses.
- `--skip-bad-lines` — ignore malformed rows in a dirty CSV (the row is dropped, not repaired).
- `--no-strict-csv` — open a CSV that breaks the standard: mixed line endings, a stray quote. Spark output does this a lot, and `--skip-bad-lines` can't help — the parser fails before there's a row to skip.
- `--skip N` — skip N preamble rows before the CSV header (comment banners, blank lines).
- `--log run.json` — write a run manifest (inputs, rows, timing).
- `--no-history` — don't record this run in `~/.kenze/history.jsonl`.
- Writes are **atomic** — a cancelled run never leaves a half-written file.

## Hand off to a dataframe

Clean a huge file, then pass the result straight to Polars / Arrow / pandas — no disk round-trip:

```python
import kenze
df = kenze.to_polars("SELECT * FROM 'big.parquet' WHERE amount > 0")   # pip install kenze[polars]
tbl = kenze.to_arrow("SELECT city, count(*) FROM 'big.parquet' GROUP BY 1")   # kenze[arrow]
```

## Commands

`profile` · `peek` · `stats` · `plot` · `check` · `validate` · `keep` · `drop` · `rename` · `cast` ·
`fillna` · `mask` · `scale` · `bin` · `encode` · `onehot` · `clip-outliers` · `filter` · `dedup` ·
`sample` · `head` · `clip` · `convert` · `join` · `diff` · `pivot` · `unpivot` · `split` ·
`partition` · `traintest` · `count` · `sort` · `report` · `sql` · `eject` · `init` · `run` · `recipe` · `history` · `shell`

Run any of these as a one-liner, or run `kenze` and do it all interactively — the
shell wraps every command above plus session helpers (`open`, `set`, `dryrun`,
`pwd`/`cd`, `undo`, `steps`). See **[SHELL.md](https://github.com/Kenzy-Zero/kenze/blob/main/SHELL.md)** for the shell guide.

## Documentation

Full docs live in **[`docs/`](https://github.com/Kenzy-Zero/kenze/tree/main/docs)**:

| doc | for |
|---|---|
| **[Python API reference](https://github.com/Kenzy-Zero/kenze/blob/main/docs/python-api.md)** | using kenze as a **library** (`import kenze`) — every public function, exact signatures, arguments, returns, examples. |
| **[CLI reference](https://github.com/Kenzy-Zero/kenze/blob/main/DOCS.md)** | every command and flag, the memory model, cloud storage, recipes. |
| **[Interactive shell](https://github.com/Kenzy-Zero/kenze/blob/main/SHELL.md)** | the live `kenze` session guide. |
| **[Reports](https://github.com/Kenzy-Zero/kenze/blob/main/docs/report.md)** | `kenze report` — data file → styled PDF / HTML. |

## Performance

The whole point of kenze is that it doesn't fall over on files bigger than your
RAM. The repo ships a **reproducible benchmark** that proves it: it generates a
large synthetic CSV and runs the same job (filter → group-by → aggregate) with
pandas, polars and kenze, each given the **same memory budget**.

- **pandas** (eager) tries to load the whole file, blows past the budget, and dies with an out-of-memory error.
- **kenze** streams the job *within* the budget — spilling to disk when it has to — and finishes.

Run it yourself:

```bash
pip install pandas polars psutil
python bench/benchmark.py --rows 60000000 --mem-gb 3
```

It prints a markdown table (outcome / wall time / peak memory) and writes
`bench/RESULTS.md`. The same benchmark runs in CI
([Benchmark workflow](https://github.com/Kenzy-Zero/kenze/actions/workflows/benchmark.yml)),
where the results are uploaded as a build artifact and posted to the run summary.

## Where it stops (on purpose)

kenze is one dependency and one machine — that's the whole point. It maxes out
your cores and spills to disk so a single laptop or VM can chew through files far
bigger than its RAM. It does **not** run a cluster. If you've genuinely outgrown
one machine (multi-terabyte, distributed pipelines with SLAs and lineage
tracking), reach for Spark/Dask — kenze is the tool you use *before* you need those.

## Troubleshooting

**`'kenze' is not recognized` / `kenze: command not found`?**
`pip` installed kenze correctly — the command just landed in a folder that isn't on
your PATH (this affects every pip-installed CLI). Options:

- **Use it now, no setup:** `python -m kenze --help`
- **Fix it for good:** reinstall Python from [python.org](https://www.python.org/downloads/)
  with **"Add python.exe to PATH"** ticked, or use `python -m pipx install kenze`.

## Feedback, bugs & feature requests

Found a bug, want a new command, or hit something confusing? **Open an issue:**
[github.com/Kenzy-Zero/kenze/issues](https://github.com/Kenzy-Zero/kenze/issues).
(The shell prints this link too — `help`.) Pull requests welcome — see
**[CONTRIBUTING.md](https://github.com/Kenzy-Zero/kenze/blob/main/CONTRIBUTING.md)**.
A [GitHub star](https://github.com/Kenzy-Zero/kenze) helps others find kenze.

Every command is covered by a test suite (`tests/`) that runs on Linux and Windows
across Python 3.9–3.13 (see the CI badge) — including the data-quality guards and
the ML-prep transforms. Run it locally with `pip install -e ".[dev]" && pytest`.

MIT licensed.

kenze

kenze Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

kenze

kenze Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README