minarrow

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #claude-code #codex #cli #developer-tools #react #windows

★ 98 Open GitHub ↗

Fast, minimal, and ergonomic zero-copy Apache Arrow tabular data implementation in Rust, with PyO3, Python bindings, NdArray, XArray, shared-memory and low-latency SIMD for data-engineering and columnar workloads. Bridge to Polars and Arrow-rs over FFI effortlessly.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

**Fast, minimal, and ergonomic Apache Arrow implementation in Rust, with PyO3 and Python bindings.**

**Highlights**:
- < 2s compilation times (_0.15s rebuilds_)
- Rust to Python for 1m rows in under 220ns¹
- Guaranteed 64-byte SIMD alignment
- Fully-typed throughout
- Embed Python ML in Rust
- Plug in Polars, Arrow, and Python with Zero-copy FFI, PyCapsule and PYO3
- NdArray and XArray implementations with native DLPack integration for bridging to Python AI/ML workloads.

**Limitations**:
- No Arrow lists or structs.

## Why Minarrow?

- **Productive**: Lean base dependencies help keep flow and library fast.
- **Fully typed**: Enum dispatch ensures constant feedback through a strong compiler + IDE feedback loop
- **Fail fast**: Errors that might slip through a dynamic dispatch boundary into run-time errors, stay compile-time.
- **Ergonomics**: Convenient dot syntax, composable opt-up abstractions, and flexible signatures via liberal `From` trait implementations.
- **Speed**: builds on the `Vec64` crate so data is fully SIMD compatible for an extra layer of low-latency parallelism, with out of the box numeric, bitmask and string SIMD kernels to help get you started.
- **Minimal Dependencies**: Small security surface - built from foundations. Only `log`, `num-traits` _(and custom-built `Vec64`)_ in the base build.

## Installation

```bash
cargo add minarrow
```

Minarrow uses the nightly toolchain for `allocator_api` and `portable_simd`:

```bash
rustup override set nightly
```

## Quick Start

```rust
use minarrow::{arr_i32, arr_f64, arr_str32, arr_bool, MaskedArray};

// Create arrays with macros
let ids = arr_i32![1, 2, 3, 4];
let prices = arr_f64![10.5, 20.0, 15.75];
let names = arr_str32!["alice", "bob", "charlie"];
let flags = arr_bool![true, false, true];

// Direct typed access - no downcasting
assert_eq!(ids.len(), 4);
assert_eq!(prices.num().f64().get(0), Some(10.5));
```

```rust
use minarrow::{tbl, fa_i32, fa_str32, Print};

// Build tables via FieldArrays with constructor macros
let table = tbl!(
    "users",
    fa_i32!("id", 1, 2, 3),
    fa_str32!("name", "alice", "bob", "charlie"),
);
table.print();
```

## Core Features

### Typed Arrays

Six typed arrays back standard workloads:

| Type | Description |
|------|-------------|
| `IntegerArray<T>` | i8 through u64 |
| `FloatArray<T>` | f32, f64 |
| `StringArray<T>` | UTF-8 with u32 or u64 offsets |
| `BooleanArray` | Bit-packed with null mask |
| `CategoricalArray<T>` | Dictionary-encoded |
| `DatetimeArray<T>` | Timestamps, dates, durations |

Semantic groupings (`NumericArray`, `TextArray`, `TemporalArray`) support flexibility call-site signatures and static dispatch.

`Array` and `Table` (`RecordBatch`) sit on top, with chunked `Super` (`Vec<Table>`) versions for streaming.

Bonus LAPACK-compatible `Matrix` and `Cube` types support analytical workload variations.

`NdArray` n-dimensional numeric data, with chunked (`SuperNdArray`) and labelled (`XArray`) forms, and hands tensors to PyTorch, JAX, and NumPy zero-copy over DLPack.

### Zero-Copy Views

Ergonomic zero-copy row and column selection.

```rust
use minarrow::*;

let table = create_table();

// Pandas-style selection
let view = table.c(&["name", "value"]);    // columns
let view = table.r(10..20);                // rows
let view = table.c(&["A", "B"]).r(0..100); // both

// Materialise when needed
let owned = view.to_table();
```

### Off-the-Wire pattern

```rust
use std::sync::Arc;
use minarrow::{st, Consolidate, SuperTable};

// Append batches as they arrive
let mut stream = SuperTable::new("trades".into());
stream.push(Arc::new(batch1));
stream.push(Arc::new(batch2));

// Assemble from existing batches
let stream = st!("trades", batch1, batch2);

// Consolidate to a single table when ready
let table = stream.consolidate();
```

### Arrow Interop

Embed Python in Rust.
Bench in /examples.

```rust
// Run a Random Forest Classifier using Python in Rust
let value = rt.with_python(&dataset, |py, obj| {
    let scope = PyDict::new(py);
    scope.set_item("table", obj)?;
    py.run(
        cr#"
        import polars as pl
        from sklearn.ensemble import RandomForestClassifier
        from sklearn.model_selection import train_test_split

df = table.to_polars()
        features = ["x0", "x1", "x2", "x3"]
        X = df.select(features).to_numpy()
        y = df["label"].to_numpy()

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
        model = RandomForestClassifier(n_estimators=100, random_state=0)
        model.fit(X_train, y_train)
        predicted = model.predict(X_test)

result = pl.DataFrame(
            {
                "actual": y_test.astype("int64"),
                "predicted": predicted.astype("int64"),
            }
        )
        "#,
        Some(&scope),
        Some(&scope),
    )?;
    scope
        .get_item("result")?
        .ok_or_else(|| pyo3::exceptions::PyKeyError::new_err("result not set"))
})?;
```

Normal Rust - use Polars, Arrow, convert at the boundary, stay native internally:

```rust
// To Arrow (feature: cast_arrow)
let batch = table.to_apache_arrow();          // RecordBatch
let column = table.cols[0].to_apache_arrow(); // ArrayRef

// To Polars (feature: cast_polars)
let df = table.to_polars();                   // DataFrame

// FFI via the Arrow C Data Interface
use minarrow::ffi::arrow_c_ffi::export_to_c;
let (array_ptr, schema_ptr) = export_to_c(array, schema);
```

## Architecture

Minarrow uses enums for type dispatch instead of trait objects.

```rust
// Static dispatch routing, full compiler inlining
match array {
    Array::NumericArray(num) => match num {
        NumericArray::Int64(arr) => process(arr),
        NumericArray::Float64(arr) => process(arr),
        // ...
    },
    // ...
}

// Get fully typed IntegerArray<i64> instead of Array:
let array = array.num().i64();
```

## Benchmarks

### Array performance
Sum of 1,000 integers, averaged over 1,000 runs¹:

| Implementation | Time |
|----------------|------|
| Raw `Vec<i64>` | 85 ns |
| Minarrow `IntegerArray` (direct) | 88 ns |
| Minarrow `IntegerArray` (via enum) | 124 ns |
| Arrow-rs `Int64Array` (struct) | 147 ns |
| Arrow-rs `Int64Array` (dyn) | 181 ns |

Minarrow's direct access is within the noise threshold of raw Vec performance whilst maintaining SIMD-compatible alignment.

### Python Roundtrip
- GIL acquisition (uncontended): **53ns**
- **Rust to Python¹** 1m rows, 2 columns : **165ns**
- **Python to Rust¹** 1m rows, 2 columns:  **2.5μs**

See `minarrow-pyo3/examples`

### Test machine
▎ ¹ = Intel Core Ultra 7 155H · 32 GB · Ubuntu 24.04 · 1.97-nightly release build.

## Feature Flags

Default features: `views`, `chunked`, `large_string`, `simd`, `select`.

| Feature | Description |
|---------|-------------|
| `views` | Zero-copy windowed access |
| `chunked` | SuperArray/SuperTable for streaming |
| `large_string` | String arrays with 64-bit offsets |
| `simd` | SIMD kernels for Bitmask and arithmetic |
| `select` | Pandas-esque `.c()` / `.r()` selection |

Interop:

| Feature | Description |
|---------|-------------|
| `cast_arrow` | Arrow-rs conversion via `to_apache_arrow()` |
| `cast_polars` | Polars conversion via `to_polars()` / `from_polars()` |
| `memfd` | Memfd-backed buffers for zero-copy cross-process sharing (Linux) |

Additional types:

| Feature | Description |
|---------|-------------|
| `datetime` | Temporal array types as raw integer offsets |
| `datetime_ops` | Datetime library functionality: ISO 8601 parsing, timezone-aware operations, arithmetic, component extraction |
| `extended_numeric_types` | i8, i16, u8, u16 variants |
| `extended_categorical` | Categorical8/16/64 dictionary index widths |
| `scalar_type` | Unified `Scalar` for aggregation results |
| `value_type` | Catch-all `Value` enum for unified typing |
| `matrix` | 2D matrix with BLAS/LAPACK-compatible layout |
| `cube` | Stacks tables along an extra axis |
| `ndarray` | N-dimensional array for bridging into ML & AI, with a chunked version and native DLPack interchange |
| `xarray` | N-dimensional array with named dims and coordinates for spatial queries |
| `dlpack` | DLPack tensor interchange with PyTorch, JAX, TensorFlow |
| `shared_dict` | Shared source of truth for categorical dictionaries. |

Performance:

| Feature | Description |
|---------|-------------|
| `parallel_proc` | Rayon parallel iterators |
| `fast_dict` | ~30% faster `shared_dict`, at cost of 3 dependencies |
| `fast_hash` | Swaps hashing to `ahash` |
| `arena` | Bump allocator for bulk array and Table construction |
| `vmap64` | Mmap-backed `Vec64` on Linux |
| `lbuffer` | Atomically updated array source |

Extras:

| Feature | Description |
|---------|-------------|
| `broadcast` | Typed arithmetic broadcasting |
| `str_arithmetic` | String arithmetic kernels for outlandish concatenation |
| `hash` | Hash and Eq for `Scalar` |
| `size` | Byte size estimation |
| `table_metadata` | Schema-level metadata map on `Table` |

See [Cargo.toml](Cargo.toml) for the full list with detailed notes on each.

## Ecosystem

| Project | Purpose |
|---------|---------|
| [`minarrow-py`](py/README.md) | Minarrow Python bindings  |
| [`minarrow-pyo3`](pyo3/README.md) | Zero-copy Python interop via PyArrow |
| [`vec64`](https://crates.io/crates/vec64) | Custom 64-byte aligned Vec for SIMD compatible workloads |
| [`lightstream`](https://crates.io/crates/lightstream) | Zero-copy Arrow streaming over Tokio, TCP, QUIC, WebSocket, Unix sockets, and Stdio |
| [Lightning Analytics Engine](https://spacecell.com) | Sub-millisecond, zero-config live streaming engine with statistical modelling and data processing. |

## Contributing

Contributions are welcome, particularly in the following areas:

1. **Nested types** - List and Struct support
2. **Bug fixes**

All contributions are subject to the Contributor Licence Agreement (CLA).
See [CONTRIBUTING.md](CONTRIBUTING.md) for details.

## License

Released under the Apache 2.0 License. See [LICENSE](LICENSE) for details.

## Acknowledgements

Minarrow is a from-scratch implementation of the Apache Arrow memory layout inspired by the standards pioneered by Apache Arrow, Arrow2, and Polars.

Minarrow is not affiliated with Apache Arrow.

## SpaceCell

Minarrow is maintained by [SpaceCell](https://spacecell.com) and forms part of its open-source foundation for high-performance data computing.

minarrow

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README