timefusion

KO | EN

gitlite — search

#python #react #javascript #golang #nodejs #machine-learning #python3 #go #css #java #tailwindcss #html-css-javascript

★ 185 Open GitHub ↗

A timeseries database created for events, logs, traces and metrics. Speaks the postgres dialect, and stores data in s3 via delta lake protocol

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# TimeFusion

[![CI](https://github.com/monoscope-tech/timefusion/actions/workflows/ci.yml/badge.svg)](https://github.com/monoscope-tech/timefusion/actions/workflows/ci.yml)
[![Rust edition 2024](https://img.shields.io/badge/rust-edition%202024-orange.svg)](https://www.rust-lang.org/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/monoscope-tech/timefusion/master/LICENSE)

TimeFusion stores observability data — logs, traces, metrics, and events — as
[Delta Lake](https://delta.io/) tables on your own S3-compatible object storage,
and lets you query it over the **PostgreSQL wire protocol**. It uses
[Apache DataFusion](https://datafusion.apache.org/) for query execution and a
write-ahead log + in-memory buffer for sub-second write durability.

If you already have a Postgres client, you already have a TimeFusion client.

> **Status:** Pre-1.0. TimeFusion runs in production at
> [APIToolkit](https://apitoolkit.io), but the schema, wire behavior, and
> configuration surface can still change between releases — pin a version
> before depending on it. (The version a client sees via `SHOW server_version`
> reflects the embedded PostgreSQL-protocol layer, not TimeFusion's own.)

## Contents

- [Why TimeFusion](#why-timefusion)
- [How it works](#how-it-works)
- [Quick start](#quick-start)
- [Querying](#querying)
- [Configuration](#configuration)
- [Performance](#performance)
- [Development](#development)
- [License](#license)

## Why TimeFusion

- **Your data stays in your bucket.** Storage is plain Parquet under a Delta
  Lake transaction log in your own S3 / MinIO / R2 bucket. No proprietary
  format, no vendor storage to pay for, nothing to migrate out of.
- **Postgres wire protocol.** Connect with `psql`, any Postgres driver, or
  existing BI/observability tooling. No new client to learn.
- **Columnar analytics.** DataFusion executes vectorized SQL with predicate
  pushdown and partition pruning over Parquet.
- **Durable, low-latency writes.** Inserts hit a write-ahead log and an
  in-memory buffer before being flushed to Delta in the background, so writes
  are acknowledged quickly without losing durability.
- **Multi-tenant by design.** Data is partitioned and isolated by
  `project_id`, with shared tables for default projects and dedicated tables
  for projects that need isolation.

## How it works

```
 Postgres client ──▶ pgwire ──▶ DataFusion query engine
 (psql, drivers)                      │
                                      ▼
              ┌───────────── write path ─────────────┐
              │  WAL (durable) ─▶ in-memory buffer    │
              │            │ flush every ~10 min       │
              │            ▼                            │
              │      Delta Lake (Parquet) on S3        │
              └────────────────────────────────────────┘
                                      ▲
                          Foyer cache (memory + disk)
```

- **Write:** `INSERT` → WAL append (durable) → in-memory buffer (fast ack).
  Completed time buckets flush to Delta in the background.
- **Read:** queries union the in-memory buffer with Delta on S3, so recently
  written rows are visible immediately.
- **Ingest:** besides pgwire `INSERT`, TimeFusion accepts gRPC streaming
  ingestion of Arrow IPC payloads (`GRPC_PORT`, default `50051`).

> **Single-writer WAL.** Exactly one TimeFusion process may run against a given
> WAL directory. Startup takes an exclusive `flock` on the WAL dir to enforce
> this; a second process waits (serving `57P03`) until the first exits. Keep
> the readiness probe a plain TCP check on the pgwire port so rolling deploys
> hand off cleanly. See [`docs/WAL.md`](docs/WAL.md) and
> [`RUNBOOK.md`](RUNBOOK.md) for deploy guidance.

**Storage layout:** events are written to a Delta table such as
`otel_logs_and_spans`, partitioned by `[project_id, date]`. Table schemas live
in [`schemas/`](schemas/) and are loaded at startup.

> Queries **must** filter on `project_id` — it is the tenancy and partition
> key. Queries without it scan across tenants and are far slower.

## Quick start

### Try it locally (zero config)

The bundled Compose stack runs TimeFusion against a local MinIO — no AWS account
or credentials needed. It builds the image, starts MinIO, creates the bucket,
and launches TimeFusion on port 5432:

```bash
git clone https://github.com/monoscope-tech/timefusion.git
cd timefusion
docker compose up
```

Then connect from another terminal — skip to [Connect](#connect).

### Run against your own S3

Images are published per commit to GitHub Container Registry, tagged with the
git short SHA (see the
[packages page](https://github.com/monoscope-tech/timefusion/pkgs/container/timefusion)
for available tags):

```bash
docker run -d -p 5432:5432 \
  -e AWS_S3_BUCKET=your-bucket \
  -e AWS_ACCESS_KEY_ID=your-key \
  -e AWS_SECRET_ACCESS_KEY=your-secret \
  ghcr.io/monoscope-tech/timefusion:<tag>
```

### Build from source

Requires a recent stable Rust toolchain (edition 2024 → **Rust 1.85+**).

```bash
git clone https://github.com/monoscope-tech/timefusion.git
cd timefusion
cargo build --release
AWS_S3_BUCKET=your-bucket ./target/release/timefusion
```

### Connect

```bash
psql "postgresql://postgres:postgres@localhost:5432/postgres"
```

### Insert and query

```sql
INSERT INTO otel_logs_and_spans (name, id, project_id, timestamp, date, hashes)
VALUES ('api.request', '550e8400-e29b-41d4-a716-446655440000',
        'prod-api-001', '2025-01-17 14:25:00', '2025-01-17', ARRAY[]::text[]);

SELECT name, COUNT(*) AS count
FROM otel_logs_and_spans
WHERE project_id = 'prod-api-001' AND date = '2025-01-17'
GROUP BY name
ORDER BY count DESC;
```

## Querying

TimeFusion speaks analytical SQL: aggregations, window functions,
`PERCENTILE_CONT`, `FILTER`, CTEs, and a TimescaleDB-compatible `time_bucket`.
Here's finding slow API endpoints from trace data:

```sql
SELECT name AS endpoint,
       COUNT(*)                       AS request_count,
       AVG(duration / 1000000)::INT   AS avg_duration_ms,
       MAX(duration / 1000000)::INT   AS max_duration_ms
FROM otel_logs_and_spans
WHERE project_id = 'prod-api-001'
  AND timestamp >= '2025-01-17 14:00:00' AND timestamp < '2025-01-17 15:00:00'
  AND duration > 1000000000          -- 1 second, in nanoseconds
GROUP BY name
ORDER BY avg_duration_ms DESC;
```
```
         endpoint        | request_count | avg_duration_ms | max_duration_ms
-------------------------+---------------+-----------------+-----------------
 POST /api/v1/payments   |             1 |            3421 |            3421
 GET /api/v1/users/:id   |             1 |            2100 |            2100
 POST /api/v1/users      |             1 |            1523 |            1523
```

<details>
<summary><b>More query examples</b> — error rates, trace lookup, time-bucketed aggregation, moving averages</summary>

```sql
-- Error rate per endpoint, per hour
SELECT name AS endpoint,
       date_trunc('hour', timestamp) AS hour,
       COUNT(*) AS total_requests,
       COUNT(*) FILTER (WHERE attributes___http___response___status_code >= 400) AS errors,
       ROUND(100.0 * COUNT(*) FILTER (WHERE attributes___http___response___status_code >= 400)
             / COUNT(*), 2) AS error_rate
FROM otel_logs_and_spans
WHERE project_id = 'prod-api-001'
  AND timestamp >= '2025-01-16 15:00:00' AND timestamp < '2025-01-17 15:00:00'
GROUP BY name, date_trunc('hour', timestamp)
ORDER BY hour DESC, error_rate DESC;

-- Find a trace by hash
SELECT id AS trace_id, name AS endpoint, timestamp,
       (duration / 1000000)::INT AS duration_ms,
       attributes___error___type AS error_type
FROM otel_logs_and_spans
WHERE project_id = 'prod-api-001'
  AND 'trace_124' = ANY(hashes)
  AND timestamp >= '2025-01-17 14:00:00' AND timestamp < '2025-01-17 15:00:00';

-- Requests and approximate p95 latency, bucketed into 5-minute windows (Timescale Toolkit-compatible)
SELECT time_bucket('5 minutes', timestamp) AS bucket,
       COUNT(*) AS requests,
       AVG(duration / 1000000)::INT AS avg_duration_ms,
       approx_percentile(0.95, percentile_agg(CAST(duration / 1000000 AS DOUBLE))) AS p95_duration_ms
FROM otel_logs_and_spans
WHERE project_id = 'prod-api-001'
  AND timestamp >= '2025-01-17 14:00:00' AND timestamp < '2025-01-17 15:00:00'
GROUP BY bucket
ORDER BY bucket DESC;

-- Per-endpoint 3-window moving average over 1-minute buckets
WITH ts AS (
  SELECT time_bucket(INTERVAL '1 minute', timestamp) AS minute,
         name AS endpoint,
         AVG(duration / 1000000) AS avg_duration_ms
  FROM otel_logs_and_spans
  WHERE project_id = 'prod-api-001'
    AND timestamp >= '2025-01-17 14:30:00' AND timestamp < '2025-01-17 15:00:00'
  GROUP BY minute, endpoint
)
SELECT minute, endpoint, avg_duration_ms::INT,
       AVG(avg_duration_ms) OVER (
         PARTITION BY endpoint ORDER BY minute
         ROWS BETWEEN 2 PRECEDING AND CURRENT ROW
       )::INT AS moving_avg
FROM ts
ORDER BY endpoint, minute DESC;
```

</details>

## Configuration

TimeFusion is configured entirely through environment variables.

**Required — object storage**

| Variable                | Description                       |
| ----------------------- | --------------------------------- |
| `AWS_S3_BUCKET`         | Bucket for Delta tables           |
| `AWS_ACCESS_KEY_ID`     | Access key                        |
| `AWS_SECRET_ACCESS_KEY` | Secret key                        |
| `AWS_S3_ENDPOINT`       | Custom endpoint (MinIO, R2, etc.) |

**Server**

| Variable      | Description                          | Default |
| ------------- | ------------------------------------ | ------- |
| `PGWIRE_PORT` | PostgreSQL wire protocol port        | `5432`  |
| `GRPC_PORT`   | gRPC ingestion port                  | `50051` |
| `GRPC_TOKEN`  | Bearer token for gRPC (open if unset)| —       |

**Write buffer & flushing**

| Variable                            | Description                      | Default |
| ----------------------------------- | -------------------------------- | ------- |
| `TIMEFUSION_BUFFER_MAX_MEMORY_MB`   | In-memory buffer size            | `4096`  |
| `TIMEFUSION_FLUSH_INTERVAL_SECS`    | Background flush interval (secs) | `300`   |
| `TIMEFUSION_BUFFER_RETENTION_MINS`  | Buffer retention before eviction | `70`    |

**Cache (Foyer, memory + disk)**

| Variable                       | Description       | Default           |
| ------------------------------ | ----------------- | ----------------- |
| `TIMEFUSION_FOYER_MEMORY_MB`   | Memory cache size | `1024`            |
| `TIMEFUSION_FOYER_DISK_GB`     | Disk cache size   | `500`             |
| `TIMEFUSION_FOYER_TTL_SECONDS` | Cache entry TTL   | `604800` (7 days) |

See [DELTA_CONFIG.md](DELTA_CONFIG.md) for the full configuration reference,
including Parquet sizing, compaction targets, and connection limiting.

## Performance

TimeFusion is built for high-throughput ingestion and low-latency analytical
queries: columnar Parquet, predicate pushdown, partition pruning by
`project_id`/`date`, a two-tier (memory + disk) cache, and background
compaction of small files.

We don't publish a single headline throughput number — real numbers depend
heavily on hardware, payload shape, batch size, and S3 round-trip latency, and
a number measured on one setup misleads on another. Instead, the
[`bench/`](bench/) directory contains the harnesses we use to measure it
ourselves, so you can reproduce results on your own infrastructure:

| Script                        | Measures                              |
| ----------------------------- | ------------------------------------- |
| `bench/run_insert_bench.sh`   | Insert throughput and tail latency    |
| `bench/run_select_bench.sh`   | Query latency                         |
| `bench/query_under_ingest.py` | Query latency during concurrent writes|
| `bench/timeseries_lifecycle.py`| End-to-end ingest → flush → query    |

**Getting good performance:**

1. **Always filter on `project_id`** — it's the tenant routing key. A
   `timestamp` range filter is automatically turned into a `date` partition
   filter, so you don't need to add an explicit `date` predicate to get
   partition pruning.
2. **Batch your inserts** — larger batches amortize WAL and Delta commit cost.
3. **Size the buffer and cache** (`TIMEFUSION_BUFFER_MAX_MEMORY_MB`,
   `TIMEFUSION_FOYER_*`) to your working set.

## Development

```bash
git clone https://github.com/monoscope-tech/timefusion.git
cd timefusion

# Start MinIO only, then run TimeFusion from source against it
docker compose up -d minio createbucket
export AWS_S3_BUCKET=timefusion \
       AWS_S3_ENDPOINT=http://localhost:9000 \
       AWS_ALLOW_HTTP=true \
       AWS_ACCESS_KEY_ID=minioadmin \
       AWS_SECRET_ACCESS_KEY=minioadmin
cargo run
```

**Tests:**

```bash
make test                           # the whole suite in one parallel run (~74s)
cargo nextest run <substring>       # one test or one .slt file, e.g. `dedup_compaction`
cargo nextest run --lib             # unit tests only
make test-e2e                       # end-to-end (Docker required; testcontainers MinIO)
RUST_LOG=debug cargo nextest run --no-capture   # with debug logging

Tests run under [cargo-nextest](https://nexte.st) (one process per test).
Install: `curl -LsSf https://get.nexte.st/latest/mac | tar zxf - -C ~/.cargo/bin`.
```

**Faster local builds (macOS arm64):** `.cargo/config.toml` links with `ld64.lld`
instead of the system linker, and `[profile.dev]` uses `debug = "line-tables-only"`
— together these roughly halve the warm recompile of the large `database.rs` crate
(~22s → ~12s). This requires `ld64.lld` on disk:

```bash
brew install llvm@15   # provides /opt/homebrew/opt/llvm@15/bin/ld64.lld
```

If it's missing the build fails with `invalid linker name`. Adjust the path in
`.cargo/config.toml` for a different llvm version, or remove that file to fall
back to the system linker.

Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).

## License

MIT — see [LICENSE](LICENSE).

## Built with

[Apache DataFusion](https://datafusion.apache.org/) ·
[Delta Lake](https://delta.io/) ·
[Apache Arrow](https://arrow.apache.org/) ·
[pgwire](https://github.com/sunng87/pgwire) ·
[Foyer](https://github.com/foyer-rs/foyer)

timefusion

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README