streambed - GitLite

KO | EN

gitlite — search

#javascript #python #react #typescript #llm #machine-learning #rust #macos #windows #cli #kubernetes #awesome-list #ios #open-source

streambed

(★ 267)

Stream Postgres to Apache Iceberg on S3 via logical replication, queryable over the Postgres wire protocol.

파일 탐색기

최종 버전 다운로드 (.zip)

- - bug_report.md
  - feature_request.md
- - ci.yml
  - deploy.yml
  - release.yml
- pull_request_template.md
- - main.go
- - main.go
- config.go
- config_test.go
- - initial-load.md
  - mutation-modes.md
  - README.md
  - update-shape.md
- architecture.excalidraw
- architecture.svg
- demo.gif
- demo.tape
- maintenance.md
- - failpoint.go
  - failpoint_test.go
- - avro.go
  - avro_test.go
  - catalog.go
  - catalog_test.go
  - compaction.go
  - cow_layout_test.go
  - cow_prune.go
  - dedup_test.go
  - maintenance.go
  - schema.go
  - schema_test.go
  - writer.go
  - writer_test.go
- - builder.go
  - builder_test.go
  - reader.go
  - reader_test.go
- - pipeline.go
  - pipeline_test.go
- - resync.go
- - catalog.go
  - catalog_test.go
  - server.go
- - runner.go
- - lock_test.go
  - store.go
  - store_test.go
- - fault_s3.go
  - mem_s3.go
  - retry.go
  - retry_test.go
  - s3.go
- - copy_text.go
  - copy_text_test.go
  - decoder.go
  - decoder_test.go
  - metadata.go
  - resync.go
  - resync_test.go
  - slot.go
  - types.go
- - format.json
- bench-pgbench.sql
- test-integration.sh
- - _index.md
- - CNAME
  - demo.gif
- go.mod
- go.sum
- hugo.yaml
- - ack_object_store_failpoint_test.go
  - docker-compose.yml
  - edge_cases_test.go
  - integration_test.go
  - mor_test.go
  - mutation_size_benchmark_test.go
  - mutation_workload_benchmark_test.go
  - oracle_test.go
  - pgbench_test.go
  - replica_identity_test.go
  - resilience_test.go
  - schema_test.go
  - snapshot_handoff_test.go
  - throughput_test.go
  - type_roundtrip_test.go
  - walsender_disconnect_test.go
- - docker-compose.yml
.gitignore
.goreleaser.yml
architecture.svg
CHANGELOG.md
CLAUDE.md
CONTRIBUTING.md
docker-compose.yml
go.mod
go.sum
LICENSE
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Streambed

[![CI](https://github.com/viggy28/streambed/actions/workflows/ci.yml/badge.svg)](https://github.com/viggy28/streambed/actions/workflows/ci.yml)
[![Go Reference](https://pkg.go.dev/badge/github.com/viggy28/streambed.svg)](https://pkg.go.dev/github.com/viggy28/streambed)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://raw.githubusercontent.com/viggy28/streambed/main/LICENSE)

Postgres-to-Iceberg CDC engine. Offload analytical queries from your production database without changing your application.

streambed streams WAL changes via logical replication, writes Parquet files to S3, and commits Iceberg metadata. Query the result with any Iceberg-compatible engine -- or use the built-in query server, which speaks the Postgres wire protocol so you can connect with `psql`.

## See It In Action

Same analytical query on pgbench (1M accounts, 500K history rows). Postgres on the left, Streambed on the right.

![Demo](https://raw.githubusercontent.com/viggy28/streambed/main/docs/demo.gif)

No ETL. No Spark. Just Postgres + S3.

## Quick Start

```bash
# Start Postgres + MinIO locally
docker compose up -d

# Build
go build -o streambed ./cmd/streambed

# Start syncing + query server on :5433
./streambed sync \
  --source-url="postgres://postgres:test@localhost:5432/postgres" \
  --s3-bucket="streambed" \
  --s3-endpoint="http://localhost:9000" \
  --s3-prefix="test" \
  --query-addr=:5433

# Query your Postgres tables via Iceberg
psql -h localhost -p 5433 -U postgres -d postgres
```

Run `streambed sync --help` for all configuration options. All flags support environment variables with `STREAMBED_` prefix (e.g. `STREAMBED_SOURCE_URL`). UPDATE/DELETE use copy-on-write by default; opt into Iceberg v2 equality deletes with `--mutation-mode=mor` (or `STREAMBED_MUTATION_MODE=mor`) after verifying reader compatibility. Once a table has active equality deletes, Streambed fails startup in COW mode; continue using MOR.

Use `--target-file-size-mb` (or `STREAMBED_TARGET_FILE_SIZE_MB`) to split large flushes into approximately target-sized Parquet data files. The default is 128 MiB. This is a target, not a hard maximum; small flushes still create small files.

## Architecture

![Architecture](https://raw.githubusercontent.com/viggy28/streambed/main/docs/architecture.svg)

## How It Works

```
Postgres WAL ──▶ Decode ──▶ Buffer ──▶ Parquet ──▶ S3 ──▶ Iceberg Commit
                                                              │
                                                    DuckDB ◀──┘ (query server)
```

Streambed connects to Postgres as a logical replication subscriber. It decodes WAL messages (inserts, updates, deletes), buffers rows per table, and periodically flushes them as Parquet files to S3 with Iceberg metadata commits. Updates and deletes use copy-on-write merging against existing Parquet data.

A query server exposes Iceberg tables over the Postgres wire protocol using embedded DuckDB, so you can query with psql or any Postgres client.

## Commands

| Command | What it does |
|---------|-------------|
| `streambed sync` | Main daemon. Streams WAL, writes Iceberg, optionally serves queries. |
| `streambed resync --table=public.users` | One-shot backfill via `COPY` under a consistent snapshot. |
| `streambed query` | Standalone query server (no sync). Points at existing Iceberg tables. |
| `streambed cleanup --table=public.users` | Deletes S3 objects and state for a table. Useful before `resync`. |
| `streambed maintenance --table=public.users` | Expires old Iceberg snapshots and can dry-run orphan planning. |
| `streambed maintenance compact --table=public.users` | Compacts many small active data files into fewer target-sized files. |

See [docs/maintenance.md](docs/maintenance.md) for snapshot expiration and small-file compaction details.

## Development

Requires Go 1.22+ and CGO (for go-duckdb and go-sqlite3).

```bash
# Build
go build -o streambed ./cmd/streambed

# Unit tests
go test ./internal/... ./config/...

# Integration tests (requires Docker)
./scripts/test-integration.sh
```

Integration tests use the `integration` build tag and run against Postgres (port 5434) and MinIO (port 9002) from `test/integration/docker-compose.yml`.