openbesluitvorming

KO | EN

gitlite — search

#javascript #python #ai #android #linux #docker #macos #awesome #cli #kubernetes #vue #security #ios #open-source

openbesluitvorming

(★ 18)

Zoekmachine voor vergaderingen, bijlagen en andere documenten van Nederlandse overheden.

openbesluitvorming Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# OpenBesluitvorming

- Zoek door **meer dan 5 miljoen documenten** van meer dan 300 organisaties.
- PDF's worden weergegeven als afbeeldingen (laad zeer snel) of als markdown text (toegankelijk)
- Vergaderingen zijn gekoppeld aan documenten voor context, losse agendapunten zichtbaar.
- **Moties en amendementen** met hun uitslag, en waar de raad digitaal stemt de stem van elk raadslid met fractie.
- **Gesproken woord**: video- en audioregistraties met hoofdstukken per agendapunt en, waar de leverancier die publiceert, het transcript — doorzoekbaar op wat er gezegd is.
- Importeert dagelijks uit verschillende vergadersoftware (Notubiz, Ibabs, GemeenteOplossingen, Parlaeus)

OpenBesluitvorming is de vervanger van [OpenRaadsinformatie](https://github.com/openstate/open-raadsinformatie/).

## Use the API

See [API.md](./API.md), rendered at
[openbesluitvorming.nl/docs/api](https://openbesluitvorming.nl/docs/api).

## Running locally

```sh
docker compose up -d
# visit http://0.0.0.0:8787
```

For frontend iteration with HMR, use one command:

```sh
pnpm run dev
```

That does two things:

- starts the Docker backend services needed for app development: `quickwit` and `openbesluitvorming`
- starts the Vite HMR frontend on the host

That gives you:

- the Docker backend on `http://127.0.0.1:8787`
- the Vite dev server with HMR on `http://127.0.0.1:4317`

Vite proxies `/api/*` calls to the Docker backend, so reruns and extraction use
the same environment as the containerized app, including the installed `transmutation`
binary.
Open `http://127.0.0.1:4317` while iterating on the frontend.

Object storage is taken from your environment configuration. If your `.env` points
to external S3-compatible storage such as Hetzner, `pnpm run dev` uses that directly
and does not start local MinIO.

Important:

- `pnpm run dev` is the intended development entrypoint
- `pnpm run dev` starts Docker-backed services and the API, then runs Vite with HMR
- `pnpm run dev` clears stale listeners on the HMR port before starting, so leftover Vite processes do not usually require manual cleanup

If you only want the infra:

```sh
pnpm run dev:infra
```

Run:

```sh
pnpm run dev
```

To stop the Docker side again:

```sh
pnpm run dev:down
```

If you want a different dev port:

```sh
WOOZI_WEB_PORT=4401 pnpm run dev:web
```

If you want a different HMR web port:

```sh
WOOZI_WEB_PORT=4401 pnpm run dev
```

## Production with Caddy

The repo includes a production-oriented compose file and Caddy config:

- [docker-compose.production.yml](docker-compose.production.yml)
- [Caddyfile](Caddyfile)

That setup is intended for:

- `openbesluitvorming`
- `quickwit`
- `caddy`

with external S3-compatible object storage from `.env`.

Preferred production deploy flow:

```sh
git push origin main
```

That triggers the GitHub Actions workflow in [.github/workflows/publish-openbesluitvorming.yml](.github/workflows/publish-openbesluitvorming.yml), which builds and publishes:

- `ghcr.io/ontola/openbesluitvorming:main`
- `ghcr.io/ontola/openbesluitvorming:sha-<git-sha>`
- `ghcr.io/ontola/openbesluitvorming:latest`

A successful publish then triggers [.github/workflows/deploy-production.yml](.github/workflows/deploy-production.yml) automatically, which deploys that image. **Pushing to `main` is the deploy** — there is no separate manual step in the normal flow.

Run it by hand only to deploy a specific revision, or when the automatic deploy did not run:

```sh
pnpm run deploy:production
```

`deploy:production` does two things: it syncs the runtime config (compose, Caddyfile, quickwit.yaml) to the server, and over SSH tells it to pull `ghcr.io/ontola/openbesluitvorming:sha-<short-git-sha>` and restart the app and worker containers.

It refuses to run with uncommitted changes, because CI only publishes an image for a committed revision. It does **not** wait for imports to finish: the worker hands claimed runs back to the queue on SIGTERM, so a deploy during an import is safe by design.

By default, `deploy:production` derives the GHCR owner from your `origin` remote. If you need to override it explicitly:

```sh
IMAGE_REPOSITORY=ghcr.io/your-org/openbesluitvorming pnpm run deploy:production
```

When production infra files change, sync those separately:

```sh
pnpm run deploy:production:infra
```

That is only for runtime config such as:

- [docker-compose.production.yml](docker-compose.production.yml)
- [Caddyfile](Caddyfile)
- [quickwit/quickwit.yaml](quickwit/quickwit.yaml)

Required production env includes:

- `DOMAIN`
- `ADMIN_PASSWORD_HASH`
- `S3_ACCESS_KEY`
- `S3_SECRET_KEY`
- `S3_STORAGE_BUCKET_NAME`
- `S3_STORAGE_ENDPOINT`
- `S3_STORAGE_REGION`
- `QUICKWIT_INDEX_ID`
- `QUICKWIT_CLUSTER_ID`
- `QUICKWIT_NODE_ID`
- `QUICKWIT_INDEX_ROOT_PREFIX`

The server should run production with:

```sh
docker compose -f docker-compose.production.yml up -d
```

Point your domain to the server first so Caddy can obtain Let's Encrypt certificates.

Important:

- the GHCR package must be public, or the server must be logged in to GHCR
- code deploys should update container images, not rsync source files

Quickwit defaults are intentionally different between local and production so both environments do not accidentally share the same S3-backed metastore and index:

- local/dev defaults:
  - `QUICKWIT_CLUSTER_ID=woozi-dev`
  - `QUICKWIT_NODE_ID=quickwit-dev`
  - `QUICKWIT_INDEX_ROOT_PREFIX=indexes-dev`
  - `QUICKWIT_INDEX_ID=woozi-events-dev`
- production defaults:
  - `QUICKWIT_CLUSTER_ID=woozi-prod`
  - `QUICKWIT_NODE_ID=quickwit-prod`
  - `QUICKWIT_INDEX_ROOT_PREFIX=indexes-prod`
  - `QUICKWIT_INDEX_ID=woozi-events-prod`

Important:

- if production previously used `indexes` + `woozi-events`, switching to `indexes-prod` + `woozi-events-prod` creates a fresh search projection
- after that change, production search needs a reindex/reimport before results appear again

## Architecture

Woozi is designed as an event-driven indexing system.

The core split is:

- extract data from source systems
- normalize it into canonical entities
- emit events only for changes
- build read projections from those events

The main flow is:

```text
source system
  -> extractor / poller
  -> canonical entity
  -> event broker
  -> projection services
  -> search index / resolver / other read models
```

## Main Components

### 1. Extractors

Extractors talk to source systems such as Notubiz, iBabs, GO, and Parlaeus.

Responsibilities:

- poll for changes in a date range or from a source cursor
- fetch raw payloads and documents
- store retrieved files in S3-compatible object storage
- extract markdown-ready document text from PDFs and office files
- normalize source-specific structures
- produce canonical entities

The current ORI extractor logic is the starting point for this layer.

### 2. Canonical Entities

The system does not index raw source payloads directly.

Instead, each source payload is transformed into a canonical entity such as:

- `Meeting`
- `Document`
- `Committee`
- `Motion` — motie/amendement, with its outcome and, where the council votes
  digitally, the vote of every individual member
- `Recording` — the video or audio registration of a meeting, with chapters per
  agenda item and the transcript where the supplier publishes one
- `Party` and `Person` — the fractie and the council member a vote points at

The schemas in [`schemas/`](schemas) are the first version of those contracts.

### 3. Change Detection

Quickwit is not used to determine whether something changed.

Change detection happens before indexing:

- each canonical entity gets a stable ID
- each canonical payload gets a content hash
- the latest known hash is compared against metadata storage
- if nothing changed, no event is emitted
- if something changed, a new commit event is emitted

Edits are modeled as new versions, not in-place mutations.
Deletes are modeled as tombstones or delete commits.

### 4. Event Broker

When an entity changes, Woozi emits an event into an event broker.

The expected event model is:

- CloudEvents envelope
- `entity.commit` payload

This broker decouples extraction from indexing and makes replay possible.

### 5. Canonical Storage

The canonical representation should not live in Quickwit.

Canonical JSON and original files should be stored in object storage, such as:

- S3
- MinIO

Example object classes:

- raw source payloads
- canonical JSON snapshots
- original files
- derived markdown and search text

### 6. Metadata Store

Woozi still needs a small metadata store.

PostgreSQL remains useful for:

- source cursors
- latest entity head per ID
- content hashes
- commit metadata
- projector checkpoints
- resolver mappings

PostgreSQL should be small and boring in Woozi.
It is not the document store and not the search index.

Current prototype note:

- extraction/admin run state is currently stored in a local SQLite file
- that is an implementation shortcut for the prototype, not the intended final metadata design

### 7. Projection Services

Projection services consume events and build read models.

Initial projections:

- search projection
- document resolver projection
- admin/reporting projection

Each projection can be rebuilt by replaying events and canonical snapshots.

### 8. Quickwit

Quickwit is the search projection, not the source of truth.

Responsibilities:

- store search-ready projected documents
- support search and filtering at large scale
- stay cheap by relying on object storage-backed indexing

Quickwit should receive projection documents derived from canonical entities.
It should not receive raw source payloads and should not be the place where updates are detected.

## Design Rules

- Raw source payloads are not the public contract.
- Canonical entities are the internal contract.
- Events describe changes to canonical entities.
- Quickwit is a projection.
- Object storage holds canonical payloads and files.
- PostgreSQL holds metadata and coordination state.

## Current State

This folder currently contains:

- minimal entity schemas
- a first Deno-based Notubiz extractor slice
- a Vite + TypeScript frontend with HMR for the public UI and admin UI
- shared frontend/backend TypeScript API types
- `entity.commit` events for canonical meetings, documents, motions and recordings
- attachment download into S3-compatible object storage
- markdown extraction for PDF and Word-style documents
- motions and per-member votes from iBabs list entries and Notubiz module items
- video/audio registrations with an agenda-item timeline and ASR transcripts
- a local Quickwit setup and projection client
- a small admin UI for reruns and extraction run inspection
- live e2e coverage that ingests Haarlem meetings and attached files into Quickwit and the GUI

## Commands

From [`woozi/`](.):

- `pnpm run dev`
- `pnpm run dev:infra`
- `pnpm run dev:docker`
- `pnpm run dev:down`
- `pnpm run dev:web`
- `pnpm run web`
- `pnpm run serve:web`
- `pnpm run build:web`
- `pnpm test`
- `pnpm test:live`
- `pnpm test:e2e`
- `pnpm test:gui`
- `pnpm test:quickwit`
- `pnpm run extract:haarlem`
- `pnpm run ingest:haarlem`
- `pnpm run lint`
- `pnpm run format`
- `pnpm run check-format`

### Tests

`pnpm test` is hermetic and runs in about ten seconds. It is what CI gates on.

The integration tests are opt-in behind `WOOZI_RUN_LIVE_INTEGRATION=1`, which
is what `pnpm test:live` (or the individual `test:e2e`, `test:gui`,
`test:quickwit`) sets. They need Docker and minio, the live Notubiz API, and
the PDF extraction fleet — and that fleet only accepts connections from the
production host. Off production they do not fail quickly: each document waits
out its retries first, so a run takes minutes and ends in
`extraction_service_failed`. Run them from a machine that can reach the fleet,
or not at all.

Quickwit helpers live in [`quickwit/`](quickwit).

For real S3-compatible storage, put these values in [`.env.example`](.env.example) copied to `.env`:

- `S3_ACCESS_KEY`
- `S3_SECRET_KEY`
- `S3_STORAGE_BUCKET_NAME`
- `S3_STORAGE_ENDPOINT`
- `S3_STORAGE_REGION`

To run against external S3-compatible storage:

```bash
docker compose up -d
```

To run the local stack with MinIO for development and tests:

```bash
docker compose --profile local-s3 up -d
```

To extract one Haarlem day and ingest the resulting commit events into Quickwit:

```bash
pnpm run ingest:haarlem
```

That command now:

- extracts public Haarlem meetings from Notubiz
- downloads attached source files
- stores the originals in S3-compatible object storage
- extracts markdown from those files and serves it lazily in the detail view
- emits `entity.commit` events for `Meeting` and `Document`
- projects both entity types into Quickwit

To start the frontend prototype:

```bash
docker compose up -d --build openbesluitvorming
```

For local production-style serving without Docker:

```bash
pnpm run web
```

## Local extractor requirements

The host-side ingest command currently uses:

- the Rust `transmutation` CLI for PDFs
- direct text decoding for `.txt`, `.md`, `.json`, and HTML-like documents
- office formats like `.doc`, `.docx`, `.rtf`, and `.odt` are not supported yet and are logged as extraction warnings

In Docker and compose, the `transmutation` CLI is installed in the image automatically.
On the host, Woozi will try to call `transmutation` from `PATH` or from `WOOZI_TRANSMUTATION_BIN`.

For Apple Silicon macOS, you can install it like this:

```sh
cargo install --locked transmutation
```

If the CLI is missing, PDF extraction fails explicitly and the import records an extraction issue.

There is no remaining host-only Office extractor in the runtime anymore. Unsupported office
formats are currently skipped with an explicit extraction warning.

openbesluitvorming

openbesluitvorming Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

openbesluitvorming

openbesluitvorming Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README