scout

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 12 Open GitHub ↗

No description available.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Scout

Scout is an interactive map of GB planning, infrastructure, and environmental constraints. It lets you see, on a single map, dozens of overlapping designations — flood zones, green belt, conservation areas, listed buildings, electricity grid infrastructure, agricultural land grades, and much more. It is built and maintained in partnership with [Tract](https://buildwithtract.com).

**Live at:** https://scout.britishprogress.org and https://scout.buildwithtract.com
**Repo:** `buildwithtract/scout` (default branch: `master`)
**Stack:** Next.js 15 · TypeScript · deck.gl + Google Maps · PostgreSQL 17 + PostGIS · deployed with Kamal on a Hetzner server
**Maintainer:** Maxx

---

## What this project does

Scout has two parts:

1. **The map.** A full-screen interactive map (built on Google Maps + deck.gl) where users can toggle on any combination of layers — planning designations from data.gov.uk, electricity distribution-network-operator (DNO) infrastructure lines and substations, Environmental Agency flood and agricultural-land data, OpenStreetMap features, and more. Active-layer state is stored in the URL so you can share a specific view.

2. **Data fetchers.** About 64 background scripts (in `src/fetchers/`) that pull data from data.gov.uk, the six UK electricity DNOs, Natural England, and OpenStreetMap into a PostGIS (geographic) database. Each fetcher is a TypeScript class; you run them manually or via Makefile helpers.

Map tiles are served from the Next.js app itself as MVT (Mapbox Vector Tile) tiles via the `/api/mvt/...` API route — there is no separate tile server.

---

## How to make changes

You do not need to be a developer to direct changes. Open a conversation in Claude Code and describe what you want. Good things to ask:

- "Add a new map layer for X — it's available on data.gov.uk as a GeoJSON feed at this URL."
- "Change the colour/label/zoom level of the green belt layer."
- "Run the flood risk zones fetcher on the production server."
- "Create a migration to add a column to the listed buildings table."
- "Deploy the latest code to production."

Things to be careful about:

- Running a fetcher or migration against the live database will change real data. Confirm you mean production before asking Claude Code to do it.
- Deploying with `kamal deploy` pushes to the live site immediately. There is no staging environment.
- Any change to a SQL query file (`db/queries/*.sql`) requires a follow-up `make sqlc` to regenerate TypeScript — without this the app may fail to build.
- `db/schema.sql` is auto-generated. Do not edit it by hand; it is overwritten whenever a migration runs.

---

## Running it locally

You need Docker, Bun, and Go installed.

**1. Copy the env file and fill in values:**

```bash
cp .env.example .env
# Edit .env — at minimum set DATABASE_URL, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB
# and NEXT_PUBLIC_GOOGLE_MAPS_API_KEY.
# For local dev you can leave DNO keys blank (those layers just won't load).
```

**2. Start the database:**

```bash
docker compose up -d
```

This starts a PostGIS 17 container. The database is empty on first run.

**3. Apply the schema:**

```bash
cat db/schema.sql | psql -d postgres://postgres:postgres@localhost:5432/scout
```

**4. Install dependencies:**

```bash
bun install
```

**5. Run the dev server:**

```bash
bun run dev
```

The app is available at http://localhost:3000 (or a `.localhost` URL if you use portless).

**6. (Optional) Fetch data into the local database:**

```bash
bun run fetch-all          # all datasets — slow, several GB
bun run fetch ext-datagovuk-green-belt   # one dataset by name
bun run list-fetchers      # see all available names
```

**To stop and delete the local database:**

```bash
docker compose down -v
```

---

## Deployment

Scout runs on a single Hetzner VPS at `157.180.79.13`. Deployment is managed by [Kamal](https://kamal-deploy.org), which builds a Docker image, pushes it to GitHub Container Registry (`ghcr.io`), and starts it on the server. The database (`scout-db`) is a separate Kamal-managed container on the same server, bound only to `127.0.0.1:5432` (not reachable from the internet).

**To deploy:**

```bash
kamal deploy
```

Run this from the `microsites/scout` folder on a machine that has Kamal installed and the secrets available (see "Keys, secrets & accounts" below). Kamal builds the image remotely on the Hetzner server itself (not your laptop), so it must be able to SSH to `root@157.180.79.13`.

**To install Kamal:**

```bash
brew install ruby
# Add to .zshrc: export PATH="/opt/homebrew/opt/ruby/bin:$PATH"
#                export PATH="$(ruby -e 'print Gem.user_dir')/bin:$PATH"
gem install kamal
```

**To SSH into the server:**

```bash
ssh root@157.180.79.13
```

**To inspect what is running:**

```bash
docker ps                                   # all containers
docker logs <container-name> -f             # stream logs
```

Scout's web container is labelled `service=scout` / `role=web`. The database container is `scout-db`.

**To roll back to the previous image:**

```bash
kamal rollback
```

**To run a database migration on the production server:**

```bash
make server-migrate-up     # runs migrations inside the live web container
make server-migrate-down   # rolls back one migration
```

---

## Keys, secrets & accounts

> **Migrated to CBP Infisical on 2026-05-26.** All secrets below now live in the CBP-owned Infisical account (`contact@britishprogress.org`, **EU Cloud** — `https://eu.infisical.com`), project **`scout`**, environment **`prod`**. They are no longer read from Freddie's Bitwarden. See `HANDOVER_SECRETS_MIGRATION.md` for the full record.

| Service | Purpose | Account owner | Where the key lives now | Notes |
|---|---|---|---|---|
| GitHub Container Registry | Push/pull Docker images | **CBP GitHub org (`britishprogress`)** | Infisical `scout/prod` (`KAMAL_REGISTRY_PASSWORD`) | Image moved to `ghcr.io/britishprogress/scout`. Token is currently a PAT on the `vogon101` account (`write:packages`); consider a CBP-owned GitHub account long-term. |
| Google Maps | Map display tiles | Freddie's personal Google Cloud | Infisical `scout/prod` (`NEXT_PUBLIC_GOOGLE_MAPS_API_KEY`) | **Key value migrated, but the upstream Google Cloud *project* is still Freddie's personal account — reissue the key from a CBP-owned GCP account and replace the value.** |
| PostgreSQL | Database password | Freddie / Hetzner | Infisical `scout/prod` (`POSTGRES_PASSWORD`, `DATABASE_URL`) | No longer hardcoded in `config/deploy.yml`. |
| JWT signing | Scout's own auth tokens | N/A | Infisical `scout/prod` (`JWT_SECRET_KEY`) | Consider rotating (briefly existed in a temp file during migration). |
| DNOS ENW | Electricity North West data | ENW | Infisical `scout/prod` (`DNOS_ENW_API_KEY`) | — |
| DNOS UKPN | UK Power Networks data | UKPN | Infisical `scout/prod` (`DNOS_UKPN_API_KEY`) | — |
| DNOS NGED | National Grid Electricity Distribution | NGED | Infisical `scout/prod` (`DNOS_NGED_API_KEY`) | — |
| DNOS NPG | Northern Powergrid | NPG | Infisical `scout/prod` (`DNOS_NPG_API_KEY`) | — |
| DNOS SPEN | SP Energy Networks | SPEN | Infisical `scout/prod` (`DNOS_SPEN_API_KEY`) | — |
| DNOS SSEN | Scottish & Southern Electricity Networks | SSEN | Infisical `scout/prod` (`DNOS_SSEN_API_KEY`) | **Value is a placeholder (`todo`) carried over from Bitwarden — needs a real key for the SSEN fetcher to work.** |

**Local dev:** Values live in `microsites/scout/.env` (gitignored). Copy from `.env.example` and fill in.

**Production deploys:** Kamal reads secrets at deploy time by running `.kamal/secrets`, which logs in to CBP Infisical with the `scout-kamal-deploy` machine identity (Universal Auth) and fetches the `scout/prod` secrets. The machine-identity `INFISICAL_CLIENT_ID` / `INFISICAL_CLIENT_SECRET` live in `microsites/scout/.env` (gitignored). Any machine with those two values and the Infisical CLI can deploy — it is no longer tied to Freddie's Bitwarden.

### Remaining handover follow-ups

The Infisical migration itself is complete. Still outstanding (see `HANDOVER_SECRETS_MIGRATION.md` Part G + the Google Cloud section):

1. **Google Maps key** (`NEXT_PUBLIC_GOOGLE_MAPS_API_KEY`): the *value* is in Infisical, but it's still issued from Freddie's personal Google Cloud project. Reissue from a CBP-owned GCP account (Maps JavaScript API, HTTP-referrer-restricted to the two Scout hosts), update the value in Infisical, redeploy, delete the old key.
2. **Revoke the old Bitwarden access** once you're confident: delete the Bitwarden Secrets Manager token/project and remove `BWS_ACCESS_TOKEN` from `microsites/scout/.env` (kept for now as a rollback path).
3. **Rotate the sensitive keys** that briefly touched a file during migration (`JWT_SECRET_KEY`, DB password) — optional but recommended.
4. **Registry token**: currently a PAT on the `vogon101` personal account. For a clean CBP handover, mint it from a CBP-owned GitHub account that's a member of the `britishprogress` org.

---

## When something breaks

**The site is down / returning errors:**

```bash
ssh root@157.180.79.13
docker ps                                       # is the scout web container running?
docker logs <scout-web-container-name> -f       # check for startup errors
docker ps -a                                    # look for exited containers
```

If the container has crashed, `kamal rollback` will restore the previous image. If the database container is down, `kamal accessory start scout-db`.

**A data layer shows nothing:**

That layer's data has probably never been fetched, or the fetch failed. Re-run the fetcher:

```bash
make server-fetch/ext-datagovuk-green-belt     # replace with the layer name
# or fetch everything:
make server-fetch-all
```

Use `bun run list-fetchers` to see exact names.

**Deploy fails because it can't read secrets:**

The `.kamal/secrets` file logs in to CBP Infisical (EU Cloud) with the `scout-kamal-deploy` machine identity and fetches `scout/prod`. If the deploy can't read secrets, check, in order: (1) `INFISICAL_CLIENT_ID` / `INFISICAL_CLIENT_SECRET` are present in `microsites/scout/.env`; (2) the machine identity still has read access to the `scout` project's `prod` env; (3) `kamal secrets print` from `microsites/scout` lists every name with a value. The CLI must reach `https://eu.infisical.com` — every call in `.kamal/secrets` is pinned to that domain.

---

## Open questions / known issues

- `documentation/database/00_setup.md` and `01_migrations.md` both contain `# todo` stubs — the sections about `init-schema.sql`, `kamal remove`, `kamal setup`, and local migration testing are incomplete.
- `documentation/new_layer.md` step 1 is noted as wrong ("We don't edit the schema file directly. How we will do migrations in Scout is TBC.") — the correct flow is to create a Goose migration, not to edit `db/schema.sql` by hand.
- `NEXT_PUBLIC_GIT_SHA` in `config/deploy.yml` is set to the literal string `TODO` — it should be wired to the actual Kamal version or git SHA.
- There is no staging environment. All deployments go directly to production.
- The `goose-builder` stage in the `Dockerfile` installs `goose@latest` against a pinned `golang:1.25-alpine` base. `@latest` is unpinned, so a future goose release that requires Go > 1.25 will break the build again — pin a goose version when convenient.
- The container image now publishes to `ghcr.io/britishprogress/scout` (CBP org). The old `ghcr.io/buildwithtract/scout` package is orphaned and can be deleted once the new one is confirmed working.

scout

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README