CandleCollector

KO | EN

gitlite — search

#ai #llm #go #rust #linux #docker #macos #nodejs #vue #pytorch #security #openai #ios #open-source

CandleCollector

(★ 14)

A real-time stock price tracker.

CandleCollector Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# CandleCollector

A real-time stock candle collector for the ESP32 ecosystem. The device fetches
real 1-minute OHLCV bars (open, high, low, close, **volume**) from the Yahoo
Finance chart API and POSTs them to a local FastAPI + SQLite server, giving you
a continuously-growing candle database that downstream apps (backtesters,
dashboards, trading tools) can read over HTTP or straight from the SQLite file.

ESP32-class devices are a great fit for this repetitive, low-compute task:
small, low power, WiFi-connected, and always on.

## How it works

```
ESP32 (CandleCollector firmware)                 Server (this repo, server/)
┌────────────────────────────────┐   HTTP POST   ┌──────────────────────────┐
│ WiFi portal (NVS + fallback AP)│  /candles     │ FastAPI ingest + query   │
│ Yahoo 1m OHLCV windowed fetch  ├──────────────▶│ SQLite (candles.db)      │
│ market-hours gating + backfill │               │ GET /candles, .csv, /stats│
└────────────────────────────────┘               └──────────────────────────┘
```

- **Real bars, not approximations**: the firmware requests Yahoo's own
  1-minute bars with `period1/period2` windows (~1-2 KB payloads), so OHLC and
  volume match what mainstream chart providers show.
- **Boot backfill**: on startup the device backfills the last 24h of bars in
  30-minute chunks, so gaps from downtime heal themselves. The server upserts
  on `(symbol, interval_sec, start_time)` — re-posting is idempotent.
- **Live tail**: during market hours it re-fetches a sliding 10-minute window
  every 20s, which both appends new candles and repairs late volume revisions.
- **Market hours enforced**: configurable session (default 6:30–13:00 PT),
  weekends skipped, plus a 5-minute grace after the close so the final candle
  of the session is captured.

### Yahoo windowed-API quirks (handled by the firmware)

Discovered empirically; all are compensated for in `market_data.cpp` / `main.cpp`:

1. A bar that would extend past `period2` is dropped from the response —
   windows overlap by one bar so boundary bars aren't lost.
2. The **first bar returned in any windowed request has its volume zeroed**
   (prices are correct). Windows over-reach by an extra bar and the parser
   skips the `period1`-edge bar, so every stored bar was mid-window somewhere.
3. When the window opens on a trading gap (the 6:30 session open), the
   gap-leading bar is unavoidably first — its volume is repaired with one
   reach-back request into the previous session (72h reach covers weekends;
   the rare post-holiday Monday-gap open may keep volume 0).
   Additionally, the live sliding window means each bar's *last* re-post is
   the one where it has become first-in-window (volume zeroed) — the server
   guards against this by never letting a zero-volume upsert overwrite a
   stored volume > 0.
4. Yahoo appends the most recent regular-market bar to every windowed
   response even when it's outside the window (harmless — upserted).

## WiFi onboarding (no hardcoded credentials)

The firmware boots a WiFi portal (same pattern as Cyclops):

1. Saved networks live in NVS; on boot the device scans and joins the
   strongest known network.
2. If none is reachable it raises a fallback access point:
   **SSID `CandleCollector`, password `candles123`, page `http://192.168.8.1/wifi`**.
3. On that page you can scan, enter credentials, join, and forget networks
   (up to 6 saved).
4. Once joined, the device is reachable at `http://candlecollector.local`
   (`/wifi` for the portal, `/status` for collector state).

A legacy `src/credentials.h` (copied from `credentials_template.h`) seeds the
saved-network list on first boot if present, but is no longer required.

## Server

```bash
cd server
uv sync
uv run uvicorn app:app --host 0.0.0.0 --port 8001
```

Endpoints:

| Endpoint | Description |
|---|---|
| `POST /candles` | Ingest one candle (the ESP32 calls this) |
| `POST /candles/bulk` | Batch ingest (the nightly EOD fetcher calls this) |
| `GET /candles?symbol=SPY&start=&end=&limit=` | Query candles as JSON |
| `GET /candles.csv?symbol=SPY` | Same data as CSV (chronological) |
| `GET /stats` | Row counts and time ranges per symbol |
| `GET /options?symbol=SPY&snapshot=latest&right=C&expiration=&limit=` | Query the options panel as JSON (newest first) |
| `GET /options.csv?symbol=SPY&snapshot=latest` | Same data as CSV (chronological) |
| `GET /options/snapshots?symbol=SPY` | Distinct snapshot times + contract counts |
| `GET /options/stats` | Per-symbol contract/snapshot/expiration counts |
| `GET /health` | Liveness check |

### Options panel (intraday IV surface)

`server/options_snapshot.py` snapshots option chains for **SPY + the Mag 7**
every 10 minutes during market hours (`candlecollector-options.timer`) into a
separate `options.db` — expirations within ~1 month, and the 20 strikes (10
each side) bracketing the day's opening price. Yahoo has no options-history
endpoint, so the table only accumulates *forward*;
replaying it by `snapshot_time` reconstructs the intraday implied-vol surface.
The feed is ~15 min delayed (the snapshotter skips the first ~15 min after the
open, when the book is still empty). See `server/DATA_ACCESS.md` for the schema,
query params, and IV caveats.

### S&P 500 EOD coverage

Beyond the live-streamed set, `server/sp500_eod.py` pulls the **full
1-minute session for every S&P 500 constituent** (plus SPY) once per day
after the close — ~500 `range=1d` requests against Yahoo, throttled, in a
few minutes. So the database carries intraday 1m bars for the whole index,
delivered at EOD, while the ESP32 streams the configured handful live.
The nightly run also re-covers the streamed symbols, acting as a daily
data healer.

- Symbol universe: `server/sp500.txt` (Yahoo-style tickers, `BRK-B`).
  Refresh from Wikipedia with `uv run python sp500_eod.py --refresh-symbols`.
- Scheduling: systemd user timer `candlecollector-sp500.timer`
  (weekdays 13:20 PT; `Persistent=true` catches up missed runs).
- Idempotent and safe to re-run by hand: `uv run python sp500_eod.py`.

**Coverage audit + heal**: after the per-symbol pull, the run audits the last
~5 trading days of stored data. SPY's `range=5d` response acts as the
trading-day calendar and per-day session length (so half-day sessions
calibrate themselves). Each symbol-day's bar count is normalized by SPY's and
compared to the symbol's own median coverage across the window — illiquid
names legitimately trade far fewer minutes than SPY, while real failures
(day missing from the DB entirely: missed timer run, server down, power
loss; or a truncated fetch) are all-or-nothing and sit near zero. Thin
symbols are refetched with `range=5d`. Yahoo serves 1m bars ~30 days back,
so a fully missed day stays recoverable for about a week, not just until the
next open. Anything still thin after healing is logged (`journalctl --user
-u candlecollector-sp500`) but not fatal — e.g. a recent IPO with no bars on
older days.

Downstream consumption options:

```python
# pandas, via HTTP
df = pd.read_csv("http://192.168.4.61:8001/candles.csv?symbol=SPY")

# pandas, straight from the SQLite file
df = pd.read_sql("SELECT * FROM candles WHERE symbol='SPY' ORDER BY start_time",
                 sqlite3.connect("server/candles.db"))
```

Schema: `candles(symbol, interval_sec, start_time, end_time, open, high, low,
close, volume, received_at)` with primary key
`(symbol, interval_sec, start_time)`. Timestamps are unix epoch UTC.
1-minute candles are the recommended base interval — longer timeframes can be
resampled from them downstream.

## Firmware configuration

All settings live in `src/config.h`:

- `stock`: symbol (`SPY`), fetch cadence (20s), candle duration (60s),
  market hours + enforcement, `useTestData` toggle
- `database`: server host/port/path (defaults to `192.168.4.61:8001/candles`)
- `collector`: backfill depth (24h), backfill chunk (30 min), live window (10 min)
- `ap`: fallback AP SSID/password and mDNS hostname

Test mode (`useTestData = true`) runs the original random-walk generator
through the local candle buffer and posts under the symbol `TEST`, so
simulated data can never collide with real rows.

## Build & flash

```bash
pio run -t upload           # build + flash over USB (ESP32-S3, see platformio.ini)
pio device monitor          # serial output at 115200 baud
```

### OTA (wireless) updates

Once an OTA-enabled build is on the board (any build since `src/ota_update.*`
exists), updates no longer need a cable:

```bash
pio run -e esp32s3-xiao-ota -t upload   # push over WiFi (espota, port 3232)
```

The OTA password is the fallback-AP password (`Config::ap.pass`); the env's
`--auth` flag must match. Confirm an update landed via the `fw` build-stamp in
`/status`. A failed transfer is harmless — it writes the inactive app slot and
the running firmware keeps going; just retry (transfers can fail if a fetch
is in flight, e.g. during boot backfill).

## Contributing

Create a fork of the repo, and hack away! If submitting a pull request, please
install `pre-commit`, or the PR will fail.

```bash
pip install pre-commit
pre-commit install
```

## Chart

A dependency-free candlestick chart (price + volume, symbol and range
filters, hover tooltips, light/dark) is served from two places:

- `http://<server>:8001/chart` — hosted by the data server (most reliable)
- `http://candlecollector.local/chart` — hosted by the device itself
  (needs mDNS on the browsing device; use the device IP where mDNS is
  unavailable, e.g. Android)

`server/chart.html` is generated from `src/chart_page.h` with a same-origin
API base — regenerate it if the page changes.

## Roadmap

- [x] Volume tracking
- [x] Local database storage (SQLite via FastAPI ingest server)
- [x] WiFi onboarding portal (NVS + fallback AP)
- [x] Multi-symbol collection (SPY + Magnificent 7 by default)
- [x] On-device chart page (`/chart`)
- [x] OTA firmware updates (ArduinoOTA / espota)
- [ ] Grafana/visualization examples

CandleCollector

CandleCollector Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

CandleCollector

CandleCollector Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README