Relay

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 35 Open GitHub ↗

Mobile PWA that turns hard-to-understand speech into clear phrases using local Gemma 4 via Ollama, no cloud LLM on the interpret path.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Relay

Relay is a **mobile-first progressive web app (PWA)** for people whose speech is hard to understand—whether from ALS, stroke-related aphasia, dysarthria, Parkinson’s, or similar conditions. It captures voice (and optional camera context), runs speech-to-text in the browser or an optional **local-only STT sidecar**, sends text to a **local Gemma model via Ollama**, and shows a clear interpretation the user can hear again with **browser text-to-speech**, confirm, and route to caregivers when configured.

**Wedge (one line):** Relay pairs **browser-native speech** with **local Gemma 4 via Ollama** so disfluent or fragmented speech can become a clear, caregiver-ready phrase—**without cloud inference** for the core interpret path—plus honest errors when the model is unreachable.

The codebase prioritizes **real browser APIs** and **honest failure modes**: if Ollama is unreachable or misconfigured, the app surfaces that error instead of inventing model output.

---

## What you can do in the app

| Area | Description |
|------|-------------|
| **Onboarding** | First-run flow for role, identity, condition, voice samples (stored locally), language, caregiver details, personal phrases (for Gemma context), and accessibility preferences. Skippable steps where marked. |
| **Home (`/`)** | Large mic control, live transcript, interpretation card, symbol board, optional camera, “type instead” inline composer, language switcher, and connection status. |
| **Caregiver (`/caregiver`)** | Today’s interactions, patient dictionary, routing log, alerts timeline, and tool-built handover note—fed from local browser data after successful interpretations. |
| **Settings (`/settings`)** | Hub lists categories; each opens its own screen — e.g. `/settings/profile`, `/settings/language`, `/settings/accessibility`, `/settings/models` (Ollama URL, model tags, **connectivity** + test), `/settings/routing`, `/settings/developer`. |
| **About (`/about`)** | High-level architecture and how Gemma fits in. |

**Developer / QA:** append `?reset-onboarding=1` once to clear onboarding and voice sample metadata (see `App.tsx`).

---

## Tech stack

| Layer | Choice |
|-------|--------|
| UI | React 18, TypeScript (strict), Tailwind CSS |
| Build | Vite 5, `@vitejs/plugin-react` |
| Routing | React Router 6 |
| Offline install | `vite-plugin-pwa` (Workbox-generated service worker) |
| Icons | Lucide React |

---

## Prerequisites

- **Node.js 20+** (recommended)
- **npm** (or compatible client) for installs
- **Ollama** running somewhere your **browser** can reach (same machine, LAN, or tunneled host), with models pulled that match **Settings → Models & connectivity** tags (defaults in Advanced unless changed)

---

## Setup and scripts

```bash
npm install
npm run dev          # Vite dev server → http://localhost:5173
npm run local-stt    # optional: local STT sidecar → http://127.0.0.1:9000
npm run typecheck    # tsc --noEmit
npm run build        # typecheck + production bundle + PWA assets
npm run preview      # serve the production build locally
```

### Local STT sidecar (optional)

Relay can POST recorded audio to a **local HTTP sidecar** when browser speech-to-text is empty, unreliable, or unavailable—see `src/services/localSttService.ts` and `scripts/local-stt-server.mjs`.

1. **Start the reference server:** `npm run local-stt` (default `http://127.0.0.1:9000`). Optional: set **`RELAY_STT_CMD`** so `/transcribe` runs your offline command (stdout = transcript); see comments in `scripts/local-stt-server.mjs`.
2. **Point the app at it:** copy [.env.example](.env.example) into **`.env.local`**, set **`VITE_RELAY_LOCAL_STT_URL=http://127.0.0.1:9000`**, then **restart** **`npm run dev`** (Vite only reads env on startup).
3. **Verify:** Home → tap-to-speak; with no real command configured, the sidecar still returns stub text so you can confirm wiring into `submit()`.

**If the UI says “Speech recognition needs a connection”:** the browser blocked Google’s cloud STT. Keep **`npm run local-stt`** running in a second terminal, ensure `.env.local` has the URL above, restart **`npm run dev`**, and hard-refresh the app.

**Trying on a phone:** run `npm run build && npm run preview -- --host`, open the printed URL on the same network, then use **Add to Home Screen** (iOS) or the install prompt (Android).

### Hackathon / judge quickstart

Use this path to verify the repo end-to-end (see also [docs/GEMMA_AND_INTEGRATIONS.md](docs/GEMMA_AND_INTEGRATIONS.md) for CORS and wire details).

1. **Clone** the repository and install **Node.js 20+**.
2. Run **`npm install`** at the repo root.
3. Install [Ollama](https://ollama.com) and start **`ollama serve`** on a host your browser can reach.
4. Pull the three default model tags (must match **Settings → Models & connectivity** unless you change both code defaults and settings):

```bash
   ollama pull gemma4:e2b
   ollama pull gemma4:e4b
   ollama pull gemma4:27b
   ```

5. Run **`npm run dev`** and open **http://localhost:5173** (or the URL Vite prints).
6. Open **Settings → Models & connectivity** (`/settings/models`) and confirm **Ollama base URL** (e.g. `http://127.0.0.1:11434`) and the **same three tags** as in step 4 (under Advanced if you overrode defaults).
7. If the app is served from **https://** but Ollama is **http://** on another host, the browser may block requests (mixed content) or require **CORS** on Ollama—see [docs/GEMMA_AND_INTEGRATIONS.md](docs/GEMMA_AND_INTEGRATIONS.md#ollama-wire-protocol-what-judges-can-grep).
8. Optional: open **`/?reset-onboarding=1`** once to reset first-run onboarding (see `App.tsx`).
9. **Happy path:** Home → allow microphone → speak → wait for interpretation card → use read-aloud (TTS).
10. **Multimodal path:** Enable camera, capture context, speak → routing should prefer the **27B** tier for `vision+speech` / compound (see [docs/MODEL_ROUTING.md](docs/MODEL_ROUTING.md)).

### Model tags vs Gemma 4 naming

Default Ollama tags are defined in **`src/lib/ollamaModelConfig.ts`** (`OLLAMA_MODEL_DEFAULT_TAG`). They must match tags you **pull** in Ollama and optional overrides under **Settings → Models & connectivity** (Advanced).

For **hackathons or updated Ollama image names**, verify the published **Gemma 4** tags against current docs—update **Settings → Models & connectivity** locally if names differ; this README may not track every rename.

---

## Ollama and Gemma

1. Install [Ollama](https://ollama.com) and start the daemon (`ollama serve`). The default API root is `http://127.0.0.1:11434` (often shown as `localhost:11434`).
2. Pull model tags on the Ollama host, then enter the **same tags** under **Settings → Models & connectivity** (Advanced) so routing and `interpret()` calls match what is installed.
3. In **Settings → Models & connectivity**, set **base URL** if Ollama is not on the default host—Relay resolves an empty base URL to the local default at runtime (`src/lib/ollamaUrl.ts`).
4. Open Relay from a context where **fetch** to that URL is allowed (same-origin, LAN, or CORS as applicable). If the server is down, you will see **`GemmaNotConnectedError`** (or related HTTP errors) in the UI—not fake transcripts.

Implementation detail: all interpretation goes through **`interpretationService.interpret()`** → **`GemmaInterpreterAdapter`** → `POST {baseUrl}/api/generate`. Prompts, timeouts, and multimodal fields live in that adapter.

---

## Architecture (short)

```
React UI
  └─ contexts: Settings, Session, ModelRouting
  └─ hooks: permissions, microphone, speech recognition / synthesis, camera, Ollama status
  └─ services: audio, STT, TTS, camera, routing
  └─ interpretationService.interpret(input)
        └─ GemmaInterpreterAdapter  →  Ollama HTTP API
  └─ IndexedDB: voice samples, patient dictionary, handover notes
```

Deeper diagrams, file-level notes, and browser caveats: **[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)**. Ollama behavior and integration seams: **[docs/GEMMA_AND_INTEGRATIONS.md](docs/GEMMA_AND_INTEGRATIONS.md)**.

---

## Implementation status (canonical)

Use this table as the source of truth for what is “real” versus stubbed. If another document disagrees, prefer **this README** and **`docs/`**.

| Area | Status | Notes |
|------|--------|--------|
| Microphone + RMS level | **Real** | `audioCaptureService`, `useMicrophone` |
| Speech-to-text | **Real** (Web Speech API where supported, optional local STT sidecar) | Unsupported / blocked browser STT → type-instead UI or `VITE_RELAY_LOCAL_STT_URL`; see architecture doc |
| Text-to-speech | **Real** | `speechSynthesis` with language-aware voice selection |
| Camera preview + frame | **Real** | Frame passed into `interpret()` as image data |
| Patient Dictionary | **Real** | IndexedDB corpus with add/edit/delete/import/export; entries injected into Gemma prompts |
| Multimodal fusion | **Real when Ollama is reachable** | Concurrent speech/camera/symbol channels route as compound input to 27B |
| Permissions UX | **Real** | Query, prompt, denied paths |
| Model routing policy | **Real** | `chooseModel` in `modelRouter.ts`—no LLM inside the router |
| Gemma / interpretation | **Real when Ollama is reachable** | Streaming + JSON parse + client-side checks; fails honestly otherwise |
| Bilingual patient / caregiver | **Real when Ollama is reachable** | Gemma returns `patientLanguageText` + `caregiverLanguageText` + `inferredSpeaker`; UI/TTS use `detectedLanguage` + bilingual routing. STT locale follows **session inference** (model + script heuristics + last turn), not biometrics — see `transcriptSpeakerHint.ts` |
| Handover agent | **Real when Ollama is reachable** | Client-side tool pipeline + **one** `POST /api/generate` JSON call (`HandoverAgent.ts`); note persisted to IndexedDB — **not** remote `/api/chat` tool loop |
---

## Key files (orientation)

| Topic | Location |
|-------|----------|
| Ollama client + prompts | `src/services/interpretation/GemmaInterpreterAdapter.ts` |
| Interpretation entry | `src/services/interpretationService.ts` |
| Bilingual hero routing | `src/lib/bilingualHero.ts` |
| Speaker inference (session + transcript) | `src/lib/transcriptSpeakerHint.ts`, `src/lib/conversationContext.ts` |
| Ollama URL resolution | `src/lib/ollamaUrl.ts` |
| Session + history | `src/contexts/SessionContext.tsx` |
| Persisted settings | `src/contexts/SettingsContext.tsx` (localStorage) |
| Patient dictionary | `src/lib/patientDictionary.ts` (IndexedDB) |
| Handover agent + tools | `src/services/interpretation/HandoverAgent.ts`, `src/services/interpretation/tools/` |
| Voice sample blobs | `src/lib/voiceSamples.ts` (IndexedDB) |

---

## Trust and safety

- **Ollama:** interpretation text and optional image payload leave the browser for **your** Ollama host. There is **no** mock interpretation path: failures surface as **`GemmaNotConnectedError`** (see `GemmaInterpreterAdapter`).
- **Local-only network posture:** production calls are limited to the configured Ollama base URL and optional local STT sidecar (`VITE_RELAY_LOCAL_STT_URL`). Relay does not use analytics, font CDNs, telemetry, or avatar fetches.
- **Urgency guard:** after each successful parse, client-side **`applyUrgencyGuard`** (`src/lib/urgencyGuard.ts`) prevents the model from **downgrading** urgency when the user’s transcript clearly indicates an emergency-class phrase.
- **Patient Dictionary:** personal signal corpus stays in IndexedDB on this device unless the user manually exports JSON.

Deployment hardening (CSP, secrets pattern): [docs/SECURITY.md](docs/SECURITY.md). License and hackathon winner obligations (informational): [docs/LICENSE_NOTES.md](docs/LICENSE_NOTES.md).

---

## Roadmap (ideas)

- Optional learned routing on top of `chooseModel`.
- Stronger voice personalization and exports (beyond current local profile + samples).

---

## Documentation index

| Document | Purpose |
|----------|---------|
| [docs/README.md](docs/README.md) | Index of `docs/` |
| [docs/SOURCE_LAYOUT.md](docs/SOURCE_LAYOUT.md) | `src/` folder map for orientation |
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Layers, flows, browser limits, persistence |
| [docs/GEMMA_AND_INTEGRATIONS.md](docs/GEMMA_AND_INTEGRATIONS.md) | Gemma/Ollama wire protocol + handover pipeline |
| [docs/MODEL_ROUTING.md](docs/MODEL_ROUTING.md) | Deterministic E2B / E4B / 27B routing (Cactus-style narrative) |
| [docs/GROUNDING.md](docs/GROUNDING.md) | Dictionary grounding + handover client tools |
| [docs/LATENCY_AND_METRICS.md](docs/LATENCY_AND_METRICS.md) | How to record interpretation latency (`latencyMs`) |
| [docs/SECURITY.md](docs/SECURITY.md) | Trust boundaries, CSP, env vars |
| [docs/LICENSE_NOTES.md](docs/LICENSE_NOTES.md) | MIT repo + hackathon CC-BY winner note (informational) |

---

## License

Relay is licensed under the [MIT License](LICENSE). Third-party notices: [NOTICE](NOTICE).

For **Kaggle / Gemma 4 Good Hackathon** rules: potential winners may have additional **sponsor license** obligations (e.g. CC-BY 4.0 on the winning submission)—see [docs/LICENSE_NOTES.md](docs/LICENSE_NOTES.md) and the official competition rules.

Relay

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README