Opentrace-ResearchTwin

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

Opentrace-ResearchTwin

★ 21 Open GitHub ↗

OpenTrace is a browser-based research agent that captures web browsing sessions, analyzes pages with LLMs, and writes structured research notes into Obsidian.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# OpenTrace

**Capture research journeys — not just webpages.**

A browser extension + local backend that records *how* you explore, what you read, and how ideas connect — then turns it into structured **Obsidian** notes with optional **LLM** analysis.

<br/>

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://raw.githubusercontent.com/776306542/Opentrace-ResearchTwin/main/LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-3776AB?logo=python&logoColor=white)](https://www.python.org/)
[![FastAPI](https://img.shields.io/badge/FastAPI-0.109+-009688?logo=fastapi&logoColor=white)](https://fastapi.tiangolo.com/)
[![Vue 3](https://img.shields.io/badge/Vue-3-4FC08D?logo=vue.js&logoColor=white)](https://vuejs.org/)
[![Chrome MV3](https://img.shields.io/badge/Chrome-MV3-4285F4?logo=googlechrome&logoColor=white)](https://developer.chrome.com/docs/extensions/mv3/)

[Features](#-features) ·
[How it works](#-how-it-works) ·
[Architecture](#-architecture) ·
[Quick start](#-quick-start) ·
[Configuration](#-configuration) ·
[Roadmap](#-roadmap)

</div>

---

## The problem

You open twenty tabs for one question. A week later you remember *what* you found — but not *why* you opened tab #7, what you skipped, or how your thinking evolved.

| Typical tools | What you get |
|---------------|--------------|
| Bookmarks | URLs without context |
| Web clippers | Static snapshots, no session story |
| Browser history | Timestamps, no summaries or structure |

**OpenTrace** treats browsing as a **research session**: ordered captures, scroll depth, structured excerpts, session summaries, and Mermaid path graphs — all landing in your Obsidian vault as plain Markdown you own.

---

## ✨ Features

### 🧭 Research session tracking

Start a session, browse normally, end when done. OpenTrace records:

- Pages visited **in order** (your exploration path)
- **Scroll depth** per page (skim vs. deep read)
- Auto-capture on navigation + manual **Save current page**
- Pause / resume without losing the session

### 📄 Intelligent page extraction

Content scripts wait for stability, scroll to trigger lazy-loaded content, then extract:

| Field | Source |
|-------|--------|
| Title, URL, canonical, meta | DOM + metadata |
| Main article | [Mozilla Readability](https://github.com/mozilla/readability) |
| Markdown body | [Turndown](https://github.com/mixmark-io/turndown) |
| Headings, tables, code, images, links | Structured DOM pass |

Large pages are **chunk-uploaded** so nothing gets silently truncated.

### 📝 Obsidian-native output

Every capture becomes a YAML-frontmatter Markdown note. Sessions get a summary with ranked materials, conclusions, **information debt**, and next steps.

```text
YourVault/
├── 00_Inbox/WebClips/          # one file per captured page
├── 10_ResearchSessions/        # session summary + Mermaid graph
├── 20_KnowledgeCards/          # reserved — knowledge cards (roadmap)
└── 30_TraceGraphs/             # reserved — trace visualizations (roadmap)
```

**Example WebClip sections:** One-sentence summary · Key points · Research value · Useful for · Limitations · Page info · Body excerpt · Source link

**Example session sections:** Topic judgment · Research path · Key materials · Preliminary conclusions · Information debt · Next steps · Mermaid diagram

### 🤖 AI analysis (optional, local-first)

| Provider | Config |
|----------|--------|
| **None** (default) | Rule-based summaries — no API key |
| **Ollama** | Local models on your machine |
| **OpenAI-compatible** | DeepSeek, OpenRouter, vLLM, etc. |

Per page: summary, tags, source type (`paper` / `docs` / `blog` / …), research value (`high` / `medium` / `low`).  
Per session: topic judgment, conclusions, gaps, recommended next steps.

### 🔒 Privacy by design

- Runs on **127.0.0.1** — data stays on your machine
- No cookies, `localStorage`, passwords, or form field capture
- Captures **only** while a research session is active
- Vault files are plain `.md` — no vendor lock-in

---

## 🔄 How it works

```mermaid
sequenceDiagram
    participant You as Researcher
    participant Ext as Browser Extension
    participant CS as Content Script
    participant API as OpenTrace Server
    participant DB as SQLite
    participant LLM as LLM (optional)
    participant Vault as Obsidian Vault

You->>Ext: Start research
    Ext->>API: POST /api/session/start
    API->>DB: Create session

You->>Ext: Browse / open tabs
    Ext->>CS: EXTRACT_PAGE (wait + scroll + Readability)
    CS-->>Ext: PagePayload
    Ext->>API: POST /api/capture/page
    API->>LLM: Summarize page (if enabled)
    LLM-->>API: Structured JSON
    API->>DB: Store page
    API->>Vault: Write WebClip .md

You->>Ext: End research
    Ext->>API: POST /api/session/end/{id}
    API->>LLM: Session trace (if enabled)
    API->>Vault: Write session summary .md
    API-->>Ext: Done
```

1. **Start** — extension creates a session on the local server.  
2. **Browse** — each completed navigation triggers extraction + capture (deduped by URL).  
3. **Analyze** — server runs rule or LLM summarization.  
4. **Write** — Markdown lands in your vault; metadata lives in SQLite for API queries.  
5. **End** — session report + Mermaid research path graph.

---

## 🏗 Architecture

### System overview

```mermaid
flowchart TB
    subgraph Browser["🌐 Chrome / Edge (Manifest V3)"]
        Popup["Popup UI<br/><i>Vue 3</i>"]
        BG["Background SW<br/><i>session state, tab events</i>"]
        CS["Content Script<br/><i>Readability + Turndown</i>"]
        Popup <--> BG
        BG <--> CS
    end

subgraph Local["💻 localhost:9876"]
        API["FastAPI<br/>session · capture · health"]
        Sum["Summarizer<br/>rules + LLM"]
        Trace["Trace Builder<br/>path + Mermaid"]
        MD["Markdown Writer"]
        DB[(SQLite)]
        API --> Sum
        API --> Trace
        API --> MD
        API --> DB
        Sum --> Trace
    end

subgraph Optional["☁️ Optional"]
        Ollama["Ollama"]
        OpenAI["OpenAI-compatible API"]
    end

subgraph Output["📓 Your knowledge base"]
        Vault["Obsidian Vault<br/>.md + YAML frontmatter"]
    end

BG -->|REST JSON| API
    Sum -.-> Ollama
    Sum -.-> OpenAI
    MD --> Vault

style Browser fill:#e8f4fc
    style Local fill:#f0fdf4
    style Output fill:#fef9c3
```

### Backend modules

```text
opentrace-server/app/
├── api/
│   ├── health.py      # GET  /api/health
│   ├── session.py     # POST /api/session/start | end
│   └── capture.py     # POST /api/capture/page (chunked upload)
├── core/
│   ├── summarizer.py      # per-page analysis
│   ├── trace_builder.py   # session summary + Mermaid
│   ├── markdown_writer.py   # Obsidian file output
│   └── llm_client.py      # none | ollama | openai-compatible
├── db/                    # SQLAlchemy models + SQLite
└── config.py              # vault path, LLM, limits
```

### Extension modules

```text
opentrace-extension/src/
├── popup/           # Vue UI — start / pause / end / save
├── background/      # session FSM, tab.onUpdated, API calls
├── content/         # enhanced-extract.ts (scroll + Readability)
└── shared/          # types, API client, chunked upload
```

---

## 🚀 Quick start

### Prerequisites

- **Python 3.10+** and `pip`
- **Node.js 18+** and `npm` (for the extension build)
- **Chrome** or **Edge**
- **Obsidian** (or any folder you treat as a Markdown vault)

### 1. Clone

```bash
git clone https://github.com/YOUR_USERNAME/opentrace.git
cd opentrace
```

### 2. Backend

```bash
cd opentrace-server
pip install -r requirements.txt
python -m uvicorn app.main:app --host 127.0.0.1 --port 9876 --reload
```

Health check: [http://127.0.0.1:9876/api/health](http://127.0.0.1:9876/api/health)

### 3. Vault path (optional)

Default output: `vault-template/` in the repo root.

```powershell
# Windows PowerShell
$env:OPENTRACE_VAULT_PATH="C:\Users\You\Documents\MyObsidianVault"
```

```bash
# macOS / Linux
export OPENTRACE_VAULT_PATH="$HOME/Documents/MyObsidianVault"
```

### 4. Extension

```bash
cd opentrace-extension
npm install
npm run build
```

Load unpacked extension:

| Browser | URL |
|---------|-----|
| Chrome | `chrome://extensions` |
| Edge | `edge://extensions` |

1. Enable **Developer mode**  
2. **Load unpacked** → select `opentrace-extension/dist`  
3. Pin **OpenTrace** to the toolbar

### 5. First session

1. Confirm the popup shows the server as online (no red backend warning).  
2. Click **Start research**.  
3. Open a few pages (docs, GitHub, articles).  
4. Click **End research**.  
5. Open your vault — check `00_Inbox/WebClips/` and `10_ResearchSessions/`.

---

## ⚙ Configuration

### Environment variables

| Variable | Description | Default |
|----------|-------------|---------|
| `OPENTRACE_VAULT_PATH` | Obsidian vault root | `./vault-template` |
| `OPENTRACE_DATABASE_URL` | SQLite URL | `opentrace-server/data/opentrace.db` |
| `LLM_PROVIDER` | `none` · `ollama` · `openai-compatible` | `none` |
| `LLM_BASE_URL` | API base URL | — |
| `LLM_MODEL` | Model name | — |
| `LLM_API_KEY` | API key (cloud providers) | — |
| `LLM_TIMEOUT` | Request timeout (seconds) | `90` |

### Ollama (local)

```powershell
$env:LLM_PROVIDER="ollama"
$env:LLM_BASE_URL="http://127.0.0.1:11434"
$env:LLM_MODEL="qwen2.5:7b"
```

### DeepSeek / OpenAI-compatible

```powershell
$env:LLM_PROVIDER="openai-compatible"
$env:LLM_BASE_URL="https://api.deepseek.com"
$env:LLM_MODEL="deepseek-chat"
$env:LLM_API_KEY="your-api-key"
```

### Rule-only mode (no LLM)

```powershell
$env:LLM_PROVIDER="none"
```

Works offline — uses heuristics for summaries, tags, and session reports.

---

## 📡 API

| Method | Endpoint | Description |
|--------|----------|-------------|
| `GET` | `/api/health` | Server status, vault path, LLM provider |
| `POST` | `/api/session/start` | Begin research session |
| `POST` | `/api/session/end/{session_id}` | End session → write summary |
| `GET` | `/api/session/{session_id}` | Session metadata + page count |
| `POST` | `/api/capture/page` | Capture page (supports chunked body) |

<details>
<summary><b>Example: health response</b></summary>

```json
{
  "status": "ok",
  "version": "0.2.0",
  "vault_path": "C:\\...\\vault-template",
  "llm_provider": "none",
  "server_port_hint": 9876
}
```

</details>

---

## 📂 Project structure

```text
opentrace/
├── opentrace-extension/     # Vue 3 + TypeScript Chrome extension
├── opentrace-server/        # FastAPI + SQLite + LLM layer
├── vault-template/          # Default Obsidian vault layout
└── README.md
```

See also: [opentrace-server/README.md](opentrace-server/README.md) for server-only docs.

---

## 🛠 Tech stack

| Layer | Technologies |
|-------|----------------|
| Extension | Vue 3, TypeScript, Vite, Manifest V3 |
| Extraction | Mozilla Readability, Turndown |
| Backend | Python, FastAPI, SQLAlchemy, Pydantic |
| Storage | SQLite (sessions + pages) |
| Output | Markdown, YAML frontmatter, Mermaid |
| AI | Ollama, OpenAI-compatible APIs (optional) |

---

## 🗺 Roadmap

| Version | Focus |
|---------|--------|
| **v0.3** | Research debt detection · knowledge card generation · richer prompts · session comparison |
| **v0.4** | Local vector search · semantic retrieval · similar-session discovery |
| **v0.5** | LangGraph · multi-agent research workflows |
| **v1.0** | Research digital twin · knowledge evolution tracking |

**Current (v0.2):** session tracking · enhanced extraction · Obsidian writes · rule + LLM summaries · Mermaid path graphs · chunked upload.

---

## 🤝 Contributing

Contributions welcome — issues, ideas, and PRs.

1. Fork the repo  
2. Create a feature branch (`git checkout -b feature/amazing-idea`)  
3. Commit your changes  
4. Open a Pull Request

For larger changes, open an issue first to align on direction.

---

## 📄 License

[MIT License](LICENSE) — use freely, attribution appreciated.

---

**OpenTrace is not a web clipper.**

It is a step toward a *digital twin of how you think while you research.*

If this project helps your workflow, consider giving it a ⭐ on GitHub.

</div>

Opentrace-ResearchTwin

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README