Grimore-MD

KO | EN

gitlite — search

#python #react #javascript #reactjs #typescript #machine-learning #php #django #flask #discord #python3 #telegram-bot

★ 33 Open GitHub ↗

Local-first cognitive daemon for automated vault management Markdown. Semantic indexing, auto-tagging, and connection discovery using LLMs and vector search.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

```text
 ____    ____    ______            _____   ____    ____
/\  _`\ /\  _`\ /\__  _\   /'\_/`\/\  __`\/\  _`\ /\  _`\
\ \ \L\_\ \ \L\ \/_/\ \/  /\      \ \ \/\ \ \ \L\ \ \ \L\_\
 \ \ \L_L\ \ ,  /  \ \ \  \ \ \__\ \ \ \ \ \ \ ,  /\ \  _\L
  \ \ \/, \ \ \\ \  \_\ \__\ \ \_/\ \ \ \_\ \ \ \\ \\ \ \L\ \
   \ \____/\ \_\ \_\/\_____\\ \_\\ \_\ \_____\ \_\ \_\ \____/
    \/___/  \/_/\/ /\/_____/ \/_/ \/_/\/_____/\/_/\/ /\/___/
```

**An automated knowledge engine for your document vault**

[![Version](https://img.shields.io/badge/version-3.2.0-6B4BCB?style=for-the-badge)](https://raw.githubusercontent.com/kahz12/Grimore-MD/main/) [![Python](https://img.shields.io/badge/python-3.11%2B-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/) [![Local-First](https://img.shields.io/badge/Privacy-Local--First-2EA043?style=for-the-badge&logo=ollama&logoColor=white)](https://ollama.com/) [![License](https://img.shields.io/badge/License-MIT-E2B100?style=for-the-badge)](https://raw.githubusercontent.com/kahz12/Grimore-MD/main/LICENSE)

<br>

</div>

---

Grimore watches your vault — **Markdown, PDF, EPUB, DOCX, ODT, RTF, HTML, TXT** — auto-tags every document, builds a hybrid semantic index, and answers questions against it. Entirely through local LLMs. Nothing leaves your machine, and no API keys are required.

## See it in action

### Scan

One sweep over a mixed-format vault. Preflight verifies every adapter; watchdog keeps the index live thereafter.

### Ask

Type a question, the Oracle streams back an answer with citations into your own notes.

### Browse

Categories and tags discovered automatically by the cognition pipeline — explore them from the shell.

### Inspect

Live counters, chosen models, daemon state — all from inside the shell.

### Discover

`/help` lists every slash-command at the prompt — no doc-diving required.

## Quick start

Requires Python 3.11+, [Ollama](https://ollama.com) with a chat model (e.g. `qwen2.5:3b`) and an embedding model (e.g. `nomic-embed-text`) pulled, and a git-initialised vault.

```bash
git clone https://github.com/kahz12/Grimore-MD.git
cd Grimore-MD
python -m venv .venv && source .venv/bin/activate
pip install -e .

grimore preflight                                       # validate config + adapters
grimore scan --vault-path /path/to/vault --no-dry-run   # first full pass
grimore daemon start                                    # keep the index live
grimore shell                                           # conversational mode
```

Supported on **Linux**, **Windows**, and **Termux/Android** (heavy engines like PyMuPDF / OCR stay opt-in to keep the mobile install lean).

## Beyond the terminal

## Architecture

```mermaid
graph TD
    A["Document Vault<br/>md · pdf · epub · docx · odt · rtf · html · txt"] -->|watchdog| B["Adapter dispatch<br/>extension → extractor"]
    B -->|file_hash + content_hash| C{"Changed?"}
    C -->|no| Z["Skip"]
    C -->|yes| D["Git Guard snapshot"]
    D --> E["Cognition<br/>tags · category · summary<br/>section-aware chunks · vectors"]
    E --> F[("SQLite<br/>WAL + FTS5<br/>+ page/heading anchors")]
    F --> G["Sidecar writer<br/>(non-MD) / inline (MD)"]
    G --> A
    F --> H["Oracle RAG<br/>BM25 + cosine → RRF"]
    H --> I["CLI / Shell"]
```

- **Ingest** — one adapter per format behind a registry. Two-tier change detection: cheap SHA-256 of the bytes gates the expensive extraction step, so re-scans of unchanged documents cost nothing.
- **Cognition** — local LLM tags, summarises and files each document into one hierarchical category. Section-aware chunking preserves PDF page numbers and DOCX/EPUB headings so citations come back as `[[Title#p.42]]`.
- **Memory** — SQLite in WAL mode with FTS5 alongside per-chunk vectors keyed by `sha256(model ‖ chunk)`, so swapping embedders invalidates cleanly.
- **Retrieval** — BM25 and cosine fused via Reciprocal Rank Fusion. Degrades to either side alone if the other is unavailable.
- **Writeback** — Markdown sources get inline frontmatter + a `## Suggested Connections` block. Binary formats (PDF, EPUB, DOCX, ODT) get a sidecar `.md` under `.grimore/sidecars/` — originals are never mutated.

## Privacy

Local-first by construction: with `cognition.allow_remote = false` (the default), LLM calls are rejected unless the endpoint resolves to a loopback address — and the connection is pinned to that validated IP so DNS can't rebind it mid-request. Every destructive operation defaults to `--dry-run`. PII detection, prompt-injection neutralisation (note content **and** client-supplied conversation history), hardened XML parsing for office/EPUB documents (entity-expansion DoS defence), constant-time and brute-force-throttled API-token checks on every `/api/*` route (with `--strict-token` to require the token even from localhost — recommended on Android, where apps share the loopback interface), automatic git snapshots, and path containment via `SecurityGuard` round out the safety model — see the [user guide](docs/USER_GUIDE_EN.md#10-privacy--safety) for the full picture.

## Documentation

- 🇬🇧 [**English User Guide**](docs/USER_GUIDE_EN.md) — every command, the shell, configuration, troubleshooting.
- 🇪🇸 [**Guía de Usuario (Español)**](docs/USER_GUIDE_ES.md) — recorrido completo en español.

Terminal: `grimore <cmd> --help`. In the shell: `/help`.

## Stack

Python 3.11+ · Ollama · SQLite (WAL + FTS5) · Typer + Rich · prompt-toolkit · pypdf · beautifulsoup4 · striprtf · watchdog · structlog · GitPython

## License

Released under the [MIT License](LICENSE).

Grimore-MD

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README