KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #typescript #ai-agent #developer-tools #claude-code #codex #deepseek-harness #ai-agents #cli #react #mcp

knowledgedigest

★ 1 Open GitHub ↗

Portable knowledge database with PySide6 GUI and web viewer. Indexes documents (PDF, DOCX, TXT, MD, HTML), FTS5 search, optional LLM summarization.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# KnowledgeDigest — Portable Knowledge Database

**[Deutsch](README_de.md)** | [English](README.md)

[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/)
[![Pytest 92 Passed](https://img.shields.io/badge/pytest-92%20passed-brightgreen.svg)](https://github.com/file-bricks/knowledgedigest)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://raw.githubusercontent.com/file-bricks/knowledgedigest/main/LICENSE)
[![SQLite FTS5](https://img.shields.io/badge/search-FTS5-green.svg)]()
[![Local-First](https://img.shields.io/badge/privacy-local--first-blue.svg)]()
[![LLM-Ready](https://img.shields.io/badge/LLM-Ready-purple.svg)]()

> Portable, self-contained knowledge database — indexes documents, chunks them, and makes them searchable via FTS5. Optional LLM summarization. PySide6 GUI + Web Viewer.

> [!NOTE]
> **For AI Agents & LLM Pipelines**: KnowledgeDigest is built as an offline, zero-telemetry document retrieval database for LLM agents. Documents are chunked into sentence-bounded sections (~350 words) with BM25 FTS5 ranking, optional Claude Haiku / Gemini Flash summary queuing, and clean Python/CLI/REST interfaces.

```bash
git clone https://github.com/file-bricks/knowledgedigest
cd knowledgedigest
pip install -r requirements.txt
pip install -e .

# Desktop GUI (PySide6, dark theme, 3-panel layout)
python -m KnowledgeDigest --gui

# Web Viewer (stdlib-only, dark theme, localhost:8787)
python -m KnowledgeDigest --web

# CLI
python -m KnowledgeDigest status
```

🇩🇪 [Deutsche Dokumentation → README_de.md](README_de.md)

---

## Architecture & Data Flow

KnowledgeDigest transforms raw documents into a structured, searchable knowledge base:

```mermaid
flowchart TD
    subgraph Input["Document Ingestion"]
        DOCS["Raw Files (PDF, DOCX, HTML, TXT, MD)"]
    end

subgraph CoreEngine["KnowledgeDigest Core Engine"]
        EXT["Text Extraction (PyMuPDF, docx, html)"]
        CHUNK["Sentence-Bounded Chunking (~350 words)"]
        FTS["SQLite FTS5 Full-Text Search (BM25)"]
        SUMM["LLM Summarizer Queue (Haiku / Gemini Flash)"]
    end

subgraph Interfaces["Multi-Interface Access"]
        GUI["PySide6 Desktop GUI (3-Panel Layout)"]
        WEB["Stdlib Web Viewer (http://localhost:8787)"]
        CLI["CLI Command Suite (python -m KnowledgeDigest)"]
        API["Python API (from KnowledgeDigest import KnowledgeDigest)"]
    end

DOCS --> EXT
    EXT --> CHUNK
    CHUNK --> FTS
    CHUNK -.-> SUMM
    SUMM -.-> FTS
    FTS --> GUI
    FTS --> WEB
    FTS --> CLI
    FTS --> API
```

## What It Does

## Features

- **Portable**: JSON config, no hardcoded paths. Copy the folder, it works.
- **Two Frontends**: PySide6 Desktop GUI + stdlib Web Viewer
- **FTS5 Search**: SQLite FTS5 with BM25 ranking and snippet highlighting
- **Document Support**: PDF, DOCX, TXT, MD, HTML
- **LLM Summarization** (optional): Haiku batch processing per chunk
- **Directory Management**: Add/remove/scan directories via GUI or config
- **No LLM Required**: Indexing, search, and browsing work without any LLM

## Desktop GUI

3-panel splitter layout with dark theme:

| Panel | Content |
|-------|---------|
| **Left** | Indexed directories with document counts |
| **Center** | Document table (sortable, filterable) |
| **Right** | Preview (text, PDF via PyMuPDF, images, metadata) |

Toolbar: Add Directory, Scan, Search, Web Viewer, Settings

## Web Viewer

Standalone browser-based viewer (Python stdlib only, no dependencies):

- **Dashboard**: Document and summary statistics
- **Browse**: All documents with pagination
- **Search**: FTS5 full-text search with highlighting
- **Folders**: Browse by directory
- **Summaries**: LLM-generated summaries catalog
- **Document Detail**: Full chunk content, metadata, keywords

Startable standalone (`--web`) or from within the GUI (toolbar button).

## Interfaces

| Interface | Command | Status |
|-----------|---------|--------|
| **Desktop GUI** | `python -m KnowledgeDigest --gui` | v0.4.0 |
| **Web Viewer** | `python -m KnowledgeDigest --web` | v0.4.0 |
| **CLI** | `python -m KnowledgeDigest status` | v0.1.0+ |
| **Python API** | `from KnowledgeDigest import KnowledgeDigest` | v0.1.0+ |
| **SQLite Direct** | Any SQLite tool on `knowledge.db` | Always |

## Python API

```python
from KnowledgeDigest import KnowledgeDigest
from KnowledgeDigest.config import get_config

kd = KnowledgeDigest(config=get_config())

# Manage directories
kd.add_directory("/path/to/docs")
kd.scan_directory("/path/to/docs", recursive=True)

# Search
results = kd.search_all("machine learning", limit=10)
for r in results:
    print(f"{r['filename']}: {r['snippet'][:80]}...")

# Status
status = kd.get_status()
print(f"Documents: {status['documents']['total_documents']}")
print(f"Chunks: {status['documents']['total_chunks']}")
```

## Configuration

JSON-based config (`knowledgedigest.json`), searched in order:
1. `--config` CLI argument
2. Current working directory
3. Next to the database file
4. `%APPDATA%\KnowledgeDigest\` (Windows) / `~/.config/knowledgedigest/` (Linux)
5. Built-in defaults

All settings optional. Key settings:

| Key | Default | Description |
|-----|---------|-------------|
| `db_path` | `data/knowledge.db` | Database location |
| `indexed_directories` | `[]` | Directories to index |
| `chunk_size` | `350` | Words per chunk |
| `web_port` | `8787` | Web viewer port |
| `bach_enabled` | `false` | BACH system integration |

## BACH Integration (Optional)

KnowledgeDigest can optionally integrate with the [BACH](https://github.com/ellmos-ai/bach) agent system.
When `bach_enabled: true` and `bach_db_path` is set, BACH skills and wiki articles
are indexed alongside regular documents. This integration is lazy-loaded and has
zero impact when disabled.

## Requirements

- Python 3.10+
- PySide6 (for GUI)
- PyMuPDF (for PDF preview in GUI)
- No additional dependencies for CLI or Web Viewer

## Runtime Data

KnowledgeDigest stores local runtime data in `data/knowledge.db` and optional
`knowledgedigest.json` config files. These files are ignored by Git and should
not be committed.

## Changelog

**v0.4.0 (Latest)**
- **Gemini Flash Summarizer**: Extreme high-speed API option (`--flash`) added for LLM summarization (default model `gemini-2.0-flash`, configurable via `GEMINI_MODEL`; requires the optional `google-genai` package).
- **Agent Tollbooth (`zoll_station.py`)**: Smart routing for AI agents processing summarize queues.
- **Physical Trash (Web Viewer)**: Added a GUI button to physically quarantine file duplicates into a dynamic `_Papierkorb` folder. Delete API route fully integrated via `do_POST`.
- **Deduplication CLI**: Added `python -m KnowledgeDigest deduplicate` to identify cryptographic duplicates and auto-clean DB and filesystem.
- **Summary Reset**: Added a Web Viewer button to trash faulty summaries and place standard documents back into pending queue.

## See Also: WikiStub-Seed

[WikiStub-Seed](https://github.com/dev-bricks/WikiStub-Seed) is a related knowledge dataset from the dev-bricks org: 630 multilingual stub articles across 12 domains (DE/EN + es/ja/ru/zh), deployable as a self-contained wiki module. Useful as a structured seed corpus for KnowledgeDigest indexing workflows.

## See Also: ProFiler

Need PDF encryption, OCR, anonymization, or file versioning? Check out [ProFiler](https://github.com/file-bricks/ProFiler) -- a professional file management suite from the same author.

| | KnowledgeDigest | ProFiler |
|---|---|---|
| **Focus** | Knowledge search, chunking, LLM summaries | File management, PDF tools, OCR, privacy |
| **Search** | FTS5 with BM25 ranking, snippet highlighting | Multi-DB, type/size/date filters |
| **AI** | LLM summarization (Haiku), keyword extraction | -- |
| **PDF** | Read-only (text extraction) | Encrypt, decrypt, extract, redact, OCR |
| **Privacy** | -- | Anonymization, redaction, clipboard guard |
| **Interfaces** | Desktop GUI, Web Viewer, CLI, Python API | Desktop GUI, System Tray |
| **License** | MIT | AGPL v3 |

## License

## Author

Lukas Geiger ([github.com/lukisch](https://github.com/lukisch))

> **Kein Rechts-/Steuer-/Anlage-Rat.** Dieses Projekt ist ein technisches Hilfswerkzeug, keine professionelle Beratung. Bei finanziellen, rechtlichen oder steuerlichen Entscheidungen konsultieren Sie qualifizierte Fachleute.
>
> **No legal/tax/investment advice.** This project is a technical tool, not professional advice. Consult qualified professionals for financial, legal, or tax decisions.

---

## Haftung / Liability

Dieses Projekt ist eine **unentgeltliche Open-Source-Schenkung** im Sinne der §§ 516 ff. BGB. Die Haftung des Urhebers ist gemäß **§ 521 BGB** auf **Vorsatz und grobe Fahrlässigkeit** beschränkt. Ergänzend gelten die Haftungsausschlüsse der MIT-Lizenz.

Nutzung auf eigenes Risiko. Keine Wartungszusage, keine Verfügbarkeitsgarantie, keine Gewähr für Fehlerfreiheit oder Eignung für einen bestimmten Zweck.

This project is an unpaid open-source donation. Liability is limited to intent and gross negligence (§ 521 German Civil Code). Use at your own risk. No warranty, no maintenance guarantee, no fitness-for-purpose assumed.

knowledgedigest

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README