KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #codex #typescript #developer-tools #ai-agents #deepseek-harness #claude-code #ai-agent #cli #react #mcp

realtime-codebase-indexing

★ 78 Open GitHub ↗

build codebase index with tree-sitter. works with large codebases, and can be updated in near real-time with incremental processing - only reprocess what's changed.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">Build Real-Time Codebase Indexing</h1>

A live, syntax-aware vector index over your repo — in ~100 lines of plain async Python. 
 Point it at a codebase, search it in natural language, and it re-embeds only what changes as you edit.

Star us&nbsp;❤️&nbsp;→&nbsp;<a href="https://github.com/cocoindex-io/cocoindex" title="Star CocoIndex on GitHub"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cocoindex.io/blobs/github/homepage/star-btn-small-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://cocoindex.io/blobs/github/homepage/star-btn-small-light.svg"><img src="https://cocoindex.io/blobs/github/homepage/star-btn-small-light.svg" alt="Star CocoIndex on GitHub" height="36" align="absmiddle"/></picture></a> &nbsp;·&nbsp;
 <a href="https://cocoindex.io/docs/examples/index-codebase/" title="Read the CocoIndex documentation"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cocoindex.io/blobs/github/homepage/docs-inline-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://cocoindex.io/blobs/github/homepage/docs-inline-light.svg"><img src="https://cocoindex.io/blobs/github/homepage/docs-inline-light.svg" alt="CocoIndex documentation" height="36" align="absmiddle"/></picture></a> &nbsp;·&nbsp;
 <a href="https://discord.com/invite/zpA9S2DR7s" title="Join the CocoIndex Discord"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cocoindex.io/blobs/github/homepage/discord-inline-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://cocoindex.io/blobs/github/homepage/discord-inline-light.svg"><img src="https://cocoindex.io/blobs/github/homepage/discord-inline-light.svg" alt="Join the CocoIndex Discord" height="36" align="absmiddle"/></picture></a>

[![GitHub](https://img.shields.io/github/stars/cocoindex-io/cocoindex?style=flat-square&label=stars&color=FB6A76)](https://github.com/cocoindex-io/cocoindex)
[![PyPI version](https://img.shields.io/pypi/v/cocoindex?style=flat-square&label=pypi&color=E59A63)](https://pypi.org/project/cocoindex/)
[![Discord](https://img.shields.io/discord/1314801574169673738?style=flat-square&logo=discord&logoColor=white&label=discord&color=5865F2)](https://discord.com/invite/zpA9S2DR7s)
[![License](https://img.shields.io/badge/license-Apache--2.0-5B5BD6?style=flat-square)](https://opensource.org/licenses/Apache-2.0)

</div>

📕 <a href="https://cocoindex.io/docs/examples/index-codebase/">Documentation</a> &nbsp;·&nbsp; 📘 <a href="https://cocoindex.io/blogs/index-codebase-v1/">Step-by-step tutorial</a> &nbsp;·&nbsp; 🎬 <a href="https://youtu.be/G3WstvhHO24?si=Bnxu67Ax5Lv8b-J2">Watch on YouTube</a>

Build a codebase index that's **always up to date**. CocoIndex has built-in, native Tree-sitter chunking, so it splits along real code structure — functions, classes, blocks — embeds each chunk, and keeps the index fresh with **incremental processing**: a one-line edit re-embeds one chunk, not the repo. You declare the transformation in native Python and your own types — `target_state = transformation(source_state)` — and a Rust engine does the incremental processing, change tracking, and managed targets underneath.

## Use cases

A wide range of applications can be built on an effective codebase index that is always up to date:

- **Semantic code context** for AI coding agents like Claude, Codex, and Gemini CLI.
- **MCP** for code editors such as Cursor, Windsurf, and VSCode.
- **Context-aware code search** — semantic code search, natural-language code retrieval.
- **Code-review agents** — automated analysis, quality checks, pull-request summarization.
- **Refactoring & migration** — large-scale, automated code changes.
- **SRE workflows** — index infra-as-code, deploy scripts, and configs for rapid root-cause analysis and change-impact assessment.
- **Living design docs** — generate documentation from code and keep it current.

## Why CocoIndex for codebase indexing

- **Syntax-aware chunking, built in.** Tree-sitter splits along real code structure, so retrieval returns whole units, not fragments cut mid-statement. Every major language; unknown types fall back to plain text.
- **Incremental by default.** `@coco.fn(memo=True)` skips unchanged files and reuses embeddings for unchanged chunks; the target upserts only the rows that moved and deletes orphans. Edit one function → one chunk is re-embedded.
- **Live updates.** `live=True` + `cocoindex update -L` keeps watching the filesystem and applies changes with low latency — always-fresh context for an agent.
- **Plain Python, your stack.** Swap the embedding model (12k+ on Hugging Face), the chunking, or the [vector store](https://cocoindex.io/docs/). No DSL.
- **Consistent index & query.** The same embedder is shared by the indexing and query paths, so what you index is what you search.

## How it works

Walk a repo → detect language → split along the **syntax tree** with Tree-sitter → embed each chunk → upsert into Postgres (pgvector). With `live=True`, the source keeps watching and the index stays fresh as you code. The whole indexing path is the snippet below — read it top-to-bottom in [`main.py`](main.py):

```python
@coco.fn(memo=True)
async def process_file(file: FileLike, table: postgres.TableTarget[CodeEmbedding]) -> None:
    text = await file.read_text()
    language = detect_code_language(filename=str(file.file_path.path.name))
    chunks = _splitter.split(text, chunk_size=1000, min_chunk_size=300,
                             chunk_overlap=300, language=language)   # Tree-sitter, syntax-aware
    id_gen = IdGenerator()
    await coco.map(process_chunk, chunks, file.file_path.path, id_gen, table)

@coco.fn
async def process_chunk(chunk, filename, id_gen, table) -> None:
    embedding = await coco.use_context(EMBEDDER).embed(chunk.text)
    table.declare_row(row=CodeEmbedding(
        id=await id_gen.next_id(chunk.text), filename=str(filename), code=chunk.text,
        embedding=embedding, start_line=chunk.start.line, end_line=chunk.end.line,
    ))

@coco.fn
async def app_main(sourcedir: pathlib.Path) -> None:
    table = await postgres.mount_table_target(PG_DB, table_name=TABLE_NAME, ...)
    table.declare_vector_index(column="embedding")
    files = localfs.walk_dir(sourcedir, recursive=True,
                             path_matcher=PatternFilePathMatcher(included_patterns=["**/*.py", ...]),
                             live=True)
    await coco.mount_each(process_file, files.items(), table)
```

📘 <a href="https://cocoindex.io/blogs/index-codebase-v1/">Full tutorial →</a> 
 Step-by-step walkthrough of the data model, the lifespan, chunking, embedding, the App, and exactly what happens on each kind of change.

## Run it

**1. Postgres + pgvector.** [Install Postgres](https://cocoindex.io/docs/getting_started/installation#-install-postgres) with the `pgvector` extension if you don't have one, then point the example at it:

```sh
export POSTGRES_URL="postgres://cocoindex:cocoindex@localhost/cocoindex"
```

CocoIndex keeps its own incremental-processing state in a local file (`COCOINDEX_DB`, default `./cocoindex.db`; see [`.env`](.env)). By default the flow indexes **this repository** — set `COCOINDEX_SOURCE_PATH` to index any other codebase.

**2. Install dependencies:**

```sh
pip install -e .
```

**3. Build / update the index** (writes rows into Postgres) — pick one:

```sh
cocoindex update main       # catch-up: scan, sync changes, exit
cocoindex update -L main    # live: catch up, then keep watching for edits
```

**4. Query it** — semantic search from the terminal:

```sh
python main.py "your query"   # one-shot
python main.py                # interactive loop
```

Each result carries `start_line`/`end_line`, so hits point straight at the lines that matched. Query uses pgvector's `<=>` cosine distance, turned into a similarity score, and reuses the **same embedder** as the indexing path.

## Incremental & real-time updates

Edit a file and re-run (or leave `cocoindex update -L` running): unchanged chunks are **reused** with no re-embedding, a removed chunk's row is **deleted**, and a new chunk is **embedded and inserted** — only the delta moves. That's what keeps the index cheap to maintain and always fresh for an agent.

## Want it production-ready, not DIY?

[**CocoIndex Code**](https://github.com/cocoindex-io/cocoindex-code) is this exact pipeline — AST-aware chunking, incremental re-index, local embeddings — hardened and packaged as a CLI and an MCP server you can plug straight into a coding or code-review agent.

```sh
npx skills add cocoindex-io/cocoindex-code      # Claude Code skill, then /ccc
claude mcp add cocoindex-code -- ccc mcp        # MCP: Codex, OpenCode, Cursor, any client
ccc index && ccc search "where we embed chunks" # CLI
```

---

If this made your agents smarter, <a href="https://github.com/cocoindex-io/cocoindex">give CocoIndex a star ⭐</a> — it helps a lot. 
 <a href="https://cocoindex.io/docs/examples/index-codebase/">Documentation</a> · <a href="https://cocoindex.io/blogs/index-codebase-v1/">Walkthrough</a> · <a href="https://discord.com/invite/zpA9S2DR7s">Discord</a> · <a href="https://github.com/cocoindex-io/cocoindex/tree/main/examples">See all examples →</a>

realtime-codebase-indexing

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README