KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #developer-tools #ai-agent #claude-code #codex #typescript #ai-agents #deepseek-harness #mcp #cli #rust

documents-vector-search

★ 41 Open GitHub ↗

Project allows document indexing in a local vector database and then search (supports Jira, Confluence and local files, can be integrated via MCP)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Local Vector Search for Jira, Confluence & Files (with MCP support)

- [Overview](#overview)
- [Updates](#updates)
- [How it works](#how-it-works)
  - [Collection structure](#collection-structure)
  - [Indexers configuration](#indexers-configuration)
- [Setup](#setup)
  - [Env setup](#env-setup)
    - [Local setup](#local-setup)
    - [Docker setup](#docker-setup)
  - [Authentication](#authentication)
  - [Create collection for Confluence](#create-collection-for-confluence)
  - [Create collection for Jira](#create-collection-for-jira)
  - [Create collection for local files](#create-collection-for-local-files)
  - [Update collection](#update-collection)
  - [Search](#search)
    - [Filtering by metafields](#filtering-by-metafields)
  - [Fetch](#fetch)
  - [Set up MCP](#set-up-mcp)
    - [Unified MCP (recommended)](#unified-mcp-recommended)
    - [Simple MCP](#simple-mcp)
  - [Run unit tests](#run-unit-tests)
- [Good to know](#good-to-know)

## Overview

Index documents from Jira, Confluence, or local files into a local vector database and search them. All data stays on your machine.

**Key features:**
- Jira & Confluence (Server/Data Center and Cloud). Jira ticket = document, Confluence page = document
- Local files (.pdf, .pptx, .docx, etc.) via [Unstructured](https://github.com/Unstructured-IO/unstructured)
- **No data sent to third parties** (except when used as MCP with a remote AI agent)
- Hybrid search: vector search + BM25 keyword search, merged by Reciprocal Rank Fusion
- Incremental updates: no need to rebuild the full index each time
- Filter results by metafields (space, project, date, etc.)
- Ability to extend: add more data sources, search engines, embeddings, etc.

**Technologies:** [ChromaDB](https://github.com/chroma-core/chroma), [FAISS](https://github.com/facebookresearch/faiss), SQLite (BM25), [sentence-transformers](https://pypi.org/project/sentence-transformers/), [Unstructured](https://github.com/Unstructured-IO/unstructured), [LangChain](https://python.langchain.com/docs/introduction/)

More context: [Medium article](https://medium.com/@shnax0210/mcp-tool-for-vector-search-in-confluence-and-jira-6beeade658ba)

**Contacts:**
- Like it? Please star the repo
- Found a bug? [Open an issue](https://github.com/shnax0210/documents-vector-search/issues)
- Want to contribute? - feel free to do it via fork and sending a pull request
- Want to chat? [LinkedIn](https://www.linkedin.com/in/oleksii-shnepov-841447158/)

## Updates

- Check [UPDATES.md](UPDATES.md) for major updates. 
- Some minor updates can be not added the file, so they can be found only in git history.

## How it works

```mermaid
flowchart TD
    A[1. Create collection] -->|loads & indexes documents| B["Collection stored in ./data/collections/${name}"]
    B --> C{What next?}
    C --> D[2. Update collection]
    C --> E[3. Search via CLI]
    C --> F[4. Search via MCP]
    D -->|indexes only new/changed docs| B
    E -->|search| B
    F -->|search| B
```

1. **Create** a collection — load and index documents from Jira, Confluence, or local files. Stored in `./data/collections/{name}`
2. **Update** — re-index only new or changed documents (much faster than full creation)
3. **Search** — find documents by text query via CLI
4. **MCP** — expose search as a tool for AI agents

### Collection structure

```mermaid
graph TD
    A["./data/collections/${name}/"] --> B["documents/"]
    A --> C["indexes/"]
    A --> D["manifest.json"]
    B --- B1["Loaded and converted documents"]
    C --- C1["Vector and keyword index files"]
    D --- D1["Collection metadata: name, last update time, reader config, index list"]
```

See `./main/core/documents_collection_creator.py` for creation/update details and `./main/core/documents_collection_searcher.py` for search details.

### Indexers configuration

When you create a collection, you can specify a list of `indexers` like: `--indexers "indexer_ChromaDb__embeddings_sentence-transformers_slash_all-MiniLM-L6-v2", "indexer_SqlLiteBM25"`. The indexers define what vector/keyword databases and embedding models are used. Database and embedding model are separated by `__`. For example:
- `indexer_ChromaDb__embeddings_sentence-transformers_slash_all-MiniLM-L6-v2` means that `ChromaDb` is used as vector database and [`sentence-transformers/all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) is used as the embedding model. You can use any embedding model from next [list](https://huggingface.co/models?pipeline_tag=sentence-similarity&library=sentence-transformers&sort=trending), you only need to add prefix `embeddings_` and replace slash symbols with `_slash_`. For example, if you want to use ChromaDb with [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) embedder model, indexer name should be: `indexer_ChromaDb__embeddings_BAAI_slash_bge-m3`;
- `indexer_SqlLiteBM25` means that SqlLite BM25 is used as search engine.

You can define as many indexers as you want, their search results will be combined by Reciprocal Rank Fusion.

## Setup

### Env setup

Either `uv` or `docker` can be used to run the project.

#### Local setup

1. Clone the repository
2. Install [uv](https://docs.astral.sh/uv/)
3. Run `uv sync` in the project root

#### Docker setup

A `Dockerfile` is included for running the tool without installing Python or uv locally.

**Build the image:**
```bash
docker build -t documents-vector-search .
```

**Run any command** by passing it as arguments and mounting a local `data/` folder:
```bash
docker run --rm \
  -v $(pwd)/data:/app/data \
  -e CONF_TOKEN="${yourToken}" \
  documents-vector-search \
  uv run confluence_collection_create_cmd_adapter.py \
    --collection "confluence" \
    --url "${baseConfluenceUrl}" \
    --cql "${confluenceQuery}"
```

- Mount `-v $(pwd)/data:/app/data` so collections and caches are persisted on your host machine
- Pass credentials as `-e ENV_VAR=value` (see [Authentication](#authentication))
- The `data/` folder structure (`collections/`, `caches/`, `local_file_input/`) is created automatically inside the container

**Run the unified MCP HTTP server with Docker:**
```bash
docker run --rm \
  -p 8000:8000 \
  -v $(pwd)/data:/app/data \
  documents-vector-search \
  uv run collection_search_unified_mcp_adapter.py --http --port 8000
```

Then configure your MCP client to connect to `http://localhost:8000/mcp`.

### Authentication

Set environment variables before creating or updating Jira/Confluence collections (not needed for local files):

| Platform | Type | Environment Variables |
|---|---|---|
| Confluence Server/DC | Bearer token (recommended) | `CONF_TOKEN` |
| Confluence Server/DC | Login/Password | `CONF_LOGIN`, `CONF_PASSWORD` |
| Confluence Cloud | Email/API token | `ATLASSIAN_EMAIL`, `ATLASSIAN_TOKEN` ([get token](https://id.atlassian.com/manage/api-tokens)) |
| Jira Server/DC | Bearer token (recommended) | `JIRA_TOKEN` |
| Jira Server/DC | Login/Password | `JIRA_LOGIN`, `JIRA_PASSWORD` |
| Jira Cloud | Email/API token | `ATLASSIAN_EMAIL`, `ATLASSIAN_TOKEN` ([get token](https://id.atlassian.com/manage/api-tokens)) |

Cloud vs Server is auto-detected: URLs ending with `.atlassian.net` are treated as Cloud.

### Create collection for Confluence

```bash
uv run confluence_collection_create_cmd_adapter.py \
  --collection "confluence" \
  --url "${baseConfluenceUrl}" \
  --cql "${confluenceQuery}"
```

- `--collection` — name of the collection (used later for update/search). Data stored in `./data/collections/{name}`
- `--url` — Confluence base URL (e.g., `https://confluence.example.com` or `https://your-domain.atlassian.net`)
- `--cql` — Confluence query, e.g., `"(space = 'MySpace') AND (lastModified >= '2025-01-01')"`

### Create collection for Jira

```bash
uv run jira_collection_create_cmd_adapter.py \
  --collection "jira" \
  --url "${baseJiraUrl}" \
  --jql "${jiraQuery}"
```

- `--url` — Jira base URL (e.g., `https://jira.example.com` or `https://your-domain.atlassian.net`)
- `--jql` — Jira query, e.g., `"project = MyProject AND created >= -183d"`

### Create collection for local files

```bash
uv run files_collection_create_cmd_adapter.py --basePath "${pathToFolder}"
```

- Collection name defaults to the last folder name. Override with `--collection {name}`
- Unreadable files are skipped by default. Use `--failFast` to stop on first error
- Filter files with `--includePatterns "regex1" "regex2"` and `--excludePatterns "regex1" "regex2"`
- Uses [Unstructured](https://github.com/Unstructured-IO/unstructured) for parsing. Some formats may need [extra software](https://docs.unstructured.io/open-source/installation/full-installation#full-installation)

### Update collection

```bash
uv run collection_update_cmd_adapter.py --collection "${collectionName}"
```

### Search

```bash
uv run collection_search_cmd_adapter.py \
  --collection "${collectionName}" \
  --query "How to set up react project locally"
```

- `--includeMatchedChunksText` — include matched text chunks in results
- `--filter` — filter by metafields (see below)
- `--rrfK` — RRF constant for multi-index fusion (default: `60`)

#### Filtering by metafields

Works with ChromaDB and SQLite BM25 indexes.

**Syntax:**
```
field operator "value" and/or field operator "value"
```

Operators: `=`, `!=`, `>`, `>=`, `<`, `<=`. Use `and` / `or` to join conditions (mixing both is not supported).

**Confluence metafields:**

| Field | Description |
|---|---|
| `space` | Space key |
| `createdAt` | Page creation date |
| `createdBy` | Creator email (lowercase) |
| `lastModifiedAt` | Last update date |

**Examples:**
```bash
--filter 'space = "SPACE_KEY"'
--filter 'space = "SPACE_KEY" and lastModifiedAt > "2026-01-01"'
--filter '(space = "SPACE_KEY1" or space = "SPACE_KEY2") and lastModifiedAt > "2026-01-01"'
```

**Jira metafields:**

| Field | Description |
|---|---|
| `project` | Project key |
| `type` | Issue type (Bug, Task, Story, ...) |
| `status` | Status (Open, In Progress, Done, ...) |
| `priority` | Priority (High, Medium, Low, ...) |
| `epic` | Epic or parent issue key |
| `assignee` | Assignee email (lowercase) |
| `createdAt` | Issue creation date |
| `createdBy` | Creator email (lowercase) |
| `lastModifiedAt` | Last update date |

**Examples:**
```bash
--filter 'project = "PROJ"'
--filter 'project = "PROJ" and lastModifiedAt > "2026-01-01"'
--filter '(project = "PROJ1" or project = "PROJ2") and lastModifiedAt > "2026-01-01"'
```

**Files metafields:**

| Field | Description |
|---|---|
| `createdAt` | File creation date |
| `lastModifiedAt` | Last modified date |
| `folder1` | First subfolder of the file path |
| `folder2` | Second subfolder of the file path |
| `folderN` | Nth subfolder of the file path (only present if path has N or more subfolders) |

**Examples:**
```bash
--filter 'folder1 = "docs"'
--filter 'folder1 = "docs" and folder2 = "api"'
--filter '(folder2 = "api" or folder2 = "presentations") and lastModifiedAt > "2026-01-01"'
```

### Fetch

```bash
uv run collection_fetch_cmd_adapter.py \
  --collection "${collectionName}" \
  --id "${documentId}"
```

- `--id` — document ID to fetch (required)
- `--startLine` / `--endLine` — line range to return (default: 1–200)
- `--format` — output format: `json`, `json_with_indent` (default), or `toon`

### Set up MCP

There are two MCP server adapters:

| Adapter | Best for | Key differences |
|---|---|---|
| `collection_search_unified_mcp_adapter.py` | Modern AI models | All collections in one server. AI model chooses collection, filter and number of chunks. Supports stdio and HTTP transport. |
| `collection_search_mcp_stdio_adapter.py` | Simpler AI models or restricted setups | One collection per server. Collection, filter and other settings are hardcoded via CLI args. Stdio only. |

#### Unified MCP (recommended)

Add to your MCP config (e.g., `.vscode/mcp.json` for VS Code + GitHub Copilot):

```json
{
    "servers": {
        "dvs": {
            "type": "stdio",
            "command": "uv",
            "args": [
                "--directory", "${fullPathToRootProjectFolder}",
                "run", "collection_search_unified_mcp_adapter.py"
            ]
        }
    }
}
```

- All collections from `./data/collections/` are available automatically
- Use `--collections "name1" "name2"` to limit which collections are exposed
- Use `--rrfK {number}` to tune Reciprocal Rank Fusion behavior for multi-index search
- Use  `--defaultNumberOfChunks {number}` and `--maxNumberOfChunks {number}` to tune the number of text chunks returned by a single search

Or start as http server:

```bash
uv run collection_search_unified_mcp_adapter.py --http --port 8000
```

And setup mcp like:

```json
{
    "servers": {
        "dvs": {
            "type": "http",
            "url": "http://localhost:8000/mcp",
        }
    }
}
```

#### Simple MCP

```json
{
    "servers": {
        "search_${collectionName}": {
            "type": "stdio",
            "command": "uv",
            "args": [
                "--directory", "${fullPathToRootProjectFolder}",
                "run", "collection_search_mcp_stdio_adapter.py",
                "--collection", "${collectionName}"
            ]
        }
    }
}
```

- Replace `${collectionName}` and `${fullPathToRootProjectFolder}` with real values
- Use `--maxNumberOfChunks {number}` to control how many text chunks are returned (more = better search, but may exceed model context window)
- Use `--rrfK {number}` to tune Reciprocal Rank Fusion behavior for multi-index search
- Use `--filter` for metafield filtering ([details](#filtering-by-metafields))

**Prompt examples:**
- "Find info about AI use cases, search on Confluence, include all used links"
- "Find info about PDP carousel, search on Jira, include all used links"

### Run unit tests

If you develop the tool, you can run unit tests:

```
uv run pytest
```

## Good to know

- **Incremental updates** — only new/changed documents are re-indexed. Uses `lastModifiedDocumentTime` from `manifest.json` (5 mins for Jira and Confluence buffer to avoid missing concurrent updates);
- **Caching** — Jira/Confluence collection creation caches downloaded documents in `./data/caches/{hash}`. Same parameters = same cache. If you need fresh data, either run an update after creation, or delete the cache folder manually;
- there are more parameters in scripts, use "--help" to get more.

documents-vector-search

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README