portable-hnsw - GitLite

KO | EN

gitlite — search

#hacktoberfest #react #golang #go #android #machine-learning #linux #docker #java #cli #kubernetes #awesome-list #pytorch #openai

portable-hnsw

(★ 122)

What if an HNSW index was just a file, and you could serve it from a CDN, and search it directly in the browser?

File Explorer

Download Latest Version (.zip)

- docs.parquet
- edges.parquet
- nodes.parquet
- docs.parquet
- edges.parquet
- frankenstein.txt
- nodes.parquet
- docs.parquet
- edges.parquet
- nodes.parquet
build_index.py
index.html
jason.css
LICENSE
poetry.lock
pyproject.toml
README.md

# Installation Guide

1. Get the code

git clone https://github.com/jasonjmcghee/portable-hnsw

Downloads the entire project code from GitHub to your computer.

cd portable-hnsw

Moves into the project folder you just downloaded.

2. Python

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

poetry install

Installs the libraries listed in pyproject.toml.

poetry run python build_index.py <path to text file> [output folder]

Runs the Python script (or module).

✅ If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1><img src="https://github.com/jasonjmcghee/portable-hnsw/assets/1522149/8fab793b-e0a2-4fc9-b813-952c95822705" height="24"> Portable HNSW </h1>

<a href="https://www.loom.com/share/a5cb417115684a3e82a5fd3f20266489"><img style="max-width:300px;" src="https://cdn.loom.com/sessions/thumbnails/a5cb417115684a3e82a5fd3f20266489-with-play.gif"></a>

To build your own index:

```bash
poetry install
poetry run python build_index.py <path to text file> [output folder]
```

Or you can jump into the code and do more complex use cases.

Then throw it in a GitHub repo and enable GitHub Pages. You can add / edit the index.html or test it by pasting the link to the folder in the "path" input [here](https://jasonjmcghee.github.io/portable-hnsw/)

Note: `rangehttpserver` works well as a simple server to support range requests for locally testing duckdb parquet + large indices.

---------------

So what's going on here?

Yeah - fair question.

So I had this idea.

What if an HNSW index ([hierarchical navigable small world graphs](https://arxiv.org/abs/1603.09320) - _a good way to enable searching for stuff by their underlying meaning_) was just a file, and you could serve it from a CDN, and search it directly in the browser?

And what if you didn't need to load the entire thing in memory, so you could search a massive index
without being RAM rich?

That would be cool.

A vector store without a server...

So yeah. Here's a proof of concept.

---

There's a Python file called `build_index.py` that builds an index using a custom hnsw algorithm that 
can be serialized to a couple of parquet files.

_There are very likely bugs and performance problems. But it's within an order of magnitude or two
of `hnswlib` which was fast enough that my development cycle wasn't impacted by repeatedly re-indexing
the same files while building the search and front-end bits. I welcome pull requests to fix the problems
and make it halfway reasonable._

Then I wrote a webpage that uses `transformers.js`, `duckdb` and some SQL to read the parquet files and 
search it (similar to HNSW approx nearest neighbor search) and then retrieve the associated text.

A big part of the original idea was how this could scale to massive indices.

So, I also tested using parquet range requests and only retrieving what we need from the parquet file,
which worked! But since the index is only like 100MB, and each range request added overhead, loading
it all into memory was about twice as fast. But, it means you could have a 1TB index and it would
(theoretically) still work, which is pretty crazy.

You can try this yourself by swapping out the `nodes.parquet` bits in the SQL for `read_parquet('${path}/nodes.parquet')`. And the same with edges. DuckDB takes care of the rest.

---

Anyway, would love feedback and welcome contributions.

It was a fun project!