MSVBASE - GitLite

KO | EN

gitlite — search

#hacktoberfest #ai #llm #go #android #machine-learning #rust #linux #nodejs #windows #awesome-list #vue #openai #open-source

MSVBASE

(★ 103)

MSVBASE is a system that efficiently supports complex queries of both approximate similarity search and relational operators. It integrates high-dimensional vector indices into PostgreSQL, a relational database to facilitate complex approximate similarity queries.

File Explorer

Download Latest Version (.zip)

- - codeql.yml
  - docker-image.yml
- - convert_bin.py
  - cook_data.sh
  - data_download.sh
  - data_gen.py
  - data_gen_spann.py
  - data_prepare.sh
  - load_data.sh
  - spann_embedding_data.py
  - spann_embedding_query.py
  - spann_process.sh
  - tag_data_gen.py
  - tag_data_gen_spann.py
- - create_vbase_docker.sh
- - latency.py
  - latency_q8.py
  - recall.py
  - recall_q8.py
- - run.sh
  - run_table_4.sh
  - run_table_5.sh
  - run_table_6.sh
- hnsw.patch
- new_pg.patch
- Postgres.patch
- spann.patch
- - config_pgbuild_dev.sh
  - docker-entrypoint.sh
  - pg_start.sh
  - pg_stop.sh
- dockerbuild.sh
- dockerrm.sh
- dockerrun.sh
- dockerstop.sh
- patch.sh
- vectordb.sql
- hnswindex.cpp
- hnswindex.hpp
- hnswindex_builder.cpp
- hnswindex_builder.hpp
- hnswindex_scan.cpp
- hnswindex_scan.hpp
- index.cpp
- index.hpp
- index_builder.cpp
- index_builder.hpp
- index_scan.cpp
- index_scan.hpp
- lib.cpp
- model_mng.cpp
- model_mng.hpp
- multicol_topk.cpp
- operator.cpp
- operator.hpp
- pase_hnswindex.cpp
- pase_hnswindex.hpp
- pase_hnswindex_builder.cpp
- pase_hnswindex_builder.hpp
- pase_hnswindex_scan.cpp
- pase_hnswindex_scan.hpp
- spannindex.cpp
- spannindex.hpp
- spannindex_builder.hpp
- spannindex_scan.cpp
- spannindex_scan.hpp
- topk.cpp
- util.cpp
- util.hpp
- CMakeLists.txt
- hnsw
- Postgres
- SPTAG
.gitmodules
azure-pipelines.yml
CMakeLists.txt
CODE_OF_CONDUCT.md
Dockerfile
LICENSE
README.md
SECURITY.md
SUPPORT.md
vectordb.control.in

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# MSVBASE
## What's NEW
* New Research Paper [SPFresh: Incremental In-Place Update for Billion-Scale Vector Search](https://dl.acm.org/doi/10.1145/3600006.3613166) - _published in SOSP 2023_
* New Research Paper [VBASE: Unifying Online Vector Similarity Search and Relational Queries via Relaxed Monotonicity](https://www.usenix.org/system/files/osdi23-zhang-qianxi_1.pdf) - _published in OSDI 2023_
## What's MSVBASE
MSVBASE is a new system capable of efficiently serving complex online queries that involve both approximate similarity search and relational operators on scalar and vector data-sets. MSVBASE builds a unified query execution engine to support a wide range of queries both on scalar and vector data, and shows superior query performance and accuracy.
It integrates high-dimensional vector indices into PostgreSQL with minimal code modifications.

## **Quickstart**
### **Clone and Patch**
```
git clone https://github.com/microsoft/MSVBASE.git
cd MSVBASE
git submodule update --init --recursive
./scripts/patch.sh
```
### **Build**
```
./scripts/dockerbuild.sh
```

### **Run**
```
./scripts/dockerrun.sh
```

## **Features and SQL syntax**
It is compatible with PostgreSQL syntax and protocol, supporting vector distance calculations for L2 and Inner product. It also supports [hnsw](https://github.com/nmslib/hnswlib) and [sptag](https://github.com/microsoft/SPTAG/) indices. Soon, we are going to integrate spann and more indices.
### **Command Line**
```
docker exec -it --privileged --user=root vbase_open_source bash
psql -U vectordb
```

### **SQL syntax**
It preserves all the features of PostgreSQL while extending query support on vector data.
* Currently, 'float array' is used to store vectors.
```
create table t_table(id int, price int, vector_1 float8[10], vector_2 float8[10]);
```
* Insert or Import data.
```
insert into t_table values(1, 10, '{1,2,3,4,5,6,7,8,9,0}', '{5,6,7,1,2,3,4,8,9,1}');
copy t_table from 'your_data_path.tsv' DELIMITER E'\t' csv quote e'\x01';
```
* When creating a vector index, it is necessary to specify the algorithm and the distance calculation method to be used.
```
create index vector_index_1 on t_table using hnsw(vector_1) with(dimension=10,distmethod=l2_distance);
create index vector_index_2 on t_table using sptag(vector_2 vector_inner_product_ops) with(distmethod=inner_product);
```
* **TopK**. When calculating distances, the '<->' operator represents the L2 distance, while '<*>' represents the inner product distance.
```
select id from t_table order by vector_1 <-> '{5,9,8,6,2,1,1,0,4,3}' limit 10;
select id from t_table order by vector_2 <*> '{5,9,8,6,2,1,1,0,4,3}' limit 5;
```
* **TopK + Filter**.
```
select id from t_table where price > 15 order by vector_1 <-> '{5,9,8,6,2,1,1,0,4,3}' limit 10;
select id from t_table where price > 15 order by vector_2 <*> '{5,9,8,6,2,1,1,0,4,3}' limit 5;
```
* **Distance Range Filter**. It also supports distance threshold-based filtering queries. The query will retrieve vector data that is within the distance threshold.
In the query, the '<<->>' operator represents the L2 distance, while '<<*>>' represents the inner product distance.
The first element of the array represents the distance threshold.
```
select id from t_table where price > 15 and vector_1 <<->> '{30,5,9,8,6,2,1,1,0,4,3}';
```
* **Multi-vector Column Query**.
```
select id from t_table
order by approximate_sum('0.5 * vector_1<->{5,9,8,6,2,1,1,0,4,3} + vector_2<*>{5,9,8,6,2,1,1,0,4,3}' ) limit 5;
```
* **Join**. Join on vector similarity with threshold
```
select t_table.id as tid, d_table.id as did
from t_table join d_table
on t_table.vector_2 <<*>> array_cat(ARRAY[cast(10 as float8)], d_table.vector_2);
```

* Example
```
create database test;
\c test;
create extension vectordb;
create table t_table(id int, price int, vector_1 float8[10]);
insert into t_table values(1, 10, '{1,2,3,4,5,6,7,8,9,0}');
insert into t_table values(2, 20, '{5,6,7,1,2,3,4,8,9,1}');
insert into t_table values(3, 30, '{9,8,7,6,5,4,3,2,1,0}');
create index t4_index on t_table using hnsw(vector_1) with(dimension=10,distmethod=l2_distance);
set enable_seqscan=false;
select id from t_table where price > 15 order by vector_1 <-> '{5,9,8,6,2,1,1,0,4,3}' limit 1;
insert into t_table values(4, 40, '{19,18,17,16,15,14,13,12,11,10}');
delete from t_table where id = 2;
```

## Contributing

This project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.

## Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft 
trademarks or logos is subject to and must follow 
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.