FlexNeuART

KO | EN

gitlite — search

#ai #golang #android #deep-learning #macos #java #awesome #cli #kubernetes #vue #ios #mcp #openai #open-source

(★ 224)

Flexible classic and NeurAl Retrieval Toolkit

File Explorer

Download Latest Version (.zip)

- basic_data_preparation_and_processing.ipynb
- basic_indexing.ipynb
- cand_generator_nmslib.ipynb
- experimentation_testing_no_need_to_train_fusion.ipynb
- experimentation_train_fusion_models.ipynb
- experimentation_tuning_bm25_and_bm25_model1.ipynb
- installation.ipynb
- py_api_demo.ipynb
- README.md
- special_data_processing.ipynb
- train_model1.ipynb
- train_neural_model.ipynb
- - __init__.py
  - utils.py
- - __init__.py
  - utils.py
- - __init__.py
  - base.py
  - utils.py
- - __init__.py
  - utils.py
- - __init__.py
  - index_manner.ipynb
  - README.md
  - utils.py
- - __init__.py
  - json.py
  - pack.py
  - qrels.py
  - queries.py
  - runs.py
  - stopwords.py
  - train_data.py
  - utils.py
  - vocab.py
- - __init__.py
  - base.py
  - bert_tokenizer.py
  - concat.py
  - copy.py
  - html_parser.py
  - krovetz_stem_parser.py
  - pipeline.py
  - rename.py
  - spacy.py
- - __init__.py
  - base.py
  - base_bart.py
  - base_bert.py
  - base_bert_late_interaction.py
  - base_bert_split_max_chunk.py
  - base_bert_split_slide_window.py
  - base_t5.py
  - bert_aggreg_p.py
  - longformer.py
  - README.md
  - utils.py
  - vanilla_bert_standard.py
- - __init__.py
  - base.py
  - bm25py.py
  - classic.py
  - neural.py
- - __init__.py
  - cand_provider.py
  - fwd_index.py
  - utils.py
- - __init__.py
  - clean.py
  - formats.py
  - parse.py
  - utils.py
- __init__.py
- config.py
- eval.py
- utils.py
- version.py
- - .classpath
  - .DS_Store
  - .project
  - build.xml
  - LICENSE.txt
  - pom.xml
  - readme.txt
- - RankLib.jar
- pom.xml
- rebuild_ranklib.sh
- knn4qa.md
- LEGACY_INSTALL.md
- manner.ipynb
- MANNER_README.md
- MSMARCO_docs_2020-12-06_complete.ipynb
- MSMARCO_docs_2020-12-06_processed_data_and_precomp_model1.ipynb
- MSMARCO_README.md
- README.md
- - check_query_overlap.py
  - check_run.py
  - check_split_leak.py
  - check_split_queries.py
  - check_split_queries.sh
  - compare_eval_tools.py
  - compare_metrics.sh
  - embed_nmslib_sparse_export.py
  - README.md
- - stopwords.txt
- - add_stemmed_field.py
  - common_conv.sh
  - convert_tsv_queries_krovetz.py
  - create_answ_based_qrels.sh
  - filter_queries.py
  - merge_bsonl.py
  - split_queries.py
  - split_queries.sh
- - eval_output.py
  - gdeval.pl
  - get_cpu_cores.py
  - parse_exper_conf.py
  - run_experiments.sh
  - run_one_experiment.sh
  - warmup_indices.sh
- - export_nmslib_common.sh
  - export_nmslib_dense_sparse_fused.sh
  - export_nmslib_sparse.sh
- - export_cedr.sh
  - export_cedr_with_scores.sh
  - export_common.sh
  - gen_pseudo_labels.py
  - mix_train_data.py
- - nn_rank_server.py
  - sample_client.py
  - sample_server.py
- - gen_bm25_tune_json_desc.py
  - gen_model1_exper_json_desc.py
  - gen_rm3_exper_json_desc.py
- - create_tran.sh
  - export_bitext_plain.sh
  - filter_tran_table_and_voc.sh
  - run_common.sh
  - run_mgiza.sh
  - sample_and_filter_long_bitext.py
- - create_fwd_index.sh
  - create_lucene_index.sh
- - ranklib_test.sh
  - ranklib_train_coordasc.sh
  - ranklib_train_lmart.sh
- - RankLib.jar
- - extract_field_vals.py
  - filter_fields.py
  - make_hybrid_run.py
  - replace_valid.sh
  - sample_docs_based_on_qrels.py
  - sample_queries.py
  - sample_queries.sh
- - count_model_params.py
  - get_collect_pardata_stat.sh
  - get_exper_results.sh
  - print_field_tok_stat.py
  - print_model1_assoc.py
  - sample_collect_stat.ipynb
- - build_vocab.py
  - eval_model.py
  - eval_model_distr.py
  - extract_weights.py
  - train_model.py
  - train_model.sh
- common_proc.sh
- config.sh
- flexneuart_install_extra_bin.sh
- grep_file_for_val.py
- __init__.py
- README.md
- test_eval.py
- test_io.py
- test_loss.py
- test_parse.py
- test_sampler.py
- test_utils.py
.gitignore
build.sh
build_clean.sh
build_main.sh
CITATION.cff
flexneuart_install_extra.sh
install_extra_flexneuart_main.sh
LICENSE
LICENSE.RankLib
README.md
requirements.txt
setup.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

[![Pypi version](https://img.shields.io/pypi/v/flexneuart.svg)](http://pypi.python.org/pypi/flexneuart)
[![Downloads](https://pepy.tech/badge/flexneuart)](https://pepy.tech/project/flexneuart)
[![Downloads](https://pepy.tech/badge/flexneuart/month)](https://pepy.tech/project/flexneuart) [![Join the chat at https://gitter.im/oaqa/FlexNeuART](https://badges.gitter.im/oaqa/FlexNeuART.svg)](https://gitter.im/oaqa/FlexNeuART?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

## FlexNeuART (flex-noo-art)
Flexible classic and NeurAl Retrieval Toolkit, or shortly `FlexNeuART` (**intended pronunciation** flex-noo-art) 
is a substantially reworked [`knn4qa` package](legacy_docs/knn4qa.md).  The overview can be found in our EMNLP OSS workshop paper: 
[Flexible retrieval with NMSLIB and FlexNeuART, 2020. Leonid Boytsov, Eric Nyberg](https://arxiv.org/abs/2010.14848).

In Aug-Dec 2020, we used this framework to generate best traditional and/or neural runs 
in the [MSMARCO Document ranking task](https://microsoft.github.io/msmarco/#docranking).
In fact, our best traditional (non-neural) run slightly outperformed a couple of neural submissions.
Please, see our write-up for details: [Boytsov, Leonid. "Traditional IR rivals neural models on the MS MARCO Document Ranking Leaderboard." arXiv preprint arXiv:2012.08020 (2020)](https://arxiv.org/abs/2012.08020).

In 2021, after being outsripped by a number of participants, we again advanced to a good position with a help of newly implemented models for ranking long documents. Please, see our write-up for details: Boytsov, L., Lin, T., Gao, F., Zhao, Y., Huang, J., & Nyberg, E. (2022). [Understanding Performance of Long-Document Ranking Models through Comprehensive Evaluation and Leaderboarding](https://arxiv.org/abs/2207.01262). At the moment of writing (October 2022), we have competitive submissions on **both** MS MARCO leaderboards.

Code corresponding to Neural Model 1 is not included as this may be subject to a third party patent. 
This model (together with its non-contextualized variant) is described and evaluated in our ECIR 2021 paper:
[Boytsov, Leonid, and Zico Kolter. "Exploring Classic and Neural Lexical Translation Models for Information Retrieval: Interpretability, Effectiveness, and Efficiency Benefits." ECIR 2021](https://arxiv.org/abs/2102.06815).

In terms of pure effectiveness on long documents, other models (CEDR & PARADE) seem to be perform equally well (or somewhat better). They are available [in our codebase](flexneuart/models/README.md). We are not aware of the patents inhibiting the use of the traditional (non-neural) Model 1.

## Objectives

Develop & maintain a (relatively) light-weight modular middleware useful primarily for: 
* Research
* Education
* Evaluation & leaderboarding

## Main features

* Dense, sparse, or dense-sparse retrieval using Lucene and NMSLIB (dense embeddings can be created using any Sentence BERT model).
* Multi-field multi-level forward indices (+parent-child field relations) that can store 
  parsed and "raw" text input as well as sparse and dense vectors.
* Forward indices can be created in append-only mode, which requires much less RAM.
* Pluggable generic rankers (via a server)
* SOTA neural ([PARADE, BERT FirstP/MaxP/Sum, Longformer, COLBERT (re-ranking), dot-product Senence BERT models](flexneuart/models/README.md)) and non-neural models (multi-field BM25, IBM Model 1).
* Multi-GPU training **and** inference with out-of-the box support for ensembling
* Basic experimentation framework (+LETOR)
* Python API to use retrievers and rankers as well as to access indexed data.

## Documentation

* [Usage notebooks covering installation & most functionality (including experimentation and Python API demo)](demo/README.md)
* [Legacy notebooks for MS MARCO and Yahoo Answers](legacy_docs/README.md)
* [Former life (as a knn4qa package), including acknowledgements and publications](legacy_docs/knn4qa.md)

We support [a number of neural BERT-based ranking models](flexneuart/models/README.md) as well as strong traditional
ranking models including IBM Model 1 (description of non-neural rankers to follow).

The framework supports data in generic JSONL format. We provide conversion (and in some cases download) scripts for the following collections:
* [Configurable dataset processing of standard datasets](/scripts/data_convert/ir_datasets/README.md) provided by [ir-datasets](https://ir-datasets.com/).
* MS MARCO [v1](scripts/data_convert/msmarco) and [v2](scripts/data_convert/msmarco_v2) (documents and passages)
* [Wikipedia DPR](scripts/data_convert/wikipedia_dpr/README.md) (Natural Questions, Trivia QA, SQuAD)
* [Yahoo Answers](/scripts/data_convert/yahoo_answers)
* [Cranfield](scripts/data_convert/cranfield/README.md) (a small toy collection)

## Acknowledgements

For neural network training FlexNeuART incorporates
a substantially re-worked variant of CEDR ([MacAvaney et al' 2019](https://github.com/Georgetown-IR-Lab/cedr)).

FlexNeuART

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

FlexNeuART

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README