Pyserini

Pyserini is an open-source Python toolkit for researchers and developers comparing document search methods or building the retrieval stage of a ranking or RAG system. It runs as a Python package on your own hardware and lets you search existing benchmark indexes or build indexes from your own documents. Its license is Apache 2.0.

You can compare keyword search with methods that use learned representations of text. BM25, uniCOIL and SPLADE use Lucene through Anserini; dense models such as DPR, Contriever and BGE work with Lucene or Faiss. Hybrid retrieval combines sparse and dense results. This gives experiments a common toolkit across several retrieval approaches, including the initial search that supplies candidates for later ranking.

Reproducibility is a central part of the package. It includes prebuilt indexes, test queries, relevance judgments and evaluation scripts, so researchers can compare results against documented experiments without assembling every benchmark component themselves. Supported collections include MS MARCO and BEIR, alongside multilingual benchmarks such as Mr.TyDi and MIRACL.

For deeper search research, its APIs expose index statistics, term vectors and document scores, along with custom Lucene queries and access to retrieved document text. Optional multimodal support includes image search. A REST API and an MCP server make retrieval available to other applications and AI tools.