RankLLM

RankLLM is a Python toolkit for search researchers and developers who want to compare how AI models reorder retrieved documents by relevance. Its main focus is listwise reranking, where a model considers several candidates together. It supports local inference and optional cloud APIs, and it's open source under the Apache 2.0 license.

Local models run through Hugging Face and PyTorch, with vLLM support specifically for RankZephyr and RankVicuna. The model selection also includes LiT5, MonoELECTRA, MonoT5 for scoring documents individually, and DuoT5 for comparing pairs. MonoT5's Med variants target biomedical retrieval. Hosted reranking uses OpenAI, OpenRouter, or Gemini, so those inference requests go to an external service rather than a local model.

The toolkit covers retrieval, reranking, and evaluation in the same research workflow. Pyserini integration supports retrieval methods such as BM25 and SPLADE, while evaluation tools measure ranking quality against benchmark judgments. Saved results and model invocation histories let researchers inspect responses, count errors, and reproduce experiments on MS MARCO and TREC Deep Learning datasets.

Custom prompt templates and support for adding models let researchers test their own reranking approaches. First-token scoring offers an alternative to generating a full ranking response to reduce inference work. RankLLM also includes fine-tuning scripts, HTTP and MCP server interfaces, and integrations with LlamaIndex, LangChain, and the rerankers library.