MTEB rankings: choose embedding models for your data

Radu Gheorghe explains how to choose an embedding model for search beyond its MTEB leaderboard position. He recommends starting with a benchmark close to the intended task, then checking model size, language support and context requirements. For teams running search on their own servers, the talk connects model selection with hardware cost and query latency.

His tests favor int8 model quantization on CPU and FP16 on GPU. He separately compares float32, bfloat16 and binary output vectors, noting that binary vectors can sacrifice relevance quality. Matryoshka embeddings allow shorter vectors; in one example, 512 dimensions score slightly better than the full vector on a particular test. These results are reasons to test on your own data rather than assume a universal configuration.

The fine-tuning walkthrough uses Vespa Embed, an open source interface over Sentence Transformers. It covers query-document pairs, triplets with negative examples and validation splits. Gheorghe suggests starting with defaults and using an LLM judge with explicit scoring examples when labeled data is scarce.

The final section explains Vespa's cheap first-phase ranking and more expensive reranking of top results. Gheorghe also warns that newly retrieved documents need relevance judgments: treating unrated results as irrelevant can hide improvements in NDCG. His product-title training experiment failed, reinforcing his advice to train on realistic queries.