Text Embeddings Inference

Text Embeddings Inference is a self-hosted server for developers building semantic search and retrieval applications. It turns text into embeddings through a REST API and can run offline once model weights are downloaded.

• Embedding models: Serve models including Nomic, E5, Alibaba GTE, Qwen3 and google/embeddinggemma-300m for text similarity and retrieval.
• Reranking: Score passages against a query with models such as BAAI/bge-reranker-base and Alibaba-NLP/gte-multilingual-reranker-base.
• Classification and sparse embeddings: Detect emotions with sequence classification models or produce sparse embeddings through SPLADE pooling for BERT and DistilBERT.
• Request batching: Group requests by token count to use compute resources efficiently when serving multiple clients.
• Service monitoring: Track requests with OpenTelemetry tracing and expose Prometheus metrics for the running server.

TEI is open source under Apache 2.0 and runs locally on CPUs or in Docker, with NVIDIA GPU support and ROCm support for AMD Instinct MI200 and MI300 GPUs. ARM64 CPU containers support Apple Silicon, Ampere and Graviton hardware. On Apple Silicon, Docker inference uses CPU cores rather than Metal or MPS. Access to private or gated Hugging Face models requires a token.