CTranslate2
A local AI inference library for C++ and Python that runs Transformer models on CPUs and GPUs, with quantization to reduce memory use.
CTranslate2 is an inference library for developers who want to run Transformer models on their own hardware or servers. It focuses on reducing execution time and memory use for applications built in C++ or Python.
- Translation and generation: Run translation models such as NLLB and M2M-100, or generate text with compatible Llama, Mistral, Gemma and Qwen2 models.
- Speech and text encoding: Use Whisper for speech recognition and BERT, DistilBERT or XLM-RoBERTa for text encoding.
- Smaller models: Quantization and reduced precision lower storage and memory requirements, with support for INT8 and AWQ INT4 weights.
- Parallel execution: Process batches asynchronously across CPU cores or multiple GPUs, and split large models across GPUs with tensor parallelism.
- Interactive decoding: Complete partial text sequences and request alternatives at a selected position.
Compatible models must use CTranslate2's optimized format. Converters support Transformers, OpenNMT, Fairseq, Marian and OPUS-MT. CPU execution supports x86-64 and ARM64 processors, including Apple Accelerate as a backend. GPU execution supports NVIDIA CUDA; separate Python packages support AMD ROCm GPUs.
The library is open source under the MIT license. The separate ctranslate2-web-server project adds an OpenAI-compatible REST API for applications that need a server interface.
Logotyper
