Candle

Candle is a Rust machine learning framework for developers embedding local AI in applications or deploying models on their own servers. It supports inference and training, including small CPU deployments for serverless applications.

• Language models: Run LLaMA, Gemma, Mistral, Mixtral and Qwen3 MoE, plus StarCoder and StarCoder2 for code generation; supported quantized models use llama.cpp quantization types.
• Speech processing: Use Whisper for multilingual transcription, MetaVoice or Parler-TTS for speech synthesis, and EnCodec for audio compression.
• Images and vision: Generate images with Stable Diffusion, detect objects with YOLO, segment images with Segment Anything, or extract captions and text with BLIP and TrOCR.
• Model development: Train models through a tensor API familiar to PyTorch developers and extend operations with custom kernels, including FlashAttention.

Candle runs on CPUs, NVIDIA GPUs through CUDA, and browsers through WebAssembly. Its browser demos perform inference entirely in the browser. CPU acceleration can optionally use MKL on x86 or Apple's Accelerate on Macs; NCCL supports work across multiple GPUs.

The framework is open source under Apache 2.0. It loads model weights from safetensors, npz, ggml and PyTorch files, and includes ONNX model evaluation.