verl

verl is a Python library for researchers and engineers training large language models on their own hardware. It implements HybridFlow for reinforcement learning after initial training, with support for supervised fine-tuning too.

• Reward-based training: Use methods such as PPO and GRPO with rewards from another model or functions that check answers, including math solutions and code.
• Tool use and multimodal models: Train multi-turn interactions with tool calling and vision-language models such as Qwen2.5-vl and Kimi-VL.
• Backend choice: Combine FSDP, FSDP2 or Megatron-LM training with response generation through vLLM, SGLang or Hugging Face Transformers.
• Distributed training: Place models across groups of GPUs and use multi-GPU LoRA to reduce memory needs; expert parallelism supports large models across clusters.

Compatible model families include Qwen, Llama3.1, Gemma2 and DeepSeek-LLM, with support for Hugging Face Transformers and Modelscope Hub. Experiments can report to wandb, swanlab, MLflow or TensorBoard.

verl is open source under Apache 2.0 and supports NVIDIA, AMD and Ascend hardware. On AMD ROCm GPUs, it supports FSDP, FSDP2 and Megatron-LM training with vLLM as the validated generation engine.