KTransformers

An open-source local LLM framework runs and fine-tunes large MoE models across CPUs and GPUs, with SGLang and LlamaFactory integrations.

KTransformers is a research framework for developers running or fine-tuning large language models on their own hardware. It focuses on mixture-of-experts (MoE) models, using CPU resources alongside GPUs to work within limited GPU memory.

Hybrid inference: Place frequently used experts on the GPU and less-used experts on the CPU to serve large MoE models.
Model serving: Connect inference to SGLang or other frameworks through a Python API.
Quantized inference: Use INT4 or INT8 weights on the CPU and GPTQ on the GPU, with CPU acceleration through Intel AMX, AVX512 or AVX2.
Model fine-tuning: Use LlamaFactory integration for LoRA and full training, including DeepSeek-V3 and DeepSeek-R1 workloads.
Kimi adapters: Fine-tune Kimi K2.5 and K2.6 with LoRA and native RAWINT4 experts; end-to-end validation uses K2.5, while K2.6 follows the same model architecture path.

Hardware requirements depend on the model and workload. Fine-tuning examples use one RTX 4090 for Qwen3-30B-A3B and four RTX 4090 GPUs for DeepSeek-V3 or DeepSeek-R1. CPU and GPU resources both contribute to these workloads. The Python project is open source under Apache 2.0.