kvpress

kvpress is a Python library for developers and researchers reducing the memory needed for long-context LLM inference. It compresses the key-value cache that grows as a model processes longer inputs.

• Compression methods: Compare approaches such as SnapKV, ExpectedAttention, StreamingLLM and PyramidKV, or develop new methods within the same framework.
• Context compression: Compress context separately from question tokens to evaluate different questions against the same input.
• Cache quantization: Combine compression with Hugging Face Transformers' quantized cache support.
• Evaluation: Test answer accuracy on long-context benchmarks, including RULER, and measure memory use and execution time.

The Apache 2.0 library runs locally with PyTorch and Hugging Face Transformers. It supports multi-GPU inference through Accelerate and model families including Llama, Mistral, Qwen and Gemma, though individual compression methods depend on model architecture.

Memory savings vary with the cache size and compression ratio. Some methods add computation or temporary memory overhead during context processing.