Tools tagged with "Quantization"

llmfit helps people choosing models for their own computers compare what will fit in memory and how fast it might run. It checks your hardware before recommending models, so you can assess candidates before downloading them.

SGLang is a self-hosted inference framework for teams serving AI models to applications. It supports production workloads on a single GPU or across distributed clusters, including AI agents and reinforcement learning rollouts.

vLLM serves language models on hardware you control. It's for developers and teams building model APIs that need to handle concurrent requests while using memory and compute efficiently.

LightLLM is a self-hosted LLM inference and serving framework for developers and teams running models on their own GPU servers. Its focus is serving requests efficiently: it schedules work around predicted memory use and handles text encoding, model inference and output decoding asynchronously to improve GPU utilization.

text-generation-webui, also called TextGen, runs language models on your own hardware for chat, writing and application integrations. You can use it as a desktop app or a self-hosted browser interface.

Optimum Quanto is an open-source PyTorch library for developers who want to reduce the memory needed to run models on their own hardware. It handles language models and image-generation models, with integrations for Hugging Face Transformers and Diffusers. The project is in maintenance mode, accepting minor fixes and documentation work rather than major features.

ExecuTorch is an on-device AI runtime for developers adding PyTorch models to apps and embedded devices. It handles inference on the user's hardware across Android, iOS, macOS, Linux and Windows, as well as embedded systems and microcontrollers.

Pallaidium is a Blender add-on for filmmakers and AI artists who want to develop scenes with generated visuals and sound before shooting. It puts generation on the Video Sequence Editor timeline, where you can test dialogue timing and edit scenes alongside music.

Nemotron is NVIDIA's family of AI models for developers building agents that reason, write code and use tools. Teams can evaluate and adapt the models using published weights, training data and training recipes.

CTranslate2 is an inference library for developers who want to run Transformer models on their own hardware or servers. It focuses on reducing execution time and memory use for applications built in C++ or Python.

CogView4 is a text-to-image model for image creators and developers who want to generate images on their own hardware. It accepts Chinese and English descriptions and can include Chinese text in the results.

Demucs separates mixed music recordings into parts for musicians who want to work with individual instruments or vocals. The author maintains a separate repository, with no active feature development and slow responses expected.

TensorRT is NVIDIA's inference toolkit for developers deploying AI applications on NVIDIA GPUs. Its compilers, runtimes and model optimizations target lower response latency and higher throughput on local machines, self-hosted servers and edge devices.

vLLM Metal lets developers serve language models on Apple Silicon Macs through vLLM. It's for people who want to use their Mac's hardware for local AI inference while keeping vLLM's API server and request scheduling.

EmbeddingGemma is a local AI embedding model for developers adding search and retrieval to applications. It represents text, images, audio and video in a shared space so apps can find related content across media types.

Lucebox is a local AI workstation for developers and teams who want to run models on their own hardware. Its inference engine is also available separately for supported AMD and NVIDIA machines.

TensorFlow Model Optimization Toolkit is an open-source Python library for developers preparing TensorFlow and Keras models for deployment on mobile, IoT and other edge devices, or in the cloud. It helps reduce inference latency and cost, and adapt models to limits on memory, processing power, energy use and storage.

Tellama is a local LLM chat app for Android users who want conversations to stay on their phone. Chat with an installed compatible model works offline, and you don't need an account.

Optimum is a collection of Python packages for developers adapting Hugging Face models to their deployment hardware. It extends Transformers, Diffusers, TIMM and Sentence Transformers with tools for faster inference and training.

KoboldCpp runs language models on your own hardware for chat, creative writing and roleplay. It's a fork of llama.cpp with a bundled KoboldAI Lite browser interface.

Lollms WebUI is a local AI interface for writing, coding assistance and media generation through a choice of models. The project receives minimal maintenance, and its interface has no built-in user authentication.

StarCoder2 is a family of code generation models for developers who want code completion on their own hardware. These models predict continuations of code; they aren't instruction models, so conversational requests aren't their intended use.

KTransformers is a research framework for developers running or fine-tuning large language models on their own hardware. It focuses on mixture-of-experts (MoE) models, using CPU resources alongside GPUs to work within limited GPU memory.

MTPLX is a native Mac app, command-line tool and local LLM server for people running chat or coding models on Apple Silicon. It uses a model's own multi-token prediction heads to accelerate generation without a separate draft model.

Hunyuan-A13B is Tencent's language model for developers and researchers building self-hosted AI applications. It handles general text and reasoning tasks, with optimization for AI agent workloads.

FastLLM is an open-source inference engine for people running large language models on their own machines or self-hosted servers. Its main draw is hardware flexibility: large mixture-of-experts (MoE) models can spread their weights across GPU memory, system RAM and disk when they don't fit entirely on a graphics card.

MNN is a C++ framework for developers running AI models on phones, PCs and embedded devices. It supports inference and training on the device, with a small application footprint and hardware acceleration.

wllama lets developers add local LLM inference to browser apps without a separate inference server. It brings llama.cpp to JavaScript and TypeScript projects through WebAssembly, so models run on the user's device.

Paritok is a self-hosted gateway for developers whose coding agents send large file reads, tool output and conversation histories to an LLM. It reduces that input before the request reaches the upstream model.

whisper.cpp runs OpenAI's Whisper speech recognition models on your own hardware. It's for developers adding transcription to applications and people who need offline speech-to-text.

Wyoming Piper provides local speech generation for Home Assistant and other Wyoming-compatible clients. It's for people building a voice assistant who want speech synthesis to run on their own machine or server.

PocketPal AI runs language models on phones and tablets for people who want an assistant they can use without sending conversations to a server. Core chat works offline after a model download and doesn't require an account.

audio.cpp runs audio models locally for developers building speech tools and people generating or processing audio on their own hardware. It brings multiple audio tasks into one C++ runtime built on ggml.

DataDreamer is a Python library for researchers and developers who need to create training datasets and adapt LLMs to specific tasks. It connects data preparation and model training in reproducible workflows.