Tools tagged with "Quantization"

PaddleSeg is an image segmentation toolkit for developers building computer vision applications on their own hardware. It covers annotation, training and deployment for tasks such as medical image analysis, industrial inspection and portrait effects.

Yi is a family of language models from 01.AI for developers and researchers who want to run English and Chinese AI on their own hardware. It includes chat models for conversation and base models you can adapt to specific tasks.

MLX Omni Server lets developers use local models through both OpenAI-compatible and Anthropic-compatible APIs on an Apple Silicon Mac. It's for applications built around those SDKs that need inference on hardware the developer controls. Apple's MLX framework provides hardware acceleration, and all AI processing stays on the Mac.

OpenRLHF is a self-hosted Python framework for researchers and teams training language models with human feedback or custom rewards. It supports fine-tuning and reinforcement learning on your own GPU hardware, including training agents that interact with an environment.

FastDeploy is a self-hosted inference toolkit for teams deploying language and vision-language models on Linux servers. Built on PaddlePaddle, it supports offline inference and online serving.

Ludwig is a framework for developers and researchers training custom AI models without writing their own training loops. It handles preprocessing, training and evaluation from a model definition, covering LLMs as well as tabular and multimodal tasks.

Pico MLX Server is a macOS app for people who want to run MLX models locally and use them with an existing AI chat client. It gives MLX Server a graphical interface and exposes local models through an OpenAI-compatible API. The project is open source under the MIT license and requires macOS Sonoma or later.

FastEmbed is a Python library for developers building semantic search and retrieval-augmented generation (RAG). It turns text into vectors and evaluates search results on your own hardware without requiring a cloud embedding API.

FastChat is a self-hosted LLM platform for developers and researchers who want to serve chat models on their own hardware and compare their responses. It powers Chatbot Arena and includes a locally hosted side-by-side comparison interface. The Python code is open source under Apache 2.0; model licenses apply separately, including Llama's license for Vicuna.

HunyuanImage is Tencent's text-to-image model for people who want to generate images on their own hardware. It handles detailed scene descriptions, including requests for lettering within an image.

lm-evaluation-harness is a Python framework for researchers and developers comparing language models on shared benchmarks. Public prompts help make results comparable across model checkpoints and research papers.

PowerInfer is a local LLM inference engine for developers and researchers running large models on a PC with a consumer GPU. Its performance gains depend on sparse models, which use only part of their network for each input.

AgileRL is an open-source Python framework for teams training AI agents on their own tasks, including LLM agents and agents for robotics, finance or logistics. Its evolutionary hyperparameter tuning adjusts training settings during a run, reducing the need for separate tuning trials. The framework uses the Apache 2.0 license.

Distributed Llama runs language models across your own networked computers. It's for people who want to share inference work and model memory across several devices instead of fitting the whole workload on one machine.

Wyoming Faster Whisper converts speech to text on your own hardware for Home Assistant and other Wyoming-compatible clients. It's for home voice systems that need local transcription and better recognition of household names.

PEFT is a library for developers adapting pretrained AI models on their own hardware. It trains a small portion of a model's parameters to reduce the compute and storage costs of full fine-tuning.

biniou brings AI chat and media creation into one browser interface on your own computer or server. It's for people who want to work with several kinds of generative AI without switching between separate apps.

FinGPT is a financial LLM framework for researchers and developers building tools that interpret market news and company information. It includes trained models, financial datasets, and tools for adapting models to specific finance tasks.

Moshi is a voice AI model and dialogue framework for researchers and developers building spoken applications. It processes speech directly and can listen while it speaks, including information such as emotion and non-verbal cues.

GPTQModel is a Python toolkit for developers compressing LLMs and running them on their own hardware or servers. It combines model preparation, quality evaluation and inference so you can assess compression choices within one workflow.

PaddleDetection is a computer vision toolkit for developers building image and video analysis systems. It covers training, adapting and deploying models for uses such as inspection and traffic analysis.

SmolVLM is a compact vision language model for developers building applications that interpret images and answer questions about them. It can run on-device, including on laptops and smaller local setups.

MFLUX is a Python toolkit for creators and developers who want to generate and edit images on an Apple Silicon Mac. It provides a command-line interface for working with models locally.

Infinity Embeddings is a self-hosted model server for developers building semantic search and retrieval-augmented generation applications. It runs retrieval models on your own hardware and can serve several models together.

llmware is a Python framework for developers building applications that answer questions using their own documents or carry out multi-step AI agent workflows. It focuses on small, specialized models that can run on laptops, edge devices, and self-hosted servers.

bitsandbytes is a Python library for developers who need to run or fine-tune large language models on their own hardware with less memory. It works within PyTorch workflows.

RKLLM is a toolkit for developers building local AI applications on Rockchip boards. It runs language and multimodal models on the chip's neural processing unit (NPU).

faster-whisper is an open source Python library for developers adding speech transcription to their software. It uses CTranslate2 to run OpenAI's Whisper models, with faster transcription and lower memory use than the original implementation in the project's benchmark comparisons.

Airux Pocket AI runs AI conversations on Android phones for people who want offline chat and photo analysis without cloud inference. You don't need an account, and downloaded models run directly on your phone.

ExLlamaV2 and V3 are local LLM inference libraries for developers building AI applications or hosting models on consumer GPUs. V2 is archived for now and isn't currently maintained; development continues in V3.

gpt-oss is a pair of OpenAI reasoning models for developers who want to run a local LLM or serve models on their own hardware. The smaller model targets local and specialized applications; the larger one targets general-purpose tasks that need more reasoning.

Unsloth is a desktop app for people who want to chat with models, train them on their own data, and create media on their own hardware. Its no-code interface brings these tasks into one workspace.

CodeFuse ChatBot is a self-hosted AI assistant for developers and DevOps teams who want answers grounded in their own code repositories and documentation. It combines repository analysis with agents that can use tools and execute code in a sandbox, covering work across design, coding, testing, deployment and operations.

DeepGEMM is an open-source CUDA kernel library for developers building LLM training and inference systems on their own NVIDIA GPUs. It supplies matrix operations and specialized computations for mixture-of-experts (MoE) models.