Tools tagged with "Quantization"

Claude Code Local lets developers use Claude Code with a model running on their own computer. It suits people who want to keep coding offline or continue working when they hit cloud usage limits.

FlashInfer is a GPU kernel library for developers building LLM inference systems on their own hardware or servers. It supplies the attention, matrix multiplication and mixture-of-experts operations used by model serving software.

MLX Tune is a Python library for adapting AI models on an Apple Silicon Mac. Formerly unsloth-mlx, it is an independent project and is not affiliated with Unsloth. Its Unsloth-compatible API lets you prototype training locally with minimal changes to existing scripts, then move to NVIDIA hardware with Unsloth. Training runs through Apple's MLX framework and doesn't require a cloud GPU.

MaxText is an LLM training library for researchers and engineers building models from scratch or adapting existing ones. It provides a reference implementation teams can modify for their own model designs, with training that scales from a single host to large clusters.

Jina Embeddings is a family of models for developers building semantic search, retrieval-augmented generation (RAG) and AI agent applications. The models convert content into vectors that a search system can compare for relevance.

WhisperWriter turns microphone speech into typed text in your active window. It's for people who want dictation in their desktop apps, with a choice of transcription on their own computer or through an API.

picoLLM is an on-device LLM inference SDK for developers adding text generation to their apps. Inference works offline, but a Picovoice account and AccessKey are required, and key validation needs an internet connection.

GLM-4.5 is an open-source language model family for developers building AI agents and coding tools on their own servers. It supports reasoning and tool use, but its supplied inference configurations require server-class NVIDIA hardware.

FlashRank is a Python library that runs reranking models locally on CPU, without Torch or Transformers. It's for developers building search systems or retrieval-augmented generation (RAG) apps who want to reorder retrieved passages by relevance before passing them to an LLM.

InfiniteYou generates images from a reference face and a text prompt while preserving the person's facial identity. It's a local AI framework for researchers studying personalized image generation, with model use restricted to academic research.

ggml is a library for developers building machine learning runtimes and local AI applications. It provides tensor computation across different kinds of hardware; GGUF is the model file format covered by the project.

Klee is a native macOS AI chat app and agent for people who want an assistant to work with files on their Mac. It runs models locally through MLX, and downloaded models can chat offline without an account or API key.

MLX LM is a Python package for developers and researchers who want to generate text and adapt language models on Apple Silicon Macs. It uses MLX for local inference and training, with access through Python or a terminal.

KittenTTS is a Python text-to-speech library for developers adding local speech generation to apps or producing spoken audio from text. It supports voice cloning without model training and doesn't require a GPU.

FlashMLA is an attention library for developers building DeepSeek-V4.1 inference systems on their own hardware. It requires NVIDIA SM100 or SM103 GPUs, or Huawei Ascend 950 NPUs.

TranslateLocally translates text on your own computer for people who want local machine translation and teams working with their own translation models. It uses Marian and Bergamot, with a desktop interface for everyday text and command-line access for batch work.

Megatron-LM is a Python framework for researchers training large language models on NVIDIA GPU infrastructure. It provides reference training scripts for experimenting with models and studying distributed training.

kvpress is a Python library for developers and researchers reducing the memory needed for long-context LLM inference. It compresses the key-value cache that grows as a model processes longer inputs.

LMDeploy is an open-source toolkit for developers who want to run language and vision-language models on their own GPU hardware. It supports offline batch processing and model servers for applications.

Intel Neural Compressor is a Python library for developers who want to compress AI models for deployment on their own hardware or servers. It supports LLMs and other deep learning models, with extensive testing on Intel platforms.

Whisper Desktop turns audio and video recordings into transcripts and subtitles on your computer. It's for people working with interviews, lectures, podcasts or videos who want a desktop interface for transcription and captioning.

Voice-Pro is a locally run speech processing and dubbing app for creators working with recordings or multilingual video. Its Gradio browser interface brings audio preparation, transcription and speech generation into one workspace. Internet access is required.

PaddleClas is a PaddlePaddle toolkit for developers and researchers building image classification and recognition applications on their own hardware. It covers fixed-category classifiers and image matching for large or frequently changing catalogs.

Wan2GP generates media on your own computer for creators who want to work with several AI models in one browser interface. It builds on Wan-Video/Wan2.1 and focuses on making generation accessible on GPUs with limited memory.

Text Generation Inference (TGI) serves LLMs on your own hardware for developers building chat apps and text generation services. The project is archived and no longer receives updates.

Intel IPEX-LLM is a library for developers running or fine-tuning language models on Intel hardware. The project is archived and no longer maintained, and Intel has identified known security issues in the code.