Tools tagged with "Quantization"

FastVLM is a family of local vision-language models for developers and researchers building applications that answer questions about images. It focuses on reducing the delay before an answer starts, especially with high-resolution images.

MonkeyOCR is a local AI document parser for developers and researchers who need to extract content from English and Chinese PDFs or images. It recognizes page structure and reading order alongside the content itself.

Open Instruct is AllenAI's toolkit for researchers and developers adapting pretrained language models with public datasets. It includes the training recipes behind TÜLU and downloadable checkpoints for comparing training stages.

Sana is a local AI image and video generation framework for creators and developers who want to generate media on their own hardware. It includes training and inference tools for working with its diffusion models.

ChatLLM Web is a browser app for people who want AI help with text and code while keeping their working data on their device. It combines chat, model selection, and an agent that can work with files you choose.

subgen creates subtitles from audio and video for personal media libraries. It's useful for films and shows that lack usable subtitles, with automatic generation tied to your media server.

WhisperX is a local speech-to-text tool for transcribing interviews, meetings and long recordings. It builds on OpenAI's Whisper, adding individual word timings and optional speaker identification for subtitles and transcript review.

LLaMA-Factory is a local model training platform for developers and researchers who want to adapt pretrained models to their own tasks. Its web UI lets you fine-tune models without writing code.

Deltafin runs full Kimi K3 on a single device for researchers and developers exploring large models on their own hardware. It preserves the original expert weights and complete expert bank, but responses are slow and long conversations take substantially longer than short completions.

JoyCaption generates image descriptions for people preparing datasets to train or fine-tune diffusion models. It can caption images on your own GPU, reducing the work of writing training descriptions by hand.

LLMLingua is a Python library for developers building LLM applications that need to fit lengthy input into a model's context window or reduce token costs. It uses smaller models to remove less useful text before the prompt reaches the answering model.

torchtune is a Python library for developers and researchers who want to adapt LLMs on their own hardware with editable PyTorch training code. The project is no longer actively maintained.

LLaVA is a vision-language model project for researchers and developers who want to ask questions about images on their own hardware.

llama.rn is a library for developers adding local LLM features to React Native apps on iOS and Android. It brings llama.cpp inference onto the device, so an app can process model requests on the user's hardware.

Pooled runs a single open model across laptops, desktops and phones in browser tabs. It lets people combine their devices' memory to use a model that wouldn't fit on one device, for shared chat or coding tasks.

Private LLM runs AI chat on iPhone, iPad and Mac for people who want to use language models without sending conversations to a cloud service. It works offline after the first model download and doesn't require an account.

Pocket AI Lab runs AI models on an iPhone and helps you judge which ones the phone can handle. It's for people who want private chat on their device and want to check memory and storage before downloading model weights.

TorchAO is a PyTorch library for developers who want to train and run models on their own hardware with less memory. It supports language models and image or video generation, and works with torch.compile and FSDP2 across most Hugging Face PyTorch models.

Cactus is an on-device AI engine for developers adding automation to mobile apps, wearables and other small devices. Its Needle model turns requests into tool calls locally, including when the device has no internet connection.

RealtimeSTT is a Python library for developers building voice assistants, dictation tools and apps that turn live audio into text. It can show text while someone speaks and produce a final transcript when they finish.

MiniCPM-V is a family of local vision-language models for developers building applications that interpret images and video. It can run on phones and other hardware you control.

ncnn is a C++ inference framework for developers adding on-device AI to mobile, desktop and embedded applications. It focuses on a small memory footprint and doesn't require third-party runtime dependencies.

Phosphene is a generative media studio for people creating videos on their own Mac. It combines video, still images, music and character training in one interface, with generation running on-device.

Docker Model Runner lets developers run AI models on their own computers or servers and connect them to applications. It combines local inference with model distribution through container registries.

Nexa SDK is an on-device AI framework for developers building apps that run models on users' hardware. It covers language, vision and audio tasks through a shared interface for local inference.

Sonar serves language and multimodal models for developers and teams running AI applications on their own hardware. It builds on vLLM and works with Hugging Face-compatible models.

TensorRT-LLM is an inference library for developers serving LLMs on their own NVIDIA GPUs. It supports application-level inference and server deployments, with performance optimizations for workloads spread across one or several machines.

ACE-Step Studio is a desktop AI music generator for musicians and anyone who wants to create songs on their own hardware. It turns a short idea, or a description with lyrics, into a full vocal track or instrumental.

RealtimeTTS is a Python text-to-speech library for developers building voice assistants and other apps that need spoken responses before the full text is ready. It accepts both complete text and incoming LLM output.

llama.cpp is a local LLM inference engine for developers building AI applications or running models on hardware they control. It supports language and vision-language models, with command-line tools and a server API for using the engine directly or connecting other software.

InsightFace is a face and person analysis toolkit for developers building identity verification, access control and media applications. Its Python tools and recognition server can process data on your own hardware.

Soup is a local LLM fine-tuning tool for developers and teams adapting models to their own datasets. It brings training, evaluation and export into one workflow that runs on your own hardware without requiring a cloud service.

SD.Next is a self-hosted web interface for artists and researchers who want to create and edit AI images and videos on their own hardware. It supports Stable Diffusion and other diffusion models.

Axolotl is a free, open-source LLM fine-tuning framework for developers and researchers adapting models to their own data. It supports training on local hardware or cloud infrastructure you control.