Tools tagged with "Multimodal input"

Morphik Core is a self-hosted multimodal retrieval engine for developers building AI applications around visually rich documents. It searches diagrams, schematics, charts, and datasheets alongside text, so applications can retrieve information that text extraction alone can miss. You can run it on your own server, including through Docker, or use Morphik's hosted service.

Marqo Open Source is a self-hosted search engine for developers building semantic document search, image search, or retrieval for AI applications. It handles embedding generation alongside storage and retrieval, so applications can submit documents without maintaining a separate embedding service. The open-source project is deprecated and no longer receives updates.

NextChat is a self-hosted AI chat interface for people who want one place to use their own LLM server and cloud models. The web and desktop project is open source under the MIT license. You can host it with Docker or on Vercel, and desktop clients run on macOS, Windows and Linux.

LLM Farm runs large language models offline on iOS and macOS. It's for people who want on-device AI chat or need to compare how different models perform on Apple hardware before choosing one for a project. The app is open source under the MIT license, with ggml and llama.cpp handling local inference.

R2R is a self-hosted AI retrieval system for developers building applications that answer questions using their own documents. It combines search, retrieval-augmented generation (RAG) and a reasoning agent behind a REST API. The project is open source under the MIT license and runs as a Python service or in Docker.

MiniCPM-V is a family of local vision-language models for developers building apps that interpret images and video on their own hardware. It supports iOS, Android and HarmonyOS, as well as Mac deployment and server inference. The current repository states that MiniCPM-o/V code and model weights use Apache 2.0.

PaliGemma is a family of downloadable vision-language models for developers and researchers building applications that work with images and text. It combines SigLIP's image processing with Gemma's language capabilities to answer questions about visual content. Its main appeal is task-specific fine-tuning: you can adapt a base model to your own image data and intended use.

LLaVA is a family of vision-language models for researchers and developers who want to ask questions about images on their own hardware. It pairs a CLIP vision encoder with a language model to support image descriptions, visual reasoning and reading text in pictures. Its Python code is open source under Apache 2.0; the project places research-use restrictions on its data and checkpoints, with additional terms from the underlying models.

Granite is IBM's family of open-source AI models for developers and businesses that want to run and customize AI on their own hardware or servers. The language-model repository listed here is archived and no longer maintained. The broader family includes models for language, speech, document understanding and forecasting, released under Apache 2.0 for research and commercial use.

Sana is an open-source framework for running image and video generation on your own hardware, with image models small enough for laptop GPUs. It's aimed at creators who want local AI generation and developers who need training and inference pipelines for their own models. The code uses the Apache 2.0 license.

CLIP is an open-source image and text model that lets developers and researchers classify images using labels written in natural language, without collecting training examples for each task. It runs locally through PyTorch on a CPU or CUDA GPU. The code and model weights use the MIT license.

JoyCaption is an open-weight image captioning model for people preparing datasets to train or fine-tune diffusion models. It runs on your own GPU and covers both SFW and NSFW images, including photography, anime, digital art and furry artwork. Automated captions reduce the need to write descriptions by hand or find images that already have usable text.

Grounding DINO finds objects in images using category names or descriptive phrases you supply. It's a local AI model for developers and computer vision researchers who need detection beyond a fixed set of labels, including people building dataset annotation tools.

OpenCLIP is a Python and PyTorch library for developers and researchers who want to match images with text on their own hardware. It implements OpenAI's CLIP approach: images and descriptions become numerical representations that the model can compare. This supports image search and zero-shot classification, where text labels define the categories without a separate classifier trained for each task.

Intel IPEX-LLM is a library for developers running or fine-tuning models on Intel hardware. The project is archived and no longer maintained. Intel reports known security issues and no longer accepts patches or provides updates. The code is open source under Apache 2.0.

OmniGen is a local AI image generation model that handles text prompts, reference images, and image editing within one model. It's for creators who want to reuse subjects across images and developers building image tools on their own hardware. The code is open source under the MIT license.

Janus-Pro is a multimodal AI model from DeepSeek that answers questions about images and creates pictures from text prompts. It runs on your own hardware and suits developers and researchers who want both capabilities in one model. A local Gradio demo provides a browser interface, while Hugging Face hosts a separate online demo.

NativeMind brings a local AI assistant into Chrome for people who want help with webpages, documents, and writing without sending that content to a cloud model. The browser extension connects to Ollama on your machine, keeping prompts and AI processing on-device. It requires no account and uses the AGPL-3.0 open-source license.

Dia is the original text-to-speech model from Nari Labs that generates a two-speaker conversation from a written script in one pass. It's for researchers and developers who want to generate English dialogue on their own hardware, with control over speaker voices and delivery. The code and model weights are available under Apache 2.0. Dia2 is a separately linked successor.

LiteRT is Google's open-source framework for developers building AI into apps that run on users' own devices. It succeeds TensorFlow Lite and covers model conversion, optimization and local inference. It's licensed under Apache 2.0.

GLaDOS is a local AI voice assistant modeled on the sarcastic character from Valve's Portal games. It's for people who want a conversational companion on their own hardware, with camera awareness and connections to home automation. The Python project is open source under the MIT license and runs on Linux and Windows. macOS support is experimental.

Hollama is an open-source LLM chat app whose interface runs entirely in your browser. It's for people who want to chat with local AI through Ollama or connect to OpenAI servers, with support for multiple server connections. The interface stores data locally in the browser; the connected server handles model requests, so where inference runs depends on the server you choose.

gptme is a self-hosted AI agent that works directly in your terminal, with access to your files and installed tools. It's for developers who want a coding assistant in their own environment, and people who want an agent for data analysis or other knowledge work. The software is free under the MIT license and doesn't require a gptme account.

Wav2Lip is a local AI lip-sync tool that changes a face's mouth movements in an existing video to match supplied speech. It's for researchers and people making academic or personal video projects who want to process their own files. Commercial use is prohibited under the project's stated terms because its models were trained on the LRS2 dataset.

ExecuTorch is PyTorch's runtime for developers building AI into mobile apps, desktop software and embedded devices. It runs models on the user's hardware, with support for Android, iOS, Linux, macOS and Windows, as well as microcontrollers. Developers can reuse a PyTorch model across targets, though hardware-specific deployments need their own exported model files.

EchoMimic turns a portrait image and an audio recording into an animated talking-head video on your own hardware. Its distinguishing feature is editable facial landmark control: you can drive animation with audio, landmarks, or audio combined with selected landmarks. It's intended for academic research and suits people comparing methods for speech animation and facial motion control.

HuggingChat UI is the open-source chat application behind Hugging Face's hosted HuggingChat. You can run it on your own computer or server and connect it to a local LLM backend or a cloud provider. It's for people and teams who want a browser-based ChatGPT alternative with control over the chat service and where its data lives. The code uses the Apache 2.0 license.

TagGUI is a desktop app for people preparing image datasets for generative AI training. It combines local AI captioning with manual tag editing, so you can generate descriptions and correct them in the same workspace. It's open source under GPL-3.0 and runs on Windows, Linux and macOS, though macOS doesn't have a packaged release.

GPTQModel is a Python toolkit for developers compressing LLMs and running them on their own hardware or servers. It brings model calibration, compression, quality checks and inference into one API, so teams can compare quantization methods without adopting a separate tool for each one.

MusicGen is Meta AI's music generation model within AudioCraft, a PyTorch library for developers and audio researchers who want to generate music in their own computing environment. It creates music from text descriptions and can use a melody to guide the result. AudioCraft includes both inference and training code, so it's suited to people building audio tools or studying music generation.

MTEB is an Apache 2.0 Python toolkit for evaluating embedding models and retrieval systems. It runs evaluations through Python or a command-line interface and publishes an interactive leaderboard.

Mochi Diffusion is a native macOS image generator for people who want to create images on an Apple Silicon Mac without sending prompts or results to a cloud service. It runs Stable Diffusion and FLUX.2 Klein locally, with completely offline generation. The app collects no telemetry or other data and is open source under GPL-3.0.

Paperless-GPT adds AI text extraction and document organization to an existing paperless-ngx library. It runs in Docker on your own server and suits people who want less manual sorting of scanned paperwork. The project is open source under the MIT license.

AIBrix is open-source infrastructure for teams serving large language models on their own Kubernetes clusters. It focuses on the work around inference: directing requests, scaling capacity and managing models across servers. Enterprise infrastructure teams can use its components to build a self-hosted model service. It's licensed under Apache 2.0.

Zerox is an MIT-licensed OCR library for developers preparing documents for AI applications. Its Node.js and Python packages run on your own machine or server, while cloud vision models read the document pages and produce Markdown. Document conversion happens locally, but page images go to the selected model provider, so this workflow needs internet access and provider credentials.

ChatWise is a desktop chat app for connecting different model providers through one interface. It stores data locally; model processing follows the provider you choose. Its official documentation includes Ollama for models running on your own machine, alongside cloud providers connected with your API keys.