LLM Inference Engines and Libraries
The engines and bindings that run models inside your own code: llama.cpp, MLX, ONNX Runtime and Hugging Face Transformers.
Open-source computer vision library for local detection, segmentation and tracking, with AGPL-3.0 licensing and exports to ONNX, TensorRT and CoreML.
62.1KUpdated 24 hours agoAGPL-3.0
#ONNX
Ultralytics YOLO is an open-source Python computer vision library for developers building applications that analyze images and video on their own hardware. It supports local and edge deployment, including NVIDIA Jetson, Raspberry Pi and mobile phones. A separate hosted platform provides browser-based annotation, cloud GPU training and managed prediction endpoints.
Computer Vision ModelsEdge and Embedded AIInference Libraries and BindingsTraining Frameworks+0
A local LLM engine that runs models in the browser with WebGPU acceleration, OpenAI API compatibility, and an Apache 2.0 license.
19.2KUpdated 2 weeks agoApache-2.0
Web · Browser Extension#OpenAI-compatible API#Streaming inference#Structured output
WebLLM runs language models directly in a user's browser, using WebGPU for GPU acceleration. It's an open-source engine for developers building web-based AI assistants and Chrome extensions that process prompts on the user's device rather than an inference server. The project uses the Apache 2.0 license.
Inference Libraries and BindingsLLM Libraries and SDKsOn-Device and In-Browser AI+0
Python library for running and training embedding and reranker models locally, with Apache 2.0 licensing and pretrained models on Hugging Face.
19.1KUpdated 1 week agoApache-2.0
#Hugging Face integration#Multilingual#Multimodal input
Sentence Transformers is an open-source Python library for developers building semantic search and document retrieval on their own hardware. It runs embedding and reranker models locally, turning content into numerical representations for similarity comparisons and scoring results against a query. The library uses the Apache 2.0 license.
Inference Libraries and BindingsRAG and Vector SearchTraining Frameworks+0
Open-source Rust ML framework for running models locally on CPUs, NVIDIA GPUs or in browsers, with Apache 2.0 licensing and quantized LLM support.
21.1KUpdated 2 days agoApache-2.0
macOS · Web#GGUF#Hugging Face integration#Multilingual
Candle is a Rust machine learning framework for developers who want to embed local AI in applications or deploy models on their own servers. It produces lightweight binaries that don't need Python in production, making it a candidate for serverless inference where a large runtime can slow startup. Its API uses tensor operations familiar to PyTorch developers.
Inference Libraries and BindingsOn-Device and In-Browser AITraining Frameworks+0
An open-source Python library for local speech transcription with Whisper models. It runs on CPUs or NVIDIA GPUs and uses CTranslate2.
25.6KUpdated 3 hours agoMIT
#Batch processing#Hugging Face integration#Quantization
faster-whisper is a Python library for people building local speech transcription into their own software. It runs OpenAI's Whisper models through CTranslate2, with faster processing and lower memory use than the original Whisper implementation in the project's comparisons. It runs on a CPU. NVIDIA GPUs are supported too, and the code is open source under the MIT license.
Inference Libraries and BindingsSpeech-to-Text and Transcription+0
