Tools tagged with "Distributed execution"

Mooncake provides storage and data transfer components for teams running distributed LLM inference. Its Transfer Engine and Mooncake Store support shared caching and deployments that separate prompt processing from token generation.

ACP Bridge lets teams and bots access coding agents running on a development machine or server through a shared HTTP API. It connects agents through the Agent Client Protocol (ACP), so callers don't need each agent installed on their own computer.

Paddler lets product and infrastructure teams serve AI models on their own servers and spread requests across multiple machines. It's built for applications that need text generation or embeddings while keeping inference within infrastructure the team controls.

Megatron-LM is a Python framework for researchers training large language models on NVIDIA GPU infrastructure. It provides reference training scripts for experimenting with models and studying distributed training.

llm-d is a distributed LLM inference stack for infrastructure teams serving models across a Kubernetes cluster. It coordinates model servers such as vLLM and SGLang to handle production traffic across multiple machines.

MLForecast is an open-source Python library for data scientists and developers who need to forecast many time series with machine learning. It combines time series feature engineering with scikit-learn-compatible models, so teams can use regressors such as LightGBM and LinearRegression rather than adopt a dedicated forecasting model. Its license is Apache 2.0.

kvpress is a Python library for developers and researchers reducing the memory needed for long-context LLM inference. It compresses the key-value cache that grows as a model processes longer inputs.

claudectl manages multiple Claude Code sessions for developers who need to track parallel work and intervene when an agent gets stuck. Its terminal dashboard brings session activity, approvals and spending into one view.

LMDeploy is an open-source toolkit for developers who want to run language and vision-language models on their own GPU hardware. It supports offline batch processing and model servers for applications.

XGBoost is a machine learning library for training and running gradient-boosted tree models on your own hardware. Data scientists use it for classification, regression and ranking, with parallel computation for larger workloads. The code is licensed under Apache 2.0.

Text Generation Inference (TGI) serves LLMs on your own hardware for developers building chat apps and text generation services. The project is archived and no longer receives updates.

MosaicML Composer is a Python library for training neural networks on your own GPUs or multi-node clusters. Built on PyTorch, it's aimed at researchers and engineers who want distributed training with room to customize their experiments. It's open source under Apache 2.0, and prebuilt Docker images are available.

Intel IPEX-LLM is a library for developers running or fine-tuning language models on Intel hardware. The project is archived and no longer maintained, and Intel has identified known security issues in the code.