MOSS
A local conversational LLM that supports tool requests, CPU and NVIDIA GPU inference, and quantized models for lower memory use.
MOSS is a conversational language model from Fudan University for developers and researchers who want to run and adapt a local LLM. It includes model weights, dialogue datasets and fine-tuning code.
- Multi-turn chat: Ask questions, generate or summarize text, and write code through conversations that retain context.
- Tool requests: Plugin-trained models can request web searches, calculations, equation solving and image generation; plugins are disabled by default and require explicit enabling, configured interfaces and application code to execute requests.
- Fine-tuning: Adapt the base model with your own dialogue data using the supplied training code.
- Chat interfaces: Use Streamlit or Gradio browser demos, a command-line demo, or an API demo for other applications.
Inference runs on your machine or server through PyTorch and Transformers, with CPU and NVIDIA GPU support. FP16 inference supports an A100/A800 or multiple RTX 3090s. Quantized inference supports a single GPU through a Linux backend. One conversation needs roughly 12 GB of VRAM at 4-bit precision or 24 GB at 8-bit; longer conversations need more memory.
The hosted chat service requires an account. Web search and image generation depend on connected tools rather than the language model alone.
Code uses Apache 2.0, model weights use GNU AGPL 3.0, and datasets use CC BY-NC 4.0. The project also requires authorization for commercial use or public deployment of its models. Responses can contain factual errors or harmful content, and instruction following can be unreliable.
