Willow Inference Server

Willow Inference Server is a self-hosted speech server for developers building voice assistants, transcription apps, and accessible interfaces. It handles speech recognition and speech generation on your own hardware.

• Whisper transcription: Transcribe uploaded audio or live speech with Whisper models, including base, medium, and large-v2, using CTranslate2 for inference.
• Live browser audio: Stream speech through WebRTC for transcription during long-running sessions, with reduced bandwidth use while audio recording is idle.
• Speech generation: Generate spoken responses for assistants and accessibility applications, and create custom TTS voices from short audio recordings.
• Application integration: Connect devices and applications through REST, WebSockets, or WebRTC, with direct support for Willow voice assistants and a TypeScript WebRTC client library.

WIS is open source under Apache 2.0 and runs locally through Docker on Linux, with Windows support through WSL. It primarily targets NVIDIA CUDA GPUs, including the Tesla P4 and GTX 1060, but can also run on CPU alone. CPU performance may fall short of its targets for latency-sensitive speech tasks. The project reports that Whisper base, medium, and large-v2 can stay loaded alongside TTS within 6GB of VRAM.

Self-hosted inference runs on your server. Tovera also operates a hosted example server for Willow users; audio sent there is processed remotely.