audio.cpp review: local audio AI vs Ollama and whisper.cpp

audio.cpp is an open source C++ project for running audio models in one native runtime. The presenter describes support for speech recognition, text-to-speech, voice cloning, voice conversion, speaker identification and music generation. The review compares its scope with whisper.cpp for transcription, Ollama and llama.cpp for language models, and Piper for speech generation.

The demonstration uses a 10-second voice reference to generate new speech on a MacBook. The presenter says inference runs entirely on the machine through GGML with the Metal backend. A command line interface selects the task, model family and backend; server mode exposes an API described as compatible with OpenAI's audio endpoints, so existing client code can point at localhost.

Setup and reliability receive close attention. According to the presenter, prebuilt releases are available only for Windows, while Mac users must compile with Xcode tools and a Metal build script. Inference needs no Python runtime, but model downloads and conversion still use a Python helper. Models carrying an MLX label reportedly use GGML and Metal rather than Apple's MLX.

The presenter cautions that advertised performance figures came from an Nvidia 5090, not Apple Silicon. Reported crashes, memory leaks and robotic output make this local AI runtime an experiment rather than a recommendation for dependable production use. The presenter also identifies the license file as Apache 2.0 despite GitHub's license detection result.