Inkling: setup with Transformers and llama.cpp

This walkthrough introduces Thinking Machines' Inkling, an open-weight multimodal model released under Apache 2.0. The supplied release details list 975 billion total parameters, 41 billion active parameters and a one-million-token context window. The speakers explain its relative attention and SConv architecture, then cover deployment through Transformers, llama.cpp, SGLang and vLLM.

Hardware is a central constraint for anyone considering a self-hosted deployment. The release notes put BF16 memory requirements near 2 TB and NVFP4 near 600 GB. Even the 1-bit GGUF remains large: speakers give figures of roughly 270 to 285 GB. They report about 40 tokens per second with llama.cpp on eight H100 GPUs. These are reported test results, not a performance guarantee for other hardware.

The llama.cpp example exposes an OpenAI-compatible endpoint and a web interface for image, audio and text input. A Pi harness demo tackles a visual reasoning puzzle, sometimes using Python. Merve also describes her own vision and audio evaluations; her preference for medium reasoning effort at 0.7 is a judgment from those tests.

The final section covers post-training with Tinker and an OpenEnv terminal environment, plus distillation options. Hugging Face Inference Providers offers a hosted route to try the model for viewers without the required compute.