Intel Arc Pro B70: llama.cpp and vLLM setup compared

This walkthrough examines local LLM inference on the Intel Arc Pro B70, a GPU with 32GB of GDDR6 memory and 608 GB/s memory bandwidth. The presenter compares it with the AMD Radeon AI PRO R9700 and explains how software support affects the results.

The setup uses Fedora containers, with separate llama.cpp images for SYCL and Vulkan. The presenter suggests distrobox for Ubuntu or Arch Linux, citing reports of toolbox compatibility problems. Llama Cockpit provides model downloads and container management; its server mode can launch models directly through Podman or Docker. The demonstration also covers accessing llama.cpp's browser interface through SSH port forwarding on port 8080.

In the presenter's llama.cpp tests, the B70 trails the AMD card. He attributes much of the gap to incomplete matrix engine support and reports SYCL memory issues that limit his context tests to 16K. His multi-token prediction trial also runs slower than generation without it.

Intel's LLM Scaler fork of vLLM produces stronger benchmark results, but compatibility is narrower. The presenter reports that Gemma 4 is unsupported in the tested version and that AWQ models fail to run, while some GPTQ models work. These results describe the tested software stack rather than a fixed performance limit of the hardware.