Qwen 3.8 27B and Hermes Agent: vLLM setup and comparison

The speaker returns to Qwen 3.8 27B at FP16 for Hermes Agent projects after finding the quantized Flash Next model less reliable in extended agent workflows. He says 27B completed a vLLM monitoring app that Flash Next could not finish, while preferring Flash Next for creative, detailed conversations. This is a comparison of his workloads and precision settings, rather than a controlled benchmark.

The setup runs on four RTX 3090 GPUs, with a separate RTX 4090 handling the desktop on the same machine. The vLLM walkthrough covers GPU memory utilization of 0.95, 20 maximum sequences, FP8 E4M3 KV cache, chunked prefill and prefix caching. He also explains his reasoning and tool parser choices. Hermes connects through a custom endpoint, then discovers the model and context length automatically.

The monitoring app tracks active requests, prefill and decode throughput, and cache hits while Hermes delegates research to sub-agents. The speaker reports aggregate decode throughput reaching roughly 222 tokens per second during the demonstration. That figure covers concurrent work, not one conversation's response speed.

For this local LLM workflow, he accepts slower individual responses in exchange for the output quality he reports. He warns that heavier concurrency increases waiting time and says four agents usually handle his research reports in about 30 to 35 minutes.