vLLM team’s startup Inferact runs Kimi K3 on 16 TPU v7 chips 57% faster than GB200
Inferact, founded by the original vLLM team, wrote a megakernel inference kernel for Google TPUs. Paired with DeepSeek’s DSpark speculative decoding, 16 TPU v7 chips served Kimi K3 at 709 tokens per second versus 452 on GB200 under the same setup, QbitAI reports. The code is open source.
Inference startup Inferact, founded by the team behind vLLM, published benchmarks showing 16 Google TPU v7 Ironwood chips serving Kimi K3 at 709 tokens per second, against 452 for 16 Nvidia GB200s under the same conditions, 57% faster, QbitAI reported on September 26.
The facts
- Setup: both sides ran Kimi K3 on the vLLM engine. The only variables were the chip and the low-level kernel implementation.
- How: Inferact wrote a megakernel for TPUs that fuses the hundreds of small programs normally scheduled during inference into one, pushing memory-bandwidth utilization close to the hardware’s theoretical peak. It adds DeepSeek’s DSpark speculative decoding, with an average acceptance length of 6.
- Without speculative decoding: at batch size 1, TPU does 249 tokens/s versus 127 for GB200; at batch size 8, 865 versus 636.
- Other models: four TPU v7 chips serve Qwen 3.8 27B at 1,515 tokens/s, versus 695 on the same GB200 configuration.
- Accuracy: with greedy decoding, Kimi K3 on TPU matched GPU scores on GPQA-Diamond (94.4%) and GSM8K (97.2%), according to Inferact. The code is open source.
Our take
The numbers show inference speed increasingly comes down to the software stack, not just chip specs: same model, same engine, a different kernel, and the gap is more than half.
Keep some skepticism: these are Inferact’s own tests, and we haven’t seen independent replication. If you’re choosing hardware for self-hosted inference, treat it as a reason to put TPUs on the shortlist, then benchmark with your own models and traffic.