Inference competition shifts to MLPerf
vLLM submitted NVIDIA’s first MLPerf VLM benchmark entry using its engine, spotlighting a wave of inference‑focused optimization work as vendors chase lower token costs and real‑world throughput. Public MLPerf inference rankings are becoming a practical signal of which stacks handle efficient token generation under real workload shapes, not just raw theoretical peaks. (x.com) (x.com)
Running an artificial intelligence model has become a scheduling problem, not just a chip problem. The expensive step is inference, which is the moment a trained model turns your prompt and an image into the next token, then the next one, while a server tries to keep thousands of users waiting as little as possible. (docs.nvidia.com) That is why a benchmark called Machine Learning Performance Inference has become a bigger deal. MLCommons says its suite measures how fast systems process inputs and produce results in representative, reproducible deployment scenarios instead of one-off lab demos. (mlcommons.org) Machine Learning Performance Inference does not just ask who can sprint. Its original paper says the benchmark uses traffic generators and scenarios like server and offline mode, which is closer to asking whether a restaurant kitchen can handle a dinner rush than whether one chef can plate one dish fast. (arxiv.org) The new twist is multimodal work. On April 1, 2026, MLCommons said version 6.0 added a new vision-language model benchmark built to turn Shopify product catalog images and text into structured metadata, so the test now checks systems that have to look and read before they answer. (mlcommons.org) A vision-language model is a model that mixes pictures with words in one request. It is the difference between asking a cashier to read a label and asking the same cashier to inspect the item in your hand at the same time. (mlcommons.org) That is where vLLM enters. NVIDIA’s documentation describes vLLM as a high-throughput, memory-efficient serving engine for large language models, and its core trick is PagedAttention, which manages the model’s growing memory cache the way an operating system manages virtual memory. (docs.nvidia.com) That memory cache is the pile of intermediate state a model keeps while it generates tokens. The PagedAttention paper says old systems wasted large chunks of that memory because every request grows and shrinks unpredictably, while vLLM cuts that waste by breaking memory into smaller pageable blocks. (arxiv.org) vLLM also uses continuous batching, which means new requests can join the line while older requests are still being processed. NVIDIA’s docs describe it as a continuous stream instead of fixed batches, which is like filling empty seats on a bus at every stop instead of waiting to reload the whole bus from scratch. (docs.nvidia.com) That matters because public benchmark tables are starting to reveal software choices, not just hardware choices. In the Machine Learning Performance Inference version 6.0 results repository, NVIDIA’s new Qwen3-VL-235B-A22B vision-language submission includes a dedicated vLLM code path under NVIDIA’s closed submission tree, marking NVIDIA’s first published Machine Learning Performance Inference vision-language entry built around vLLM. (github.com) The competitive center of gravity has shifted with that move. A few years ago, the headline number was training a bigger model; in 2026, the harder commercial question is how cheaply a system can generate useful tokens all day, under mixed traffic, with quality targets still met. MLCommons says the published results are meant to help customers procure and tune systems, which is why these rankings now look less like marketing and more like a buyer’s guide for inference stacks. (mlcommons.org)
