Via bizon-tech.com
vLLM hits 500K GPUs as co-founder Simon Mo makes the case for open models
The UC Berkeley-born inference engine is now a cornerstone of production AI infrastructure, and its co-founder just raised $150M to prove it
Half a million GPUs are now running vLLM. That number, cited by the project’s own maintainers, is the kind of figure that stops a conversation cold.
What vLLM actually is, and why it matters
vLLM started at UC Berkeley’s Sky Computing Lab as a research project tackling one of inference’s most stubborn problems: GPU memory is expensive, and large language models waste a lot of it.
The core innovation is something called PagedAttention. Think of it like virtual memory management for your laptop, but applied to the KV cache that language models use to track context. It lets a GPU serve far more requests simultaneously by being smarter about how it allocates memory on the fly.
The project now supports over 500 model architectures and more than 200 accelerator types. In May 2025, it became a PyTorch Foundation project. The GitHub star count climbed into the tens of thousands rapidly.
Simon Mo, Inferact, and the $150M bet on open inference
Simon Mo, one of vLLM’s core maintainers, co-founded Inferact in 2025. The company raised a $150M seed round at an $800M valuation, led by a16z and Lightspeed.
Mo has been vocal about why open-weight models deserve serious consideration for production deployments. His argument lands on three pillars: control, customization, and cost.
On control, when a proprietary API changes its pricing, deprecates a model version, or experiences an outage, every company built on top of it absorbs that risk with no recourse. On customization, open-weight models can be fine-tuned, quantized, and adapted to specific domains in ways that closed APIs do not allow. On cost, API pricing is convenient at low volumes, but at production scale the per-token costs compound quickly, and the economics of running your own infrastructure on optimized hardware start to look compelling.
The real shape of 500K GPUs
The 500K GPU figure reflects the aggregate scale across all organizations running vLLM in production, spanning cloud providers, on-premise clusters, and enterprise deployments worldwide. Individual deployments typically operate across hundreds to thousands of GPUs, using a combination of tensor parallelism and pipeline parallelism. Tensor parallelism splits a single model layer across multiple GPUs simultaneously. Pipeline parallelism distributes different layers of the model across different GPU groups.