vLLM

vLLM

vLLM is an open-source library for running and serving large language models with high throughput and efficient memory use. You install it with pip or Docker, point it at a model from Hugging Face, and vLLM starts a server with a drop-in OpenAI-compatible API, plus Anthropic Messages and gRPC support. Its core idea is PagedAttention, which manages the attention key-value cache in pages so more requests fit on the same GPU, combined with continuous batching, prefix caching and speculative decoding.

Production credibility: vLLM was first developed in the Sky Computing Lab at UC Berkeley, and its README says it is now maintained by dozens of academic institutions and companies with more than 2,000 contributors. The GitHub repository had more than 96,000 stars on 8 November 2026 and is released under Apache 2.0. Release v0.33.0, published on 2 November 2026, contained 927 commits from 389 contributors. The project lists cash donations from a16z, Sequoia Capital, Skywork AI and ZhenFund, and compute from Alibaba Cloud, AMD, AWS, Google Cloud, IBM, Intel, NVIDIA, Red Hat and others. It supports more than 200 model architectures and runs on NVIDIA, AMD and Intel GPUs, CPUs, Google TPUs, AWS Neuron, Huawei Ascend, Apple Silicon and more through hardware plugins. vLLM is free; you pay only for the hardware you run it on.

Key Features

• PagedAttention memory management with continuous batching
• OpenAI-compatible API server, plus Anthropic Messages and gRPC
• Support for 200+ model architectures, including MoE and multimodal models
• Quantisation formats including FP8, INT4, GPTQ, AWQ and GGUF
• Tensor, pipeline, data and expert parallelism for multi-GPU serving
• Hardware plugins for NVIDIA, AMD, Intel, TPU, Ascend and Apple Silicon

Ideal Use Case

Platform and ML engineers who self-host open models in production and need to serve many users from each GPU, such as companies running Llama, Qwen or DeepSeek models behind an internal API for cost or data-control reasons. Researchers also use it for fast batch generation and reinforcement learning rollouts.

How vLLM differentiates

Ollama and LM Studio are built for one person running a model on a laptop; vLLM is built for serving many concurrent requests on data-centre hardware. Managed platforms such as Baseten and Fireworks AI sell hosted inference, often with engines like vLLM underneath, so choosing vLLM directly means taking on the operations work in exchange for control and no per-token markup.

FAQ

Q: What is vLLM? A: vLLM is an open-source library for fast large language model inference and serving. It runs models from Hugging Face behind an OpenAI-compatible API server, using PagedAttention to use GPU memory efficiently.

Q: Is vLLM free? A: Yes. vLLM is open source under the Apache 2.0 licence and free to use. You pay only for the GPUs or other hardware you run it on.

Q: What hardware does vLLM support? A: vLLM runs on NVIDIA, AMD and Intel GPUs and on x86, ARM and PowerPC CPUs, with plugins for Google TPUs, AWS Neuron, Intel Gaudi, Huawei Ascend, Apple Silicon and others.

Q: What is PagedAttention? A: PagedAttention is vLLM's method for storing the attention key-value cache in fixed-size pages, much like virtual memory, which reduces wasted GPU memory and lets more requests run at once.

Q: How is vLLM different from Ollama? A: Ollama is designed for running models on a single personal machine. vLLM is designed for production serving, with batching and multi-GPU parallelism to handle many concurrent users.

tl;dr

vLLM is a free, Apache 2.0 open-source engine for high-throughput LLM inference and serving, first developed at UC Berkeley. It offers an OpenAI-compatible server, PagedAttention memory management and support for 200+ model architectures across many kinds of hardware.