How vLLM Works for Serving Large Language Models

Aug 9, 2026 · 22 min read · artificial intelligence large language models software engineering system design ·

Introduction

Serving a large language model, or LLM, is different from calling an ordinary function. A normal function usually receives an input, does some work, and returns one result. An LLM server keeps a large model loaded in memory, accepts many requests, produces one token at a time, and sends partial results back to users while the answer is still being created.

This creates a difficult engineering problem. The model weights use a lot of memory on a graphics processing unit, or GPU. Each active request also needs memory that grows as the prompt and answer grow. The server must use the GPU well, keep response time reasonable, avoid memory errors, and control cost.

vLLM is an inference and serving engine designed for this problem. It provides a Hypertext Transfer Protocol, or HTTP, server, an application programming interface, or API, layer, a request scheduler, a key value cache manager, model execution code, and many performance features in one system. Its main ideas are continuous batching and efficient management of the key value, or KV, cache.

The goal of this article is to explain how vLLM works without requiring a data science background. The focus is on the engineering decisions behind the system and on how vLLM compares with other ways to serve an LLM.

Serving tools change quickly. This article describes the architecture and product tradeoffs documented by the projects in August 2026. Always check the versioned documentation before using a command or setting in production.

The short answer

vLLM keeps model weights on the GPU and manages many active requests in a shared execution loop. The scheduler decides which requests should run at each model step. The KV cache manager stores the attention state for each request in small blocks instead of reserving one large continuous area for every request. The model runner then executes the selected work on the GPU and streams new tokens back to clients.

This design matters because LLM requests are not equally long and they do not finish at the same time. A request can enter while other requests are already generating text. vLLM can add it to later steps, remove requests that finish, and reuse memory as requests leave.

The result is a serving system that can often process more concurrent work with the same hardware. The exact benefit depends on the model, the GPU, the prompt length, the answer length, the request pattern, and the chosen configuration.

What happens inside one LLM request

Before discussing vLLM, it helps to understand one request.

1. The server tokenizes the input

The user sends text. The tokenizer converts that text into tokens. A token is a small unit of text. It may be a whole word, part of a word, punctuation, or a special symbol.

The model works with token identifiers, not directly with human readable text.

2. The model reads the prompt

The model processes the input tokens in a stage often called prefill. During this stage, it computes the internal state needed to continue the sequence. It also creates two kinds of internal data for each attention layer. They are called keys and values. Keys help attention decide which earlier tokens matter. Values carry the information that attention reads from those tokens.

The server keeps this internal data in temporary working memory called the key value cache, or KV cache. It is not a text summary and it is not the model's permanent knowledge. It is numeric state that helps the model use the prompt and the tokens it has already generated.

The cache belongs to one request. It grows as the request grows. A long prompt creates more cache at the start, and every generated token adds a little more. When the request finishes, the cache can be released and used by another request.

3. The model generates the answer

For the common autoregressive design, the next token depends on the tokens that came before it. The model therefore generates the answer one token at a time. This stage is often called decode.

For every new token, the model reads the existing KV cache and adds the new key and value data. Because the earlier data is already stored, the model does not need to calculate it again for every output token.

4. The server sends the result

The server can wait until the full answer is ready. For chat products, it usually streams tokens as they are produced. This makes the product feel faster because the user sees the beginning of the answer while the rest is still being generated.

Why ordinary serving is difficult

There are four main problems.

First, the model weights are large and mostly stay in GPU memory. The server needs enough memory for the weights, temporary workspaces, and the KV cache of active requests.

Second, the KV cache grows during generation. A short request may need a small amount of cache. A long request may need much more. The server does not know the final answer length when the request starts.

Third, requests have different shapes. One request may have a short prompt and another may contain a long document. One answer may end after ten tokens while another may continue for thousands of tokens.

Fourth, the GPU is most useful when it has enough work to process together. The server wants large batches, but waiting too long to form a batch increases user latency.

This is a scheduling problem, a memory problem, and a product experience problem at the same time.

The simple approach and its limits

The simplest server loads a model and calls a generation function for each request.

1for request in requests:
2    answer = model.generate(request.prompt)
3    send_answer(answer)

This is useful for experiments and small workloads. It is easy to understand and it gives the application full control over the model code.

It becomes less attractive when many users send requests at the same time. The application must then build its own queue, decide when to batch requests, stream output, limit memory use, cancel work, expose metrics, and recover from errors.

The application may also use a fixed batch. A fixed batch waits for a group of requests and processes them together. This can waste work when the requests have different lengths. A short request may finish while the rest of the batch continues. The server may also need padding so that different sequences fit one tensor shape.

Some of these problems can be solved in application code. A dedicated serving engine solves them closer to the model execution path, where it can see token counts, cache blocks, and GPU capacity.

Continuous batching

Continuous batching is one of the most important ideas in modern LLM serving.

In a fixed batch, the server forms a group, runs the group, and often waits for the group to finish. In continuous batching, the active group can change at every model step.

Imagine three requests.

Request A is already generating an answer.
Request B arrives with a long prompt.
Request C arrives while A and B are still active.

The scheduler can keep A in the running group, process B in smaller prompt pieces, and admit C when there is enough capacity. When A finishes, its place and memory can be used by another request.

This is sometimes described as iteration level batching because the batch is rebuilt at each generation iteration. The server does not need to wait for every request in the group to finish before making progress.

Continuous batching does not mean that every request runs at exactly the same time. The scheduler still has to respect memory, token budgets, model rules, priorities, and maximum sequence limits. It means that the group is flexible.

The main parts of vLLM

The current vLLM architecture separates the serving path into clear responsibilities.

 1Client
 2  ↓
 3API server
 4  ↓
 5Engine core
 6  ↓
 7Scheduler and KV cache manager
 8  ↓
 9GPU worker processes
10  ↓
11Model runner

The API server

The API server receives HTTP requests, validates input, tokenizes text, loads multimodal input when needed, and streams results back to clients. vLLM provides an API that follows important parts of the OpenAI Completions and Chat API shapes. This lets an application use a familiar client while running an open model on its own infrastructure.

The API layer is not the same as a complete product API. A real product still needs authentication, authorization, request limits, usage tracking, and business rules around the model.

The engine core

The engine core runs the scheduler, manages the KV cache, and coordinates model execution. It repeatedly looks at waiting and running requests, decides what can fit in the next step, and sends that work to the GPU workers.

The GPU workers

Each GPU worker loads its portion of the model weights and runs the model forward pass on its assigned device. For a large model, vLLM can place one model across several GPUs through tensor parallelism, pipeline parallelism, expert parallelism, or other supported strategies. Data parallelism creates multiple model execution ranks that handle different requests, and can be combined with other forms of parallelism.

The best strategy depends on model size, GPU memory, the connection between GPUs, and whether the deployment uses one machine or several machines.

The model runner

The model runner prepares input tensors and calls the model. Sampling code then selects the next token, and output processing turns token identifiers back into text. The model runner can use optimized kernels and graph capture when the model and hardware support them.

The important design point is separation. The API server handles network work. The engine core makes scheduling decisions. The workers focus on GPU execution. This keeps the hot execution loop focused on the work that affects throughput and latency.

How vLLM handles the KV cache

The KV cache is central to serving performance.

The problem with one large memory area

Suppose a server reserves one large continuous memory area for every request. It has to guess how much space the request will need. If it reserves too much, memory sits unused. If it reserves too little, the request may need to move or grow into another area. Many requests with different lengths can also leave gaps between used areas.

This is similar to the memory allocation problem that operating systems solve for running programs. The details are different, but the design question is familiar. How can a logical sequence receive more memory over time without requiring one large uninterrupted region?

PagedAttention

vLLM uses PagedAttention to store KV data in fixed size blocks. A request has a logical sequence of tokens. The KV cache manager maps that logical sequence to physical blocks in GPU memory.

For example, the logical sequence can look like this.

1Tokens 1 to 16     → physical block 4
2Tokens 17 to 32    → physical block 9
3Tokens 33 to 48    → physical block 2

The physical blocks do not need to be next to each other. An internal block table tells the attention kernel where to find the data. When the request grows, the cache manager obtains another free block. When the request finishes, its blocks return to the pool.

This has several benefits.

The server does not need to reserve a large continuous region for each request.
Different request lengths create less wasted space.
Finished requests release blocks that other requests can use.
Shared prompt prefixes can reuse cache blocks when the configuration and request pattern allow it.

The original PagedAttention paper reports near zero waste in KV cache memory under its design and benchmark conditions. It also reports higher serving throughput than the systems it compared with at that time. Those results are useful for understanding the idea, but they should not be treated as a promise for every current model and GPU.

PagedAttention adds a block lookup step to attention. That step has a cost. The reason the design is useful is that the memory savings can allow more useful requests to fit at the same time.

Automatic prefix caching

Automatic prefix caching stores the KV cache for an existing prompt prefix. If a later request begins with the same token prefix, vLLM can reuse that cached work and process only the part that is new.

This is useful for several workloads.

Many users ask questions about the same long document.
A chat session sends the same conversation history again with a new user message.
An agent uses a stable system instruction and tool description across many calls.

Prefix caching reduces work in the prefill stage. It does not reduce the time needed to generate new output tokens. It also provides little benefit when requests do not share a prefix or when generation is much longer than the prompt.

If there are several vLLM replicas, each replica normally has its own memory cache. A product that wants strong cache reuse may route related requests to the same replica. This has to be balanced with load distribution and user isolation.

Chunked prefill

Prefill can be compute heavy because the model reads many prompt tokens at once. Decode is different because the model usually processes a small amount of new work for many active requests.

If a very long prompt occupies the GPU for a long time, users who are already receiving answers may see a pause between output tokens. Chunked prefill lets vLLM split a long prompt into smaller pieces. The scheduler can place those pieces into later iterations with decode work.

This is a good example of engineering and product design meeting each other. The system is trying to keep the GPU busy, but it is also protecting the smoothness of streaming output.

The right balance depends on the workload. Small chunks can improve responsiveness but add scheduling overhead. Large chunks can improve prompt processing efficiency but may interrupt active generations for longer.

Other optimizations in vLLM

PagedAttention and continuous batching are not the whole system. Depending on the model and hardware, vLLM can also use several other techniques.

Quantization stores model weights or cache data with fewer bits. This can reduce memory use and may improve speed, but it can change model quality and is not equally supported for every model and device.
Optimized attention and matrix multiplication kernels reduce the work needed for common operations. The best kernel depends on the model architecture, data type, and GPU.
Compute Unified Device Architecture, or CUDA, graphs can reduce repeated launch overhead on supported NVIDIA workloads by recording and replaying GPU work with suitable shapes.
Speculative decoding uses a smaller draft model or another prediction method to propose tokens. The main model checks those proposals. When enough proposals are accepted, the system can reduce the number of expensive steps. The benefit depends strongly on how often the proposals are accepted.
Parallelism splits model work across GPUs or places several copies of a model on different GPUs. This helps with model size or request capacity, but communication between GPUs also costs time.
vLLM can support more advanced layouts in which prompt processing and token generation use different worker groups. These designs can help when the two stages have very different resource needs, but they increase deployment complexity.

The important lesson is that vLLM is a collection of coordinated optimizations. One feature rarely explains the performance of a complete deployment.

The main alternatives

There is no single best serving approach. The right choice depends on the product, the hardware, the model, and the amount of control the team needs.

Direct use of Transformers

Hugging Face Transformers gives developers model implementations and a Python interface. A team can load a model, call its generation method, and put an API around it.

This is a strong choice for experiments, evaluation, custom model changes, offline jobs, and low traffic services. It is also a good way to understand model behavior before adding a serving engine.

The cost is that the application team owns more of the serving work. The team must design concurrency control, batching, output streaming, cache management, request cancellation, metrics, and overload behavior. Transformers can be combined with other serving systems, so using Transformers for model code does not prevent later migration to a specialized runtime.

vLLM

vLLM is a strong general choice for GPU based online serving when the team wants a ready server, broad model support, continuous batching, streaming, and good memory management.

It is especially attractive when the product needs an API shaped like a common chat completion API and the team wants to run open models on its own machines.

vLLM is still an engine, not a complete cloud product. Authentication, rate limits, billing, autoscaling, model routing, and tenant policy remain outside the engine.

SGLang

SGLang is another high performance runtime for language and multimodal models. Its design puts strong emphasis on structured language model programs and RadixAttention, which is a prefix caching approach based on a radix tree.

SGLang can be a good fit when an application has repeated prompt prefixes, structured output requirements, complex agent workflows, or model specific performance needs. It also supports continuous batching, chunked prefill, and several parallel execution strategies.

The feature sets of SGLang and vLLM overlap in many areas. The correct choice should come from a benchmark using the real model, prompt lengths, output lengths, concurrency, and hardware.

TensorRT LLM

TensorRT LLM is NVIDIA software for optimized language model inference. It provides features such as in flight batching, paged KV cache, chunked context, quantization, speculative sampling, and multi GPU execution. Its current stack also includes a PyTorch backend in addition to highly optimized NVIDIA execution paths.

TensorRT LLM is attractive when a company uses NVIDIA GPUs, needs very high performance, and has a team that can work close to the hardware. It can require more attention to model support, data types, build choices, and hardware specific tuning than a general serving engine.

Text Generation Inference

Text Generation Inference, from Hugging Face, provides a production server with streaming, continuous batching, tensor parallelism, metrics, and quantization support for many models.

Its current official documentation says that the project is in maintenance mode. The same documentation recommends vLLM and SGLang, as well as local runtimes such as llama.cpp, for new optimized inference work. TGI can still matter when a company already has a working deployment, but this maintenance status is an important product and maintenance consideration for a new project.

llama.cpp

llama.cpp is a lightweight C++ runtime focused on local and edge inference. It supports quantized model files, central processing unit, or CPU, execution, Apple silicon, several GPU backends, and CPU plus GPU execution for some models.

Its llama server provides API compatible routes, parallel decoding, continuous batching, monitoring endpoints, and a simple local user interface.

llama.cpp is a strong fit for laptops, private local applications, edge devices, and deployments where a small native runtime matters. It is not a direct replacement for vLLM in every large GPU cluster. The two systems optimize for different deployment shapes.

Managed inference APIs

The last approach is to avoid running the model server yourself. A managed inference provider runs the GPUs, keeps the model available, and exposes an API.

This can be the fastest way to validate a product. The team does not need to manage GPU drivers, model startup, memory limits, or replica scaling.

The tradeoffs include less control over model versions and hardware, network latency, data handling requirements, provider availability, vendor dependence, and a cost model that may be harder to predict at high usage.

Runtime, deployment, and product are different layers

Many serving discussions mix together tools that solve different problems.

The model runtime executes the model. Examples include Transformers, vLLM, SGLang, TensorRT LLM, and llama.cpp.

The deployment layer manages replicas, machines, health checks, rollout, scaling, and placement. Examples include Kubernetes, Ray Serve, KServe, NVIDIA Triton, and other platform tools.

The product layer handles authentication, user accounts, prompt construction, retrieval, tools, safety checks, quotas, billing, and user experience.

One product can use all three layers.

1Product API
2  ↓
3Deployment platform
4  ↓
5vLLM or another runtime
6  ↓
7Model and GPU hardware

Choosing vLLM does not remove the need for deployment design. Choosing Kubernetes does not decide which runtime should execute the model. Keeping the layers separate makes architecture decisions clearer.

How to choose an approach

Start with the product workload.

For a local tool, a laptop application, or a small private service, start with llama.cpp or direct use of Transformers. The lower setup cost may matter more than maximum throughput.
For a GPU backed chat API with several concurrent users, start with vLLM. It provides a useful default serving path and lets the team adopt more advanced features over time.
For an agent or retrieval application with long repeated prefixes, benchmark vLLM against SGLang. Prefix reuse may matter more than a general benchmark score.
For a large NVIDIA deployment where every millisecond and every GPU hour matters, evaluate TensorRT LLM as well as vLLM. Include the cost of engineering time and model upgrades in the decision.
For a new project that wants a Hugging Face serving engine, treat TGI maintenance mode as a serious factor. It may still be suitable for an existing system, but a new system should also evaluate the actively developed alternatives.
For the fastest product launch with little infrastructure work, use a managed inference API. Revisit self hosting when volume, privacy, or model control becomes important.

Do not choose based only on tokens per second from a public chart. Run a test with the real model and the real request pattern.

What to measure in production

A good serving system is not defined by one speed number.

Time to first token

Time to first token, or TTFT, measures how long the user waits before the first output token appears. It strongly affects how fast a chat product feels. For server analysis, keep queue time separate from model processing time. vLLM per request metrics define TTFT from the time a request is scheduled, and report queue time as a separate value.

Inter token latency

Inter token latency, or ITL, measures the time between output tokens. High ITL makes streaming feel slow or uneven. A long prompt can create an ITL spike if the server lets prompt processing block active generations.

End to end latency

End to end latency measures the time until the full answer is complete. This matters for non streaming applications, background jobs, and service level objectives.

Throughput

Measure output tokens per second across the whole service, not only one request. Also measure how throughput changes as concurrency increases. A system that is fast for one request may be expensive or slow under real traffic.

Queue and capacity

Track waiting requests, running requests, request rejection, request cancellation, KV cache capacity, and GPU memory use. A growing queue is a product signal. It may mean that users need a faster model, a shorter response limit, more replicas, or a different service policy.

Quality and cost

Measure answer quality after quantization or model changes. Track the cost per request, cost per generated token, and cost of idle GPU capacity. A small speed improvement is not useful if it reduces answer quality or increases operational work more than it saves.

vLLM exposes production metrics through its metrics endpoint. These include request latency, time to first token, inter token latency, queue time, request counts, and KV cache related measurements.

Product design choices around vLLM

The runtime is only one part of the user experience.

Set a maximum input size and a maximum output size. Without limits, one request can consume a large part of the shared cache.
Use streaming for interactive chat and code completion. Use non streaming calls for batch work when the client does not need partial output.
Separate interactive traffic from long batch jobs when they have different latency goals. A single shared queue can make the user experience unpredictable.
Apply rate limits and concurrency limits before the GPU becomes overloaded. Returning a clear capacity error is better than allowing every request to wait for an unbounded time.
Give users a cancellation path. If a user closes the page, the product should stop work when possible and release its resources.
Warm up new replicas before sending normal traffic. Model loading and graph capture can make the first requests slower.
Test model, tokenizer, chat template, sampling settings, and runtime version together. A serving upgrade can change output details even when the model file is unchanged.
Treat prefix caching as a workload feature, not as a free global cache. Decide how it interacts with user isolation, memory limits, and replica routing.
Keep a simple fallback path. A smaller model, a managed API, or a queue for batch work can help the product remain useful during GPU capacity problems.

Common misunderstandings

PagedAttention is not a new model architecture

PagedAttention is a way to organize and read the KV cache during attention. It does not change the knowledge stored in the model weights.

Higher throughput does not always mean lower latency

Batching more requests can improve total work per GPU step. It can also increase queueing or delay a request if the server is overloaded. Product teams should measure both throughput and tail latency.

Prefix caching is not full response caching

Prefix caching reuses internal work for a shared input prefix. The model still needs to process the new part of the prompt and generate the new answer.

Quantization is not free

Quantization can reduce memory and cost. It can also change output quality, supported features, and performance. Test it with the tasks that matter to users.

vLLM is not a complete cloud platform

vLLM serves models. It does not automatically provide the full set of product features such as identity, billing, tenant management, or business level observability.

Final view

The main lesson is simple. LLM serving is a resource management problem hidden inside a user facing product.

The model generates text one token at a time. The prompt and the answer have different lengths. The KV cache grows during the request. GPUs prefer shared work. Users want the first token quickly and the rest of the answer smoothly.

vLLM addresses these pressures with a flexible scheduler, continuous batching, block based KV cache management through PagedAttention, prefix caching, optimized GPU execution, and support for several scaling strategies.

It is a strong default for many self hosted GPU services, but it is not the only good choice. Transformers is simpler for experiments and custom work. llama.cpp is excellent for local and edge use. SGLang is compelling for structured and cache heavy workloads. TensorRT LLM is worth serious evaluation for tuned NVIDIA deployments. Managed APIs are often best for fast product validation.

The best engineering decision comes from measuring the real product workload. Start with the user experience, understand the memory and scheduling behavior, then choose the smallest serving stack that can meet the quality, latency, reliability, and cost goals.