Running a Large Language Model is easy.

Running it fast, at scale, and without wasting expensive GPUs is much harder.

Imagine you deploy an open source LLM on an NVIDIA A100 GPU. Your API works, but users sometimes wait several seconds before they see the first word. When many users arrive at the same time, things get even slower.

Buying more GPUs is one solution.

But it is often not the best first solution.

There are many ways to make LLM inference much faster using the GPUs you already have.

In this article, I will explain the most useful techniques in simple language.

First, What Is LLM Inference?

When you send a prompt to an LLM and it generates an answer, that process is called inference.

For example:

1User:
2What is the capital of Bangladesh?
3
4LLM:
5The capital of Bangladesh is Dhaka.

Everything the model does after receiving the question is part of inference.

There are two important stages.

1. Prefill

First, the model needs to read and understand your input.

Imagine your prompt contains 5,000 tokens.

The model needs to process those 5,000 tokens before it can start generating the answer.

This is called prefill.

Long prompts usually mean more prefill work.

2. Decode

After processing the prompt, the model starts generating new tokens.

It might generate:

1The
2The capital
3The capital of
4The capital of Bangladesh
5The capital of Bangladesh is
6The capital of Bangladesh is Dhaka

This stage is called decode.

The model generates tokens one after another.

These two stages behave differently on a GPU. That becomes important when we start optimizing inference.

1. Use an Inference Engine

One of the biggest mistakes is serving a production LLM using a basic PyTorch or Hugging Face setup.

It works, but it may not use your GPU efficiently.

Instead, use an inference engine designed specifically for serving LLMs.

Two popular choices are:

vLLM

and

TensorRT LLM

These engines contain many optimizations for running LLMs efficiently on GPUs.

Think about it like this.

You bought a Ferrari.

But you are driving it through city traffic at 30 km/h.

The problem is not the car. The problem is how you are using it.

Your A100 is extremely powerful. A good inference engine helps you use much more of that power.

For many teams, vLLM is a great place to start because it is relatively easy to deploy and already contains several important optimizations.

2. Use Continuous Batching

Suppose four users send requests to your LLM.

1User A
2User B
3User C
4User D

A simple server might process them inefficiently.

It may wait for one group of requests to finish before starting another group.

That means some GPU capacity can sit unused.

Continuous batching solves this problem.

The inference server keeps adding new requests whenever GPU capacity becomes available.

Imagine a restaurant.

Without continuous batching, the restaurant might say:

We will not seat anyone new until everyone currently eating has finished.

That would waste many empty tables.

With continuous batching, whenever a table becomes free, another customer can immediately use it.

This allows the GPU to process many requests efficiently.

If your LLM receives many requests at the same time, continuous batching can make a very large difference.

3. Use Paged Attention

LLMs need memory while generating text.

One important part of this memory is called the KV cache.

The KV cache can become very large, especially when you have:

• long conversations
• large prompts
• many users
• large context windows

Traditional memory management can waste GPU memory.

vLLM introduced an approach called PagedAttention.

The basic idea is similar to how operating systems manage computer memory.

Instead of requiring one large continuous area of memory for every request, memory can be divided into smaller blocks.

This makes memory usage much more efficient.

More efficient memory means you can usually serve more requests using the same GPU.

4. Use Prefix Caching

This is especially useful for AI agents.

Imagine every request starts with the same system prompt:

1You are an AI assistant for our company.
2
3Follow these 50 instructions.
4
5Here are 30 available tools.
6
7Here are their schemas.
8
9Here are several examples.

Imagine this system prompt contains 6,000 tokens.

Then the user asks:

1What were my recent orders?

Another user asks:

1Cancel my latest order.

The first 6,000 tokens might be exactly the same.

Without caching, the model processes those same tokens again and again.

That is wasted computation.

Prefix caching allows the inference engine to reuse previous computation for repeated prefixes.

Instead of:

1Process 6,000 tokens
2Process 6,000 tokens
3Process 6,000 tokens
4Process 6,000 tokens

you can reuse some of the work.

This can significantly improve Time to First Token, especially when your application has large repeated system prompts.

For agent systems with large tool definitions, prefix caching should be one of the first things you test.

5. Quantize the Model

LLMs contain billions of numbers called parameters.

Those numbers need to be stored in GPU memory and accessed during inference.

For example, a model might normally use:

1FP16

You may be able to represent the model using lower precision formats such as:

1INT8
2
3or
4
54 bit

This process is called quantization.

A simple analogy is image compression.

A high quality photo might require 20 MB.

A compressed version might require only 4 MB while still looking almost identical.

Quantization does something conceptually similar with model weights.

Smaller weights mean:

• less GPU memory
• less memory movement
• potentially faster inference
• more room for KV cache
• potentially more concurrent users

But there is a tradeoff.

Aggressive quantization can reduce model quality.

So always test the model after quantization.

Do not only ask:

Is it faster?

Also ask:

Is the model still good enough?

6. Use Speculative Decoding

LLMs normally generate tokens one after another.

Imagine the model wants to generate:

1Machine learning is changing software development.

The process is roughly:

 1Machine
 2↓
 3learning
 4↓
 5is
 6↓
 7changing
 8↓
 9software
10↓
11development

Every step requires computation.

Speculative decoding tries to speed this up.

It uses two models:

• a small, fast model that writes a short draft
• the large model that decides which tokens are actually used

Think of a senior engineer working with a junior engineer.

Instead of the senior engineer writing everything from scratch, the junior engineer prepares a draft.

The senior engineer checks the draft and keeps the parts that are right.

For example, the small model might propose these next four tokens:

1learning is changing software

The large model checks all four tokens in one pass.

There are two possible outcomes:

1. If it agrees with all four tokens, the system can use all four at once.
2. If it agrees with only learning is, it keeps those tokens. The large model supplies the next token itself, then a new round begins.

Without speculative decoding, the large model would need a separate expensive step for each token. With a good draft, one large-model check can move the response forward by several tokens.

The large model still controls the final answer. Speculative decoding does not make it smarter or lower its quality. It only avoids waiting for the large model to generate every token one by one.

Speculative decoding can reduce the time required to generate output.

However, it does not improve every workload.

It works best when the small model often makes the same predictions as the large model. If the draft is often wrong, the large model rejects more tokens and the speedup becomes small.

You should benchmark it with your actual traffic.

7. Be Careful With Tensor Parallelism

Suppose you have four A100 GPUs.

It may seem obvious that:

14 GPUs = 4 times faster

Unfortunately, it does not always work like that.

You can divide one model across several GPUs.

This is called tensor parallelism.

For example:

1One model layer, split across four GPUs
2
3[ Part 1 ]  [ Part 2 ]  [ Part 3 ]  [ Part 4 ]
4    ↓           ↓           ↓           ↓
5  GPU 1       GPU 2       GPU 3       GPU 4

This is useful when the model is too large for one GPU.

But now those GPUs need to communicate with each other.

Communication takes time.

If your model already fits comfortably on one A100, keeping one full model copy on each GPU can sometimes provide better total throughput than splitting one model across all four GPUs.

1Four separate full model copies
2
3GPU 1 → Full model
4GPU 2 → Full model
5GPU 3 → Full model
6GPU 4 → Full model

instead of:

1One model split across four GPUs
2
3GPU 1 → Model part 1
4GPU 2 → Model part 2
5GPU 3 → Model part 3
6GPU 4 → Model part 4

There is no universal answer.

Benchmark different configurations.

Here, TP means tensor parallelism. The number tells you how many GPUs run one model together.

For example:

1TP = 1  → One GPU runs the model
2TP = 2  → Two GPUs run the model together
3TP = 4  → Four GPUs run the model together

Then compare latency and throughput.

8. Use Chunked Prefill for Large Prompts

Imagine one user sends a 30,000 token prompt.

At the same time, several other users are already generating answers.

Processing that huge prompt can consume a lot of GPU compute.

This can make other requests slower.

Chunked prefill breaks large prompts into smaller pieces.

Instead of processing:

130,000 tokens

as one huge piece, the inference engine can process smaller chunks.

It can then mix this work with token generation for other requests.

This is especially useful when your system has both long prompts and interactive requests.

9. Do Not Use Huge Context Windows Unless You Need Them

Suppose your model supports 128,000 tokens.

That does not mean every request needs 128,000 tokens.

Large context windows can increase memory requirements significantly.

Look at your real production traffic.

Maybe you discover:

150% of requests < 2,000 tokens
2
390% of requests < 8,000 tokens
4
599% of requests < 20,000 tokens

That information should influence how you configure your inference server.

Do not optimize your entire infrastructure around a theoretical maximum that almost nobody uses.

10. Optimize Your Prompts

Sometimes the easiest inference optimization does not involve CUDA, GPUs, or inference engines.

Simply send fewer tokens.

Imagine your system prompt contains 12,000 tokens.

After reviewing it, you discover that 4,000 tokens are unnecessary.

Now every request has 8,000 input tokens instead of 12,000.

That is 4,000 fewer tokens for the model to process.

If you process millions of requests, this becomes a huge amount of saved computation.

This is particularly important for AI agents because tool definitions, examples, memory, retrieved documents, and instructions can make prompts very large.

Before buying more GPUs, inspect what you are actually sending to the model.

11. Optimize Routing

Imagine you have four replicas of your model.

1Request
2   ↓
3Load Balancer
4   ↓
5┌─────┬─────┬─────┬─────┐
6│GPU 1│GPU 2│GPU 3│GPU 4│
7└─────┴─────┴─────┴─────┘

A normal load balancer might simply send requests to whichever replica looks available.

But LLM inference has another consideration: cache locality.

Suppose GPU 1 already cached a large system prompt.

A new request uses exactly the same system prompt.

Sending that request to GPU 1 may allow you to reuse cached computation.

Sending it to GPU 3 may require processing everything again.

At larger scale, intelligent routing can therefore improve both latency and GPU efficiency.

What Should We Measure?

This is extremely important.

Do not simply say:

The model feels faster.

Measure it.

For LLM inference, I would track at least these metrics.

Time to First Token

How long does the user wait before seeing the first generated token?

For interactive applications, this is extremely important.

Time Per Output Token

After generation starts, how quickly do new tokens appear?

Tokens Per Second

How many tokens can your system process or generate every second?

Requests Per Second

How many requests can the system handle?

GPU Utilization

Are your expensive A100 GPUs actually busy?

If you are paying for A100s while GPU utilization stays at 20 percent, you probably have an optimization opportunity.

KV Cache Utilization

How much of your available KV cache capacity are you actually using?

P50, P95, and P99 Latency

Average latency alone can hide serious problems.

Your average request might take two seconds while some users wait ten seconds.

Percentiles help you find these slow requests.

Putting Everything Together

A strong production setup might look something like this:

 1                    User Requests
 2                          ↓
 3                   Vertex AI Endpoint
 4                          ↓
 5                    Smart Routing
 6                          ↓
 7                 vLLM / TensorRT LLM
 8                          ↓
 9              Continuous Batching
10                          ↓
11                   Paged Attention
12                          ↓
13                   Prefix Caching
14                          ↓
15                   Chunked Prefill
16                          ↓
17              Quantized Model Weights
18                          ↓
19                  Speculative Decoding
20                          ↓
21                      A100 GPUs

You do not need to implement everything at once.

Start with the changes that are most likely to matter.

A practical order would be:

First: Use a proper inference engine such as vLLM or TensorRT LLM.

Second: Enable continuous batching and efficient KV cache management.

Third: Enable prefix caching if your prompts share large common prefixes.

Fourth: test quantization.

Fifth: benchmark different tensor parallel configurations.

Sixth: test speculative decoding.

Seventh: optimize routing when you have many replicas.

And throughout the entire process, measure everything.

The Most Important Lesson

Making LLM inference faster is not simply about buying faster GPUs.

A powerful GPU running an inefficient inference stack can still perform poorly.

The goal is to make better use of the hardware you already have.

Sometimes a software optimization can give you more improvement than adding another expensive GPU.

So before asking:

Should we add more A100s?

Ask:

Are we actually using our current A100s efficiently?

That question can save both milliseconds and money.