Training a large language model usually happens in more than one stage.

The first stage is pretraining.

This is where the model learns language, facts, patterns, coding concepts, reasoning patterns, and many other things from a very large amount of data.

But a pretrained model is not automatically a good assistant.

It might know a lot, but it may still struggle to follow instructions, answer in the right format, understand human preferences, or behave the way we want.

That is where post training comes in.

Post training takes a pretrained model and teaches it how we actually want it to behave.

Some of the most common post training techniques are:

1. Supervised fine tuning
2. Preference training
3. Reinforcement learning
4. Knowledge distillation

In this blog, we will understand what each technique does, why we need it, and how they fit together.

1. Pretraining vs Post Training

Before looking at individual techniques, we first need to understand the difference between pretraining and post training.

Pretraining

During pretraining, an LLM learns by reading a huge amount of text.

For example, imagine the model sees a sentence like:

1The capital of France is Paris.

Later, it may see:

1Paris is the largest city in France.

And millions of other examples involving countries, cities, history, programming, science, conversations, and many other topics.

The model learns by repeatedly trying to predict what comes next.

A simplified example would be:

1The capital of France is _____

The model tries to predict:

1Paris

It does this billions or even trillions of times across a massive dataset.

Over time, the model learns patterns in language and information.

The result is called a base model.

Conceptually:

1Huge amount of text data
2
3        ↓
4
5     Pretraining
6
7        ↓
8
9     Base LLM

The base model may know a lot, but knowledge alone does not make it a good assistant.

Post Training

Post training happens after pretraining.

Instead of teaching the model general language knowledge, we now focus on improving its behavior.

For example, we may want the model to:

 1Follow instructions
 2
 3Give useful answers
 4
 5Write in a certain style
 6
 7Solve reasoning problems
 8
 9Avoid unwanted responses
10
11Use tools correctly
12
13Follow human preferences

So the overall process looks like:

 1Large training dataset
 2
 3        ↓
 4
 5     Pretraining
 6
 7        ↓
 8
 9      Base LLM
10
11        ↓
12
13    Post Training
14
15        ↓
16
17 Assistant Model

Chat based models that people interact with are usually the result of both stages.

A simple way to remember the difference is:

Pretraining teaches the model what it knows.

Post training teaches the model how we want it to use what it knows.

Now let us look at the main post training techniques.

2. Supervised Fine Tuning

Supervised fine tuning, often called SFT, is probably the easiest post training technique to understand.

The idea is simple.

We give the model examples of good answers and train it to produce similar answers.

Suppose our base model receives this question:

1User:
2Explain photosynthesis in simple language.

We prepare a high quality answer:

1Assistant:
2Photosynthesis is the process plants use to make food.
3
4Plants take sunlight, water, and carbon dioxide
5and use them to produce energy in the form of sugar.

Our training dataset contains many examples like this:

1Instruction → Good response
2Instruction → Good response
3Instruction → Good response
4Instruction → Good response

Then we train the base model on those examples.

Conceptually:

 1Base Model
 2
 3    +
 4
 5High Quality Examples
 6
 7    ↓
 8
 9Supervised Fine Tuning
10
11    ↓
12
13Instruction Following Model

The model starts learning what a good answer should look like.

What Can SFT Teach?

SFT can teach the model many useful behaviors.

For example:

 1How to answer questions
 2
 3How to follow instructions
 4
 5How to generate JSON
 6
 7How to write code
 8
 9How to summarize documents
10
11How to use a particular tone
12
13How to solve domain specific tasks

Imagine that we are building an LLM for customer support.

We could prepare examples such as:

1Customer question:
2How can I reset my password?
3
4Good answer:
5Go to the login page and select "Forgot password."
6Then follow the instructions sent to your email.

After seeing thousands of examples like this, the model becomes much better at answering customer support questions.

Where Does The Training Data Come From?

SFT data can come from several places.

For example:

1Human written answers
2
3Existing company datasets
4
5Expert generated answers
6
7Synthetic answers generated by another LLM
8
9Carefully cleaned public datasets

The quality of this data matters a lot.

If the training examples are poor, the model will learn poor behavior.

This gives us an important rule:

Good fine tuning starts with good training data.

3. Preference Training

SFT teaches the model what a good answer looks like.

But there is another problem.

Sometimes there are several reasonable answers to the same question.

Consider this prompt:

1Explain machine learning to someone with no technical background.

Imagine the model produces two answers.

Response A

1Machine learning uses statistical optimization methods
2to estimate functions from observed data distributions.

Response B

1Machine learning is a way of teaching computers
2to learn patterns from examples instead of writing
3every rule manually.

Both responses might contain correct information.

But most people would probably prefer Response B.

How do we teach the model that?

This is where preference training comes in.

Instead of simply saying:

1This is the correct answer.

we provide information like:

1For this question:
2
3Response B is better than Response A.

The model learns which kinds of answers humans prefer.

Conceptually:

 1Prompt
 2
 3  ↓
 4
 5Multiple Responses
 6
 7  ↓
 8
 9Which Response Is Better?
10
11  ↓
12
13Preference Training
14
15  ↓
16
17Model Learns Human Preferences

What Kind Of Preferences Can We Teach?

Preference data can represent many things.

For example:

1Clear answer > confusing answer
2
3Correct answer > incorrect answer
4
5Concise answer > unnecessarily long answer
6
7Helpful answer > unhelpful answer
8
9Instruction following > ignoring instructions

The interesting part is that preference training does not always tell the model exactly what to write.

Instead, it teaches the model to understand which response is better.

Where Do Preferences Come From?

Traditionally, humans compare model responses.

For example, an evaluator sees:

1Prompt
2
3Response A
4
5Response B

Then selects the better response.

But using humans for millions of comparisons can be expensive.

Today, stronger LLMs are also commonly used to judge responses and generate preference datasets.

This is sometimes called AI feedback.

4. Reinforcement Learning

Preference training tells us which outputs are better.

Reinforcement learning takes this idea further.

Instead of only showing the model examples, we allow the model to generate answers and then give it a reward based on how good those answers are.

Think about training a dog.

When the dog performs the correct action, it receives a reward.

Over time, the dog learns which actions lead to rewards.

The basic idea in reinforcement learning is similar.

For an LLM:

 1Prompt
 2
 3  ↓
 4
 5Model Generates Response
 6
 7  ↓
 8
 9Response Is Evaluated
10
11  ↓
12
13Reward
14
15  ↓
16
17Model Updates Its Behavior

If an answer receives a high reward, the training process encourages similar behavior.

If the answer receives a low reward, the model learns to avoid that behavior.

RLHF

One of the best known approaches is Reinforcement Learning from Human Feedback, commonly called RLHF.

A simplified RLHF pipeline looks like this:

 1Humans compare responses
 2
 3        ↓
 4
 5Build preference data
 6
 7        ↓
 8
 9Train a reward model
10
11        ↓
12
13LLM generates responses
14
15        ↓
16
17Reward model scores responses
18
19        ↓
20
21Reinforcement learning
22
23        ↓
24
25Improved LLM

The reward model tries to predict what humans would prefer.

The LLM then learns to generate responses that receive better reward scores.

Reinforcement Learning For Reasoning

Reinforcement learning has also become important for reasoning models.

Imagine that we give the model a math problem.

115 × 12 = ?

The model generates a solution.

If the final answer is correct, we can reward it.

If the answer is incorrect, it gets a lower reward.

This works particularly well for tasks where answers can be checked automatically.

Examples include:

1Mathematics
2
3Coding
4
5Logic problems
6
7Games
8
9Structured reasoning

For programming tasks, for example, we can run the generated code against test cases.

If all tests pass, the model receives a strong positive signal.

This reduces the need for a human to manually judge every answer.

5. Knowledge Distillation

Now we come to another powerful post training technique.

Knowledge distillation means using a stronger model to teach a smaller model.

Suppose we have:

1Teacher Model
2
370 billion parameters
4
5Very capable
6
7Expensive

and:

1Student Model
2
37 billion parameters
4
5Less capable
6
7Much cheaper

We ask the teacher model to solve a large number of tasks.

For example:

1Prompt
2
3↓
4
5Teacher Model
6
7↓
8
9High Quality Answer

We collect these answers and use them to train the student.

Conceptually:

 1Prompts
 2
 3   ↓
 4
 5Large Teacher Model
 6
 7   ↓
 8
 9Teacher Responses
10
11   ↓
12
13Training Dataset
14
15   ↓
16
17Small Student Model

The student learns patterns from the teacher's responses.

What Can We Transfer?

The teacher can help transfer many kinds of behavior.

For example:

 1Instruction following
 2
 3Writing style
 4
 5Domain knowledge
 6
 7Coding behavior
 8
 9Reasoning patterns
10
11Answer structure
12
13Tool usage

Suppose the teacher solves a reasoning problem like this:

 1Question:
 2If five machines make 100 products in four hours,
 3how many products can ten machines make in four hours?
 4
 5Teacher:
 6Ten machines means twice as many machines.
 7
 8If the time stays the same, production should double.
 9
10100 × 2 = 200.
11
12Answer: 200 products.

We can train the smaller model using examples like this.

After seeing many strong examples, the student may become much better at solving similar problems.

Why Use Distillation?

The main reason is often cost.

Imagine that using the teacher model costs:

1$10 for a certain amount of traffic

while the smaller student costs:

1$1 for the same amount of traffic

If the student becomes good enough for the application, production costs can drop significantly.

This makes distillation especially useful when we have a strong but expensive model.

The teacher performs expensive work during training.

The student performs cheaper work during production.

The Student Is Not An Exact Copy

There is an important limitation.

Knowledge distillation does not magically turn a small model into the large model.

A 7 billion parameter model still has much less capacity than a 70 billion parameter model.

Some behaviors may transfer very well.

Others may not.

The goal is usually not:

1Student = Teacher

A more realistic goal is:

1Student becomes good enough
2for the tasks we actually need.

This can still be extremely useful.

6. How These Techniques Work Together

These techniques should not always be thought of as competitors.

In real LLM training pipelines, several techniques can be used together.

For example:

 1             Pretraining
 2                 ↓
 3             Base Model
 4                 ↓
 5       Supervised Fine Tuning
 6                 ↓
 7      Instruction Following Model
 8                 ↓
 9         Preference Training
10                 ↓
11      Better Aligned Model
12                 ↓
13      Reinforcement Learning
14                 ↓
15        Better Reasoning Model

Knowledge distillation can appear at different stages.

For example, after building a very strong model:

 1Strong Model
 2
 3    ↓
 4
 5Generate High Quality Data
 6
 7    ↓
 8
 9Knowledge Distillation
10
11    ↓
12
13Smaller Model

So a real training process could look something like:

 1                         Raw Data
 2                            ↓
 3                       Pretraining
 4                            ↓
 5                        Base Model
 6                            ↓
 7                 Supervised Fine Tuning
 8                            ↓
 9                   Preference Training
10                            ↓
11                Reinforcement Learning
12                            ↓
13                     Strong Model
14                            ↓
15               Knowledge Distillation
16                            ↓
17             Smaller Production Model

This is only one possible pipeline.

Different companies and research teams use different combinations depending on what they are trying to build.

7. How Should You Choose A Technique?

A simple way to think about the techniques is based on the problem you are trying to solve.

These techniques can also solve overlapping problems.

For example, reasoning ability may be improved using SFT, reinforcement learning, distillation, or a combination of them.

The important question is not:

Which technique is the best?

A better question is:

What behavior are we trying to improve, and what training signal can teach that behavior?

8. A Simple Example

Imagine that we want to build a small coding assistant.

We start with a pretrained model.

Stage 1: Supervised Fine Tuning

We train it on high quality examples:

1Coding question → Good solution

Now the model becomes better at following coding instructions.

Stage 2: Preference Training

We compare different generated solutions.

For example:

1Solution A:
2Works but is difficult to understand.
3
4Solution B:
5Works and is simple and readable.

We teach the model that Solution B is preferred.

Stage 3: Reinforcement Learning

Now we allow the model to generate code.

We run automated tests.

1All tests pass → high reward
2
3Some tests pass → medium reward
4
5Tests fail → low reward

The model learns to write code that performs better on actual tests.

Stage 4: Knowledge Distillation

Suppose our final model is very powerful but expensive.

We ask it to solve millions of coding tasks.

Then we use those answers to train a smaller model.

Now we have:

1Powerful Teacher
2
3        ↓
4
5Millions of Coding Examples
6
7        ↓
8
9Smaller Student

The smaller model may preserve much of the useful coding ability while being cheaper to run.

This example shows how different post training techniques can work together rather than separately.

9. The Bigger Picture

A useful way to understand modern LLM development is to separate knowledge acquisition from behavior improvement.

Pretraining gives the model a broad foundation.

1Learn language
2
3Learn facts
4
5Learn patterns
6
7Learn basic reasoning

Post training then shapes how the model uses that foundation.

1Follow instructions
2
3Understand preferences
4
5Improve reasoning
6
7Learn specialized tasks
8
9Become cheaper through distillation

This is why simply increasing the amount of pretraining data is not enough.

A model can know a huge amount of information and still be a poor assistant.

Post training turns that knowledge into useful behavior.

Final Thoughts

Modern LLMs are not created through one giant training process.

They go through multiple stages.

The simplest way to remember the process is:

1Pretraining
2      ↓
3Teach the model what the world looks like
4
5Post Training
6      ↓
7Teach the model how we want it to behave

And within post training, four important techniques are:

Supervised fine tuning teaches the model using examples of good answers.

Preference training teaches the model which answers are better.

Reinforcement learning teaches the model through rewards.

Knowledge distillation allows a stronger model to teach a smaller model.

Once you understand these four ideas, many modern LLM training approaches become much easier to understand.

The implementation details can become complicated, but the core ideas are surprisingly simple.

At the end of the day, post training is mainly about one question:

What signal should we give the model so that it learns the behavior we want?