Talking with AI, Part 6: From a Base Model to an Assistant

Part 6 of 11 in Talking with AI: From LLM Training to AGI

A conversation between Dr. Suttisak Denduangchai and ChatGPT-5.6 Reasoning Instant, translated from the original Thai chat. ChatGPT’s explanations and examples are preserved, with headings, tables and mathematical notation adjusted for reading on the web.

Read the complete series

Me: Tell me more.

ChatGPT-5.6 Reasoning Instant:

Sure. This time we have reached the stage where the Base Model is refined into an Assistant 🧠🔧

After pre-training, we have a model that is good with language and has a great deal of knowledge, but there is a major problem:

It was trained to “continue text”; it was not originally trained to “help the user.”

1. What does a Base Model look like?

Suppose we give it:

User: What is Docker?

A base model might continue with:

User: What is Docker?
Assistant: What is Kubernetes?
User: I don't know.
Assistant: ...

Why?

Because it may see this as text from a forum/chat log and faithfully perform its original task:

Predict what the next text is likely to be.

It does not inherently have the concept that:

USER asks
     ↓
I should answer helpfully

So we need Post-training.

2. Supervised Fine-Tuning: SFT

We begin by preparing examples that we want it to imitate:

USER:
Explain DNS to a beginner.

ASSISTANT:
DNS is the system that translates domain names such as
example.com into IP addresses that computers use to communicate.

Another example:

USER:
Write a Python function that returns the larger number.

ASSISTANT:
def maximum(a, b):
    return max(a, b)

Many other types include:

Summarization
Coding
Translation
Reasoning
Writing
Question answering
Following constraints
...

Then we train further from the Base Model:

Base Model
    ↓
High-quality instruction examples
    ↓
Fine-tuning
    ↓
Instruction-following Model

The weights genuinely change, but much less data is used than in pre-training.

3. What SFT teaches is “behavior”

This is very important.

Pre-training may make the model know what Docker is.

SFT helps teach it:

When a user asks about Docker, how should it use the capabilities it already has?

So we can roughly think of it as:

Pre-training
    ↓
"What do you know?"

Post-training
    ↓
"How should you behave?"

They are not completely separate, but this mental model is useful.

4. But SFT has a problem

If we have the question:

Explain quantum computing.

A human writes one very good answer and we train on it.

The model learns from:

Question → Ideal Answer

But in reality, there may be thousands of possible answers.

Some answers are:

correct + concise
correct + long
correct but difficult to understand
easy to read but slightly wrong
confident nonsense
does not answer the question

We also want to teach something else:

Which answer is better than which?

This is the world of Preference Training.

5. Let the model generate several answers

Suppose we ask:

Explain TCP vs UDP.

We get:

A:
TCP is always better than UDP...

B:
TCP provides an ordered, reliable byte stream,
while UDP sends independent datagrams without
guaranteeing delivery or ordering...

C:
TCP and UDP are basically the same...

Humans might rank them:

B > A > C

We get a new kind of data:

Preference data

It is not merely:

Question → Answer

but:

Question
   ↓
Answer B preferred over Answer A

6. RLHF

Its full name is:

Reinforcement Learning from Human Feedback

The classic idea is roughly:

Human preferences
      ↓
Reward Model
      ↓
score(model answer)
      ↓
Reinforcement Learning
      ↓
improve LLM

A Reward Model learns roughly:

Question + Answer
       ↓
Reward Model
       ↓
Score

For example, conceptually:

Answer A → 0.34
Answer B → 0.91
Answer C → 0.08

Reinforcement learning is then used to make the LLM more likely to generate output with higher rewards.

7. The Reward Model is the “judge”

Imagine that we have:

Player
  =
LLM

Judge
  =
Reward Model

The LLM generates:

Answer

The judge gives:

Reward

Then training tries to make:

E[Reward] ↑

That is, to increase expected reward.

8. But there is a very interesting problem: Reward Hacking 👹

Suppose we teach that:

Detailed answers are often better.

The model may learn the wrong lesson:

The longer, the better.

From:

"What is 2+2?"

"4."

it becomes:

"To fully understand this fascinating mathematical
question, we must first explore the historical
foundations of arithmetic..."

before answering 4 three pages later 😵‍💫

This is a general optimization problem:

A system will optimize what you measure, which is not necessarily what you intended.

It is just like KPIs in an organization.

Set the wrong KPI and people do not become more productive; they become better at gaming the KPI.

AI cannot escape this problem either.

9. There are therefore other Preference Optimization methods

We do not always need to use the classic RLHF pipeline.

There is a family of methods such as:

DPO: Direct Preference Optimization

The idea is to use pairs:

Prompt
  ↓
Preferred answer
Rejected answer

and train the model to increase its preference for the better answer directly, without necessarily following the classic reward-model + RL pipeline in every case.

The conceptual picture:

             Prompt
               │
        ┌──────┴──────┐
        ↓             ↓
     Answer A       Answer B
     preferred      rejected
        │             │
        └──────┬──────┘
               ↓
         Update model

Today, post-training is a very large research area, and each company has its own recipe.

10. What about AI Feedback?

Having humans create all preference data themselves is very expensive.

So AI can help evaluate output in some cases.

Roughly:

LLM generates answers
        ↓
Another model evaluates
        ↓
Preference / critique
        ↓
Training

This leads to broad ideas such as RLAIF.

Reinforcement Learning from AI Feedback

Of course, a question immediately follows:

What if the AI judge is wrong?

Exactly 😂

So evaluation, calibration, human oversight, and other verification methods are needed.

11. Reasoning Training enters the picture

This is an important step for newer models.

Some problems have a verifiable answer.

For example:

Mathematics

137 × 49 = ?

It can be checked.

Programming

Implement this function.

Then run:

unit tests

Formal proof

Let a proof checker verify it.

So instead of asking humans:

“Does this answer look good?”

we have an environment that can answer:

PASS

FAIL

This is called verifiable reward.

12. Reinforcement Learning is therefore very powerful for reasoning

Suppose the model tries to solve a problem in 100 ways:

Attempt 1   ✗
Attempt 2   ✗
Attempt 3   ✓
Attempt 4   ✗
...
Attempt 71  ✓

We know which trajectories led to a correct answer.

Training can then increase the probability that the model produces successful reasoning behavior.

Conceptually:

Problem
   ↓
Model attempts solution
   ↓
Environment / Verifier
   ↓
       Correct?
       ↙     ↘
     No       Yes
      ↓        ↓
   low reward high reward
       \       /
        \     /
       RL update

This begins to differ greatly from SFT.

SFT is roughly:

“Follow this example.”

RL is roughly:

“Here is the goal. Find a way to accomplish it.”

This is a very important distinction.

13. Tool use can be trained too

Suppose the model is given the question:

What's the weather in Tokyo right now?

A good answer is not:

Tokyo is probably 24°C.

because the weights should not be the source of current weather information.

The behavior we want is:

Question
   ↓
Recognize:
"I need current information."
   ↓
Call weather/search tool
   ↓
Receive result
   ↓
Answer

So there can be training examples:

USER:
What's the weather in Tokyo?

ASSISTANT/tool:
weather(location="Tokyo")

TOOL:
27°C, rain

ASSISTANT:
It's currently 27°C with rain...

The model therefore learns the pattern:

When should it use a tool, and how?

14. Coding Agents are the same

Good behavior:

User:
Fix the bug.

Model:
Need repository context.

→ search

→ read file

→ inspect call sites

→ edit

→ run tests

FAIL

→ inspect error

→ edit

→ tests

PASS

This capability may come from several things combined:

Pre-training
+
SFT
+
Preference training
+
RL
+
Tool-use training
+
Agent environment
+
Inference-time reasoning

So when we say:

“Model X is better at coding than Model Y”

what we see in the product may not come from base model intelligence alone.

15. Where does Safety Training fit?

Post-training is also used to shape safety behavior.

For example, it teaches the model to distinguish:

Benign request
       ↓
Answer normally

Dangerous request
       ↓
Apply appropriate safeguards

The goal is not merely:

Refuse everything suspicious

because that would create a very annoying model.

Instead, it must learn the distinction:

Educational
Defensive
Benign
Transformative
Dangerous
...

This is a difficult classification/reasoning problem in its own right.

16. Is a System Prompt Training?

No.

This deserves a red box around it 🔴

Suppose we have:

SYSTEM:
You are a software architect.
Be concise.
Prefer C# examples.

This belongs in:

Context

It does not update the weights.

                MODEL
          ┌───────────────┐
          │    Weights    │
          │               │
          │  unchanged    │
          └───────┬───────┘
                  ↑
                  │
            System Prompt
                  +
             User Prompt

So it is an inference-time instruction.

It is not training.

17. AGENTS.md is not Training either

For example, you might have:

AGENTS.md

- inspect code before editing
- run tests
- use C# conventions
- don't modify unrelated files

A coding agent reads it into context.

So:

AGENTS.md
   ↓
Context
   ↓
LLM behavior

not:

AGENTS.md
   ↓
Backpropagation
   ↓
Change weights

This is why changing AGENTS.md changes behavior immediately, without having to train the model again.

18. This is how Fine-tuning differs from a System Prompt

Suppose we want a model to answer in a particular kind of Thai.

Prompting

SYSTEM:
Answer in Thai, concisely, using standard medical terminology.

weights:

unchanged

Fine-tuning

Take thousands of examples:

Input → Desired Thai medical answer
Input → Desired Thai medical answer
Input → Desired Thai medical answer
...

Then:

Backprop
↓
Update weights

The behavior is embedded in the model more deeply.

19. When should you Fine-tune?

People often rush to think:

“I have company data, so I should Fine-tune an LLM!”

Most of the time, I would first ask:

Do you want it to “know information” or to “change behavior”?

If you want:

“Answer questions from the latest company handbook”

then this is often more suitable:

RAG / Search / Files

because documents can change.

Today:

Vacation = 10 days

Next month:

Vacation = 12 days

RAG:

update document
→ done

Fine-tuning:

prepare dataset
→ train
→ evaluate
→ deploy

And fine-tuning should still not be treated as a database that can retrieve facts accurately.

But if what you want is:

“Have the model always write medical recommendations in this specific format”

Fine-tuning may be more interesting, because we are changing behavior/style/task specialization.

20. We can now draw almost the full life of an LLM

RAW DATA
   ↓
Data cleaning / filtering
   ↓
Tokenization
   ↓
════════════════════════════
      PRE-TRAINING
════════════════════════════
   ↓
Next-token prediction
   ↓
Backpropagation
   ↓
Base Model
   ↓
════════════════════════════
       POST-TRAINING
════════════════════════════
   ↓
SFT
   ↓
Preference optimization
   ↓
RL / verifiable rewards
   ↓
Tool-use / behavior / safety
   ↓
Assistant / Reasoning Model
   ↓
════════════════════════════
         DEPLOYMENT
════════════════════════════
   ↓
System Prompt
   +
User Prompt
   +
Context
   +
Memory / Retrieval
   +
Tools
   ↓
Inference-time compute
   ↓
Agent loop
   ↓
Answer / Action

And this now lets us answer our original question in more precise language:

“What does it mean to train an LLM?”

It is using data + an objective + optimization to change the parameters of a neural network so that the model has the probability distribution and behavior we want.

Meanwhile, most of what happens while you are talking with me right now is inference, not training.

But there is still an elephant standing in the middle of the room 🐘:

If we pour an enormous amount of data from the Internet into it, how much of the training data does the model “remember”?

For example, does it remember an entire book? Can it remember source code? When you ask about something in the training data, is it retrieving from memory or reconstructing from patterns? Why does it remember some things exactly but make things up about others? And how is memorization different from generalization?

This will take us to memorization, overfitting, generalization, data contamination, and hallucination—all very important pieces for understanding why LLMs are both “amazing” and “not 100% trustworthy” at the same time 🧠🧩

← Previous: GPUs, VRAM and the Infrastructure Behind LLM Training

Next: Does an LLM Memorize, Learn or Guess? →

Read the series

Back to Notes