Talking with AI, Part 10: More Compute for Training or for Answering?
Scaling beyond parameter counts through training-time and inference-time compute, best-of-N sampling, verifiers and distillation.
Part 10 of 11 in Talking with AI: From LLM Training to AGI
A conversation between Dr. Suttisak Denduangchai and ChatGPT-5.6 Reasoning Instant, translated from the original Thai chat. ChatGPT’s explanations and examples are preserved, with headings, tables and mathematical notation adjusted for reading on the web.
Me: Tell me more.
ChatGPT-5.6 Reasoning Instant:
Absolutely. This time we have reached one of the biggest questions in AI:
If we had 100 times more compute, should we use it to train a larger brain, or give the existing brain more time to think and experiment?
The current answer is that we need to do both, which gives “Scaling” a much broader meaning than simply “increasing the number of parameters.” 🧠📈
1. What is a Scaling Law?
Early on, researchers found an interesting pattern: when they increased
Model size ↑
Training data ↑
Training compute ↑in suitable proportions, training/test loss generally improved in a fairly systematic way.
A simplified picture:
Performance
↑
│ ●
│ ●
│ ●
│ ●
│ ●
└────────────────────→ ComputeThis does not mean that using ten times as many GPUs makes a model ten times as intelligent. It means that some capabilities can continue to improve predictably as scale increases.
This was an important reason the field became willing to invest in enormous training clusters.
2. But increasing Parameters alone is not enough
Suppose we build:
Model A
70B parameters
trained on 10T tokensand then build:
Model B
700B parameters
trained on the same 10T tokensThat does not mean B will use its additional capacity efficiently.
It is like building a university ten times larger while keeping the same number of books in the library. 📚
We therefore need to balance:
Model Size ↔ Data ↔ ComputeA smaller model trained on more and better data may outperform a larger model that is undertrained.
3. This leads to Compute-optimal Training
The question becomes:
Given a budget of
C, how much should we allocate to model size and how many training tokens should we use?
instead of asking:
Can we make the model as large as the GPUs can hold?
Conceptually:
Compute Budget
│
├── Model capacity
│
├── Training tokens
│
└── Training durationWe have to balance them.
A smaller model trained on more high-quality data may beat a larger, undertrained model.
4. Then we begin to hit walls
Scaling pre-training becomes increasingly expensive.
If you want:
More computeyou need:
More accelerators
More electricity
More cooling
More networking
More storage
More data
More engineeringBut an even harder problem is that high-quality data does not grow without limit.
The Internet is very large, but:
Internet
≠
Infinite high-quality training dataIt contains:
spam
duplicates
SEO sludge
wrong information
machine-generated content
low-quality codeDigital garbage has a character all its own. 🗑️
5. This creates a new question
Instead of asking only:
How can we build a larger brain?
we begin asking:
How can we make the brain we already have use computation more intelligently?
This is Inference-time Scaling.
6. Training-time vs Inference-time Scaling
Let us separate them clearly.
Training-time
Uses compute before deployment.
Data
↓
GPU cluster
↓
Train 3 months
↓
ModelOnce training is complete, every user benefits from that set of weights.
Inference-time
Uses compute when the user asks a question.
Question
↓
Model
↓
Think / Search / Try / Verify
↓
AnswerAn easy problem uses little compute.
A difficult problem uses more.
This is a very powerful idea.
7. Consider a Coding problem
The task:
Implement parser X
Approach A
The model answers once:
Prompt
↓
SolutionIt might be correct 65% of the time.
But suppose we ask it to generate ten solutions:
Problem
│
┌────────┼────────┐
↓ ↓ ↓
Sol A Sol B ... Sol J
↓ ↓ ↓
Tests Tests Tests
│ │ │
FAIL PASS FAIL
↓
choose BThe success rate may increase substantially.
The weights have not changed at all.
What increased was:
Compute per problem
8. Best-of-N
This is one of the easiest forms of inference scaling to understand.
Instead of:
Generate 1 answerwe do this:
Generate N answers
↓
Evaluate
↓
Choose bestIf each attempt has a probability p of succeeding, and we assume the attempts are reasonably independent, the probability that at least one succeeds is:
1 − (1 − p)ᴺFor example, suppose each attempt is correct 50% of the time.
One attempt:
50%Five attempts:
1 − (0.5)⁵ = 96.875%But this number is only an idealized example. In the real world, the attempts are not independent, and more importantly, the system must still know which answer is correct.
This is where the verifier returns.
9. A Verifier makes Scaling valuable
If we generate 100 answers but do not know which one is correct:
100 answers
↓
🤷that does not help very much.
But if we have:
Compiler
Unit tests
Proof checker
Simulator
Known answer
External datawe can do this:
100 attempts
↓
Verifier
↓
3 pass
↓
select / further evaluateIt is therefore unsurprising that coding, mathematics, and formal reasoning are important arenas for inference-time scaling.
10. Search is also a form of Compute
Suppose a problem has several possible paths:
Problem
│
┌────────┼────────┐
↓ ↓ ↓
Path A Path B Path C
│ │ │
bad maybe maybe
│ │
Path B1 Path C1
│ │
good dead endInstead of committing immediately to the first token path, the system can explore alternatives.
This idea connects to an older approach in AI:
Search
Something that looks very new therefore reaches back to the classical roots of AI.
Neural Network
+
Search
+
Verificationis a powerful combination.
11. An Agent is also a form of Inference Scaling
A coding agent that does this:
Read
↓
Think
↓
Edit
↓
Compile
↓
FAIL
↓
Think again
↓
Edit
↓
Compile
↓
PASS
↓
Tests
↓
FAIL
↓
Debug
↓
Fix
↓
PASSis using vastly more compute than:
Prompt → AnswerBut that compute creates feedback from the real world.
This matters more than simply thinking for a long time.
12. “Thinking longer” is not always better
We should say this directly.
If the model’s reasoning starts from a wrong assumption:
Wrong assumption
↓
Think harder
↓
Elaborate wrong reasoning
↓
Beautifully wrong answer🔥
Simply adding compute is therefore not enough.
We also need:
Better search
Better feedback
Better verification
Better tools
Better training13. This makes Adaptive Compute interesting
Why should we use the same amount of compute for every question?
"What is HTTP?"and:
"Find a concurrency bug across this
400,000-line repository."should not receive the same budget.
A better idea is:
Request
↓
Estimate difficulty
↙ ↘
Easy Hard
↓ ↓
small compute large compute
↓ ↓
answer reason/tools/
verify/iterateThis is why the idea of reasoning effort is meaningful.
14. We now have a new Trade-off
Suppose we have:
Model BIGwith an inference cost of:
$$$$and:
Model SMALL
+
Tools
+
Search
+
Reasoning
+
Verifierwith a cost of:
$$but performance that is similar on some tasks.
The second option may offer better value.
So competition in AI is not only about:
Who can build the largest neural network?
It is also about:
Who can build the most efficient intelligence stack?
15. Distillation makes this even more interesting
Suppose we have a large model:
Teacherthat reasons well but is expensive.
We can use it to help create data for a smaller model:
Teacher
↓
high-quality examples
solutions
feedback
reasoning behaviors
↓
Student training
↓
Smaller modelThis broad idea is called distillation.
As a result, some of the capabilities of an expensive system can be transferred into a cheaper model.
16. The cycle begins to form a Flywheel
This is fascinating:
Strong Model
↓
Generate better synthetic data
↓
Verify / filter
↓
Train next model
↓
Stronger Model
↓
Generate even better data
↓
...But this does not mean the cycle will automatically turn into superintelligence. There are still problems involving:
error accumulation
diversity
verification
grounding
compute
data quality
evaluation17. Hardware is improving at the same time
On another front:
Better GPU/accelerator
↓
Cheaper training
↓
Bigger/better models
↓
More AI demand
↓
More accelerator investmentAnd inference optimizations such as:
Quantization
KV-cache optimization
Speculative decoding
Better kernels
Better batching
MoEallow the same model to be served more cheaply and quickly.
Capability per unit of price can therefore improve even while the main architecture remains a Transformer.
18. This answers the question, “Why is AI improving so quickly?”
There is not just one lever.
Many levers are moving at once:
AI Capability
↑
┌──────────────┼──────────────┐
│ │ │
Better Data Better Training Better Models
│ │ │
├──────────────┼──────────────┤
│ │ │
Better RL Better Tools Better Search
│ │ │
├──────────────┼──────────────┤
│ │ │
Better Hardware Better Serving More Test-time
ComputeIf each one improves a little every year, the combined effect can look like a dramatic leap.
19. Another very important factor: Algorithmic Efficiency
Suppose that in 2024 we needed:
100 units of computeto achieve performance level X.
Later, algorithms and training recipes improve until we need only:
20 unitsto achieve the same performance X.
For that task, it is as if our hardware became five times faster, even though no transistor changed.
Two trends are therefore moving together:
Hardware efficiency ↑
Algorithmic efficiency ↑And when multiplied together, the effect can be enormous.
20. This makes “number of Parameters” an increasingly deceptive metric
People used to ask:
How many billion parameters does this GPT model have?
The question still has meaning, but it is no longer enough.
Suppose:
Model A
1 trillion total parameters
MoE
50B active/tokenand:
Model B
200B dense
200B active/tokenYou cannot simply say:
A = 1T
B = 200Band conclude that A is five times more capable.
You need to examine:
Architecture
Active parameters
Training tokens
Data quality
Training compute
Post-training
RL
Inference compute
Tools
Context
EvaluationThe parameter count alone is almost like saying:
“This car has a 3,000 cc engine, so it must be faster than every car with a 1,500 cc engine.”
That is not enough. 🏎️
The full picture of AI we have built together now looks like this
DATA
│
▼
PRE-TRAINING
│
▼
BASE LLM
│
▼
POST-TRAINING
SFT / RL / etc.
│
▼
REASONING MODEL
│
═════════════════════════╪════════════════════════
│
INFERENCE
│
┌───────────┼───────────┐
↓ ↓ ↓
Context Memory RAG
│ │ │
└───────────┼───────────┘
↓
MODEL
│
┌────────┼────────┐
↓ ↓ ↓
Think Search Tools
│ │ │
└────────┼────────┘
↓
Observe
↓
Verify
↓
Good enough?
↙ ↘
No Yes
│ │
└──↺ AnswerAnd now we have reached what I think is the most interesting question in this series:
If we keep improving
Train → Reason → Tool use → Agent → Verification, what is the difference between an LLM that is “extremely capable” and AGI?
To answer that, we first have to unpack the term AGI, because people use it very loosely—from “a slightly better ChatGPT” all the way to “an AI that can replace people in every occupation,” which are completely different standards. 🤖🧩
Reading notes
Note added to the conversation: Classical scaling laws describe empirical relationships in loss within the regimes studied. They do not guarantee that every downstream capability or benchmark will improve smoothly as model size, data or compute increase.
Original research: Scaling Laws for Neural Language Models.
← Previous: From an LLM to an AI System and Agent
Next: From More Capable LLMs to the Question of AGI →
Read the series
- Part 1: What Does Training an LLM Mean?
- Part 2: How Can Predicting the Next Token Lead to Coding?
- Part 3: Context, Attention and Memory During a Conversation
- Part 4: Why More Thinking Can Improve an Answer
- Part 5: GPUs, VRAM and the Infrastructure Behind LLM Training
- Part 6: From a Base Model to an Assistant
- Part 7: Does an LLM Memorize, Learn or Guess?
- Part 8: Following a Token Through a Transformer
- Part 9: From an LLM to an AI System and Agent
- Part 10: More Compute for Training or for Answering? (this episode)
- Part 11: From More Capable LLMs to the Question of AGI