Talking with AI, Part 11: From More Capable LLMs to the Question of AGI

Part 11 of 11 in Talking with AI: From LLM Training to AGI

A conversation between Dr. Suttisak Denduangchai and ChatGPT-5.6 Reasoning Instant, translated from the original Thai chat. ChatGPT’s explanations and examples are preserved, with headings, tables and mathematical notation adjusted for reading on the web.

Read the complete series

Me: Tell me more.

ChatGPT-5.6 Reasoning Instant:

Absolutely. This time we are moving beyond “How does an LLM work?” to a question for which the field still has no shared answer:

When does an increasingly capable LLM or Agent become AGI? 🤖🧠

The first misconception to dismantle is that AGI is not the name of an architecture. There is no specification that says:

Transformer + 10T parameters = AGI

And there is no AGI.exe that you can run and see PASS. 😆

1. What is AGI?

AGI stands for Artificial General Intelligence.

The word General matters most.

A narrow AI is good at one task:

Chess AI
   ↓
Chess

or:

Image classifier
   ↓
Image classification

Human intelligence is more general than that.

You can:

Write C#
↓
Learn a new framework
↓
Design an architecture
↓
Read a contract
↓
Plan a trip
↓
Learn to cook
↓
Debug a washing machine

without retraining your brain for every task.

The idea of AGI is therefore roughly:

A system that can learn and perform a very broad range of intellectual tasks at a high level, transferring capabilities across domains.

But people still debate exactly what level would count as passing that threshold.

2. The problem is that current LLMs are already very General

This makes the term AGI increasingly complicated.

One model can:

Translate Thai
Write C#
Explain biology
Analyze financial statements
Write SQL
Summarize a contract
Solve mathematics
Generate poetry
Plan software architecture

Twenty years ago, this would have sounded almost like science fiction.

But we still see very strange failures:

A very difficult problem
✓

An easy problem with slightly unusual framing
✗

or:

Write a complex algorithm
✓

Count objects in an image
✗

This is jagged intelligence.

3. Jagged Intelligence matters enormously

Human abilities tend to be reasonably correlated.

If someone can write a compiler, we expect that person can probably:

count to 20
understand simple instructions
remember something said 30 seconds ago

But an AI can have a profile like this:

Programming        ██████████
Mathematics        █████████
Writing            ██████████
Planning           ████████
Common sense       ██████
Reliability        █████
Long-horizon work  ████

So the statement:

“The AI achieved a PhD-level benchmark score.”

does not mean:

“It has the capabilities of a person with a PhD in every dimension.”

Those two statements are very far apart.

4. Capability vs Reliability

This may be a more important gap than simply increasing an AI’s IQ.

Suppose a model can complete each step correctly 95% of the time.

That sounds excellent.

But suppose the task has 50 steps, and make the oversimplified assumption that every step is independent:

0.95⁵⁰ ≈ 7.7%

The chance of completing every step without a single error falls to about 7.7%.

Of course, real-world steps are not independent like this, but the example illustrates the problem well:

Small errors can accumulate across a long-horizon task.

An agent therefore needs:

Planning
Checkpoints
Verification
Recovery
Replanning

It is not enough for the model to be intelligent.

5. Long-horizon autonomy is an important barrier

The task:

“Write this function.”

is easier than:

“Maintain this software project for three months.”

The latter requires:

Understand goal
   ↓
Plan
   ↓
Work
   ↓
Encounter unknown problem
   ↓
Research
   ↓
Change plan
   ↓
Remember decisions
   ↓
Coordinate dependencies
   ↓
Test
   ↓
Notice regression
   ↓
Recover
   ↓
Continue tomorrow

This requires more than next-token prediction in a single use.

It requires system architecture around the model.

6. Memory is another problem

Humans have several memory systems.

In a rough sketch:

Working memory
Long-term memory
Skills
Experiences
Semantic knowledge

An LLM core has:

Weights
+
Context

An agent system therefore has to add:

Working context
       +
External memory
       +
Retrieval
       +
Project state

If this is managed poorly:

Today, the agent changes an architectural decision.

Tomorrow:

“Why is the architecture like this?”

even though it was the one that changed it yesterday. 🤦

7. Learning on the job is another gap

Humans:

Work
↓
Make a mistake
↓
Learn
↓
Become better next month

Normal LLM inference:

Work
↓
Make a mistake
↓
Receive a correction
↓
Use the correction in context

But the weights are:

unchanged

When the context disappears, that new knowledge does not automatically become parameter learning.

This leads to a research question called:

Continual Learning

How can we make an AI continue learning without a major retraining run and without destroying its old capabilities?

8. Catastrophic Forgetting

Suppose we fine-tune a model heavily to make it good at law.

Before:

Law       ██████
Coding    ████████
Math      ████████

After poor fine-tuning:

Law       ██████████
Coding    █████
Math      ████

The model learned something new but broke what it previously knew.

This broad problem is called catastrophic forgetting.

Humans forget too, but neural networks have a form of this problem that must be managed seriously.

9. World Model

This is another very important term.

If an AI is going to plan well, it needs to predict:

If I do X, what will the world be like afterward?

For example:

Deploy database migration
       ↓
Will old app still work?
       ↓
Will schema break?
       ↓
Can rollback?

This means modeling:

P(future state | current state, action)

An LLM already has some world knowledge and representations from training.

But the question is whether it has a world model robust enough for long-term planning.

There is still a great deal of research to be done here.

10. Grounding

An LLM learns primarily from tokens.

Humans learn from:

Vision
Sound
Touch
Movement
Pain
Objects
People
Consequences
Language

We do not learn the word hot from Wikipedia alone. 🔥

We connect it to the real world.

Modern AI is therefore moving toward multimodal systems:

             Text
              │
      Image ───┼─── Audio
              │
            Model
              │
      Video ───┼─── Actions

When a model understands images, sound, video, and interaction with an environment, its knowledge is not bound to text alone.

11. Robotics is Hard Mode

In software:

mistake
↓
unit test fails
↓
try again

In the real world:

Robot:
"I think I should grab the glass here."

*the glass breaks*

💀

The physical world has:

Latency
Noise
Physics
Partial observability
Safety
Irreversible actions

Embodied intelligence is therefore more difficult than a coding agent.

12. AGI does not necessarily need a Robot

This distinction also matters.

Some definitions of AGI focus on:

cognitive/economic tasks

An AI could work through:

Computer
Browser
APIs
Software
Communication

without having arms or legs.

If a system can perform most knowledge work generally and autonomously, some people may already call it AGI.

Others will not accept the label until it is more general than that.

This is why you will see some people argue:

“AGI is already here.”

while others say:

“We are still far from AGI.”

Both sides are using different definitions of AGI.

13. I prefer to think in terms of an “AGI Ladder” rather than a single line

Instead of:

Not AGI │ AGI
─────────┼──────

think across multiple dimensions:

Dimension: Breadth

Question: How many domains can it work in?

Dimension: Depth

Question: How difficult can the work be?

Dimension: Reliability

Question: Does it perform correctly and consistently?

Dimension: Adaptability

Question: Can it learn when it encounters a new problem?

Dimension: Autonomy

Question: How long can it work independently?

Dimension: Memory

Question: Can it preserve long-term context?

Dimension: Tool use

Question: How well can it use the external world?

Dimension: Planning

Question: Can it plan across multiple steps?

Dimension: Recovery

Question: Does it recognize and correct its mistakes?

Dimension: Grounding

Question: Can it connect what it knows to the real world?

An AI can be very strong on some axes and still weak on many others.

The single term AGI flattens all of this detail too much.

14. What is ASI?

Beyond AGI, there is a term:

Artificial Superintelligence

or ASI.

The idea is an AI that greatly exceeds humans across a broad range of intellectual capabilities.

Roughly:

             Human
               │
ANI ───────── AGI ───────────── ASI
specific      general         superhuman

But do not interpret this as an evolutionary roadmap proven to proceed like this:

LLM → AGI → ASI

We do not know that it will happen.

15. Recursive Self-Improvement

This is the idea that makes people both excited and worried.

Suppose an AI becomes capable enough to:

Improve AI research
      ↓
Design better algorithms
      ↓
Improve training
      ↓
Create better AI
      ↓
Better AI improves AI research
      ↓
...

This creates a feedback loop:

AI₁
 ↓ improves
AI₂
 ↓ improves faster
AI₃
 ↓
AI₄

This is the idea of recursive self-improvement.

But moving from the concept to reality faces enormous obstacles:

Compute
Experiments
Hardware
Data
Evaluation
Physical constraints
Research uncertainty
Deployment
Safety

An AI writing a paper that says “GPUs should be 100 times faster” does not make silicon sprout from the desk. 😂

16. An Intelligence Explosion is therefore not an automatic conclusion

Some scenarios propose:

Better AI
→ faster AI research
→ even better AI
→ even faster research
→ ...

until capability increases very rapidly.

This is conceptually possible, but the size and speed of the feedback loop are empirical questions.

It could look like:

🚀

or:

📈

or hit a bottleneck:

📈📈📈──────
          compute/data/physics

We should not pretend to know the answer in advance.

17. More interesting than the term AGI is “Economically Useful Autonomy”

Suppose an AI still fails strange benchmarks, but can reliably:

receive a GitHub issue
↓
read the repository
↓
fix the bug
↓
write tests
↓
open a PR
↓
respond to review
↓
fix CI
↓
become merge-ready

That would have an enormous effect on the real world, even if people still argued:

“Is this AGI yet?”

Economically, the label may matter less than the capability.

18. For Software Development, we may see a progression like this

Autocomplete
     ↓
Chat assistant
     ↓
Code generation
     ↓
Repository-aware assistant
     ↓
Coding agent
     ↓
Issue → PR agent
     ↓
Long-running development agent
     ↓
Multi-agent software team

Notice that many of these steps do not require only a larger model.

They require:

Better model
+
Tools
+
Memory
+
Environment
+
Verification
+
Orchestration

This aligns with everything we have learned throughout the series.

19. Multi-Agent systems are interesting, but they are not magic

Suppose we have:

              Root
                │
       ┌────────┼────────┐
       ↓        ↓        ↓
 Architect   Coder      Reviewer
       │        │        │
       └────────┼────────┘
                ↓
             Result

It looks excellent.

But if every agent makes the same reasoning error:

Agent A: wrong
Agent B: agrees
Agent C: LGTM

🎉 We have achieved distributed wrongness.

A multi-agent system is useful when the division of labor, context isolation, independent verification, and coordination genuinely make sense, rather than when agents are added merely to make the diagram look impressive.

20. This brings us back to what matters most: Feedback

Consider the whole progression:

LLM
Prompt → Answer

Add reasoning:

Prompt
 ↓
Think
 ↓
Answer

Add tools:

Think
 ↓
Act
 ↓
Observe
 ↓
Answer

Add an agent:

Think
 ↓
Act
 ↓
Observe
 ↓
Evaluate
 ↓
Replan
 ↓
Act
 ↓
...

Add verification:

Plan
 ↓
Act
 ↓
Observe
 ↓
VERIFY
 ↓
wrong?
 ↙   ↘
yes   no
 ↓     ↓
fix   continue

The feedback loop is what turns an LLM from a text generator into a system that can perform work.

If we bring the entire series together from the first question

We started with:

“What does it mean to Train an LLM?”

Now we have reached:

DATA
 │
 ▼
PRE-TRAINING
 │
 ▼
BASE MODEL
 │
 ▼
POST-TRAINING
 │
 ▼
REASONING MODEL
 │
 ▼
┌───────────────────────┐
│       AI SYSTEM       │
│                       │
│ Context               │
│ Memory                │
│ Retrieval             │
│ Tools                 │
│ Reasoning             │
│ Verification          │
└──────────┬────────────┘
           ▼
         AGENT
           │
           ▼
    Long-horizon work
           │
           ▼
 Greater generality /
      autonomy
           │
           ▼
        AGI ?

The ? at the end is the most important part.

Because we do not have an AGI finish line that everyone agrees on.

And I think the next chapter should step back from philosophy and return to something tangible:

If you really want to train an LLM yourself, how do you do it?

We can begin with a tiny model on a single computer: build a small Transformer, prepare a tokenizer and dataset, train it on next-token prediction, and watch the loss fall with our own eyes. Then we can move on to LoRA / QLoRA → fine-tuning an open model → RAG → Agent.

At that point, you will see that all the ideas we have discussed are not spells known only to giant AI companies. You can build a miniature version yourself. 🔬💻

← Previous: More Compute for Training or for Answering?

Read the series

Back to Notes