Talking with AI, Part 9: From an LLM to an AI System and Agent
What happens between pressing Send and receiving an answer: context construction, tools, verification and model-serving infrastructure.
Part 9 of 11 in Talking with AI: From LLM Training to AGI
A conversation between Dr. Suttisak Denduangchai and ChatGPT-5.6 Reasoning Instant, translated from the original Thai chat. ChatGPT’s explanations and examples are preserved, with headings, tables and mathematical notation adjusted for reading on the web.
Me: Tell me more.
ChatGPT-5.6 Reasoning Instant:
Absolutely. This time, we will put everything together and trace what happens from the moment you press Send until you see the answer, while separating LLM, Model, AI System, and Agent from one another. 🧠🏭
LLM ≠ ChatGPT
This is the first point where we need to lock the mental model firmly in place.
An LLM is the neural network we have been discussing throughout:
Tokens
↓
Embeddings
↓
Transformer × N
↓
Logits
↓
Next tokenIt is the engine.
The AI product we use has many more things around it:
┌──────────────────────────────┐
│ AI SYSTEM │
│ │
│ Memory Retrieval │
│ ↘ ↙ │
│ Context Builder │
│ ↓ │
│ ┌─────────┐ │
│ │ LLM │ │
│ └────┬────┘ │
│ ↓ │
│ Tool Calling │
│ ↓ │
│ Web / Code / Files / APIs │
│ ↓ │
│ Verification │
│ ↓ │
│ Response │
└──────────────────────────────┘So when an AI becomes better at answering, that does not necessarily mean only that the neural network itself has become better.
The system around it may have improved as well.
First, separate these four terms
I will use the terms in a practical way:
An LLM is a neural network that processes tokens.
A Model usually means a model that has been trained and post-trained until it is ready for use, although in real products the term may be used more broadly.
An AI System is a model + infrastructure + context + tools + retrieval + safety + serving, and so forth.
An Agent is an AI system that can loop through think → act → observe the result → adjust → continue acting to achieve a goal.
That gives us a progression roughly like this:
LLM
↓
Model
↓
AI System
↓
AgentThese are not rigid, standardized definitions, but they form a very useful mental model.
Now you press Send
Suppose you type:
Please inspect this repository and find the bug.
The first thing the system has to do is not simply throw this message into a Transformer.
It has to construct context.
Something like:
System instructions
+
Developer instructions
+
Relevant conversation
+
User message
+
Potential memory/context
+
Tool definitions
↓
ContextThis is one reason the same model can behave very differently in different products.
The System Prompt is here
For example, conceptually:
SYSTEM:
You are an AI assistant.
Follow these rules...There may also be tool definitions:
Available tools:
search_web(...)
read_file(...)
run_command(...)
...And the conversation:
USER:
We're working on TidySpace.
ASSISTANT:
...
USER:
Find the MFT performance problem.All of this becomes context that the model uses for inference.
No training happens here.
Context Management may happen next
Suppose the conversation is 500,000 tokens long, but the model or system does not want to send all of it directly.
The system might select:
Recent conversation
+
Relevant older information
+
Summary
+
Retrieved documentsinstead of:
EVERYTHING SINCE THE DAWN OF TIME😆
Context has a cost.
And more information does not always mean a better result.
What is Routing?
A modern AI system may make a decision before handling a request about how that request should be processed.
For example:
User request
↓
Router
┌─────────┼─────────┐
↓ ↓ ↓
simple coding complex
↓ ↓ ↓
path A path B path COr it may decide:
Need web?
Need tools?
Need deeper reasoning?
Need image generation?
Need retrieval?The actual details are implementation-specific, and this does not mean every product must have the same kind of router.
The important concept is:
What a user sees as “one AI” may have orchestration behind it.
Suppose the request requires a repository
If the LLM has no access to the repository and you ask:
Where is the bug in
MftScanner.cs?
it has two options.
The bad path:
No file
↓
Guess
↓
Answer confidently☠️
The correct path:
Need repository
↓
Use file/code tool
↓
Search "MftScanner"
↓
Receive actual codeThe tool result is brought back into the context.
TOOL RESULT:
MftScanner.cs
public async Task...Then the LLM runs inference again.
This is where an Agent begins to emerge
The LLM might see the code and think:
I need to find the caller.
So it calls a tool:
search("MftScanner")and gets:
DiskAnalyzer.cs
ScanService.cs
...It reads further:
read("DiskAnalyzer.cs")It may then discover:
For every MFT record
→ allocate buffer
→ parse
→ throw awayand form the hypothesis that the allocation may be the bottleneck.
But a good agent should not declare victory immediately. 🏁
It should continue looking for evidence.
The Agent Loop looks like this
┌──────────────────────┐
│ │
▼ │
MODEL │
│ │
▼ │
Decide next action │
│ │
▼ │
TOOL │
│ │
▼ │
RESULT │
│ │
▼ │
Update context ───────────────┘It loops until:
Goal achievedor:
Budget exhausted
Need user input
Blocked
ErrorA Coding Agent is therefore not merely “an LLM writing code”
It is better understood like this:
LLM
│
┌─────────┼─────────┐
↓ ↓ ↓
Search Editor Shell
│ │ │
↓ ↓ ↓
Source Files Compiler
│
↓
Tests
│
↓
Result
│
└────→ LLMWhat makes it capable is not only that it knows C#.
It is also its ability to:
inspect
→ reason
→ modify
→ execute
→ observe
→ debug
→ verifyThe compiler is a Coding Agent’s superpower
This is one reason coding is a domain in which AI agents have progressed so quickly.
Suppose the model writes:
int x = "hello";No human needs to tell it that this is wrong.
dotnet build
↓
CS0029
Cannot implicitly convert type
'string' to 'int'The outside world sends objective feedback back to the agent.
The agent can then do this:
Error
↓
Understand
↓
Fix
↓
Build againThis is an excellent feedback loop.
Web Search works the same way
Ask:
Who is the CEO of company X right now?
It should not work like this:
LLM weights
↓
"I remember that..."It should work like this:
Question
↓
Current information required
↓
Search
↓
Official/reliable source
↓
Read
↓
Synthesize
↓
Cite
↓
AnswerThe capability shifts from:
recall
to:
research
That is a major difference.
RAG is also a kind of tool in the larger picture
User:
How many vacation days does the company provide?
↓
Search internal documents
↓
Employee-Handbook-2026.pdf
↓
Relevant section
↓
LLM
↓
"12 days, according to the handbook..."The LLM does not need to memorize the handbook.
And next year:
Employee-Handbook-2027.pdfit can use the new information without retraining the model.
Memory is also an external system
For example, conceptually:
Conversation A
↓
Relevant information
↓
External memoryThe following month:
New conversation
↓
Retrieve relevant memory
↓
Context
↓
LLMTherefore:
Memory
≠
WeightsThis is worth remembering.
Where does Reasoning fit?
For some requests:
2+2?
the path may be short:
Input
↓
Model
↓
4But for a problem such as:
Analyze this distributed architecture and produce a migration plan.
the system may allow more inference-time computation.
Conceptually:
Problem
↓
Decompose
↓
Analyze dependencies
↓
Consider alternatives
↓
Check constraints
↓
Find contradictions
↓
Refine plan
↓
AnswerIt therefore uses:
same/similar learned capabilities
+
more computationto improve quality.
Verification is the layer I most want to emphasize
A good AI system should not ask only:
“Is the model confident?”
because an LLM can be confident and wrong at the same time, with remarkable elegance. 🎩
It should ask:
What can we check this against?
For example:
Claim about web
→ source
Code
→ build/test
Calculation
→ calculator/code
Database state
→ query database
File
→ read actual fileThis gives us a broadly useful principle:
Use an LLM for reasoning and synthesis, but use external sources of truth when the matter can be checked.
What about Serving Infrastructure?
There is another entire world behind the scenes.
Once a request is ready to enter the model:
Request
↓
Inference server
↓
Scheduler
↓
GPU clusterBut you are not using the GPU alone.
There may be many requests:
User A ─┐
User B ─┤
User C ─┼→ Scheduler → GPUs
User D ─┤
User E ─┘The system has to use the GPUs efficiently.
For example, it may batch parts of several requests together.
Continuous Batching
Suppose:
User A: generating token 123
User B: generating token 51
User C: just arrivedInstead of waiting for A to finish first:
A A A A A A ... finish
B B B B ...
C ...the inference engine can arrange the batch dynamically:
Step 1:
[A][B][C]
Step 2:
[A][B][C]
B finished
Step 3:
[A][D][C]This improves accelerator utilization.
This is another world of engineering that the user never sees.
The latency you experience therefore has many components
When you press Send:
Total latency
=
Network
+
Context preparation
+
Routing
+
Queue
+
Prefill
+
Reasoning
+
Tool calls
+
DecodeThis gives us important metrics such as:
Time To First Token
The time from pressing Send until you see the first token.
and:
Tokens per second
How quickly the text flows after the response begins.
These are different things.
Some systems may behave like this:
wait 10 sec...
↓
then stream extremely quicklyThey may have high TTFT but good decode throughput.
Now let us assemble the whole ChatGPT-like system
Conceptually:
YOU
│
▼
User Prompt
│
▼
┌─────────────────┐
│ Product Layer │
└────────┬────────┘
│
┌───────────┼───────────┐
↓ ↓ ↓
Memory Retrieval Safety/
Policies
│ │ │
└───────────┼───────────┘
▼
Context Builder
│
▼
Routing
│
▼
┌─────────────┐
│ MODEL │
│ │
│ Transformer │
│ Weights │
└──────┬──────┘
│
▼
Decide
answer or action?
↙ ↘
Action Answer
↓
┌──────────────────┐
│ TOOLS │
│ │
│ Web │
│ Files │
│ Code │
│ APIs │
│ Database │
└────────┬─────────┘
│
▼
Result
│
└────────────→ Model
│
loop
│
▼
Final response
│
▼
YOUThe details of a real product may differ from this diagram, but this mental model is much closer to reality than:
User → Magic AI → AnswerThis also explains something important about the future of AI
We began this conversation with:
What does it mean to train an LLM?
At first, the answer seemed to be:
More data
+
More GPUs
+
Bigger neural network
=
Better AINow we can see that this is not enough.
The real picture is closer to:
TRAINING
│
▼
Better Model
│
▼
┌─────────────────────────────────┐
│ AI SYSTEM │
│ │
│ Model │
│ + Reasoning compute │
│ + Context management │
│ + Memory │
│ + Retrieval │
│ + Tools │
│ + Verification │
│ + Agent loops │
│ + Serving infrastructure │
└─────────────────────────────────┘
│
▼
CapabilityAnd this brings us to a major question at the heart of the field:
If we had 100 times more compute, should we use it to train a larger or more capable model, or give that compute to the existing model so it can “think, act, experiment, and verify its answer” at inference time?
This is the subject of Scaling Laws, training-time scaling versus inference-time scaling, and the reason the path to more capable AI may not consist only of building ever-larger Transformers.
That is an excellent next chapter, because it will help us begin to answer “Why is AI improving so quickly even though we are not increasing the parameter count wildly with every generation?” 🚀
← Previous: Following a Token Through a Transformer
Next: More Compute for Training or for Answering? →
Read the series
- Part 1: What Does Training an LLM Mean?
- Part 2: How Can Predicting the Next Token Lead to Coding?
- Part 3: Context, Attention and Memory During a Conversation
- Part 4: Why More Thinking Can Improve an Answer
- Part 5: GPUs, VRAM and the Infrastructure Behind LLM Training
- Part 6: From a Base Model to an Assistant
- Part 7: Does an LLM Memorize, Learn or Guess?
- Part 8: Following a Token Through a Transformer
- Part 9: From an LLM to an AI System and Agent (this episode)
- Part 10: More Compute for Training or for Answering?
- Part 11: From More Capable LLMs to the Question of AGI