Talking with AI, Part 3: Context, Attention and Memory During a Conversation
Why a model can use information you just supplied without changing its weights: context windows, in-context learning, attention and the KV cache.
Part 3 of 11 in Talking with AI: From LLM Training to AGI
A conversation between Dr. Suttisak Denduangchai and ChatGPT-5.6 Reasoning Instant, translated from the original Thai chat. ChatGPT’s explanations and examples are preserved, with headings, tables and mathematical notation adjusted for reading on the web.
Me: Tell me more.
ChatGPT-5.6 Reasoning Instant:
Picking up from there 🧠⚡, first we need to separate two kinds of “learning.” They look similar, but the engines underneath are different.
Training changes a model’s weights. Inference / In-context learning does not change weights; the model uses information in the context to calculate its current answer.
Suppose you tell me:
In this project,
Honk = gooseandQuack = duck
Then 20 messages later you ask:
What animal is Honk?
I can answer “goose” without training the model again.
The overall picture is:
Training
────────
Training data
↓
Gradient descent
↓
Change weights
↓
Model
Conversation
────────────
Model + Context
↓
Inference
↓
Answer
Weights do not changeWhat is a Context Window?
When you send a message, the model does not see only the latest message. The system can send relevant conversation context along with it.
Roughly:
SYSTEM:
You are a helpful assistant...
USER:
I am working on a TidySpace project...
ASSISTANT:
...
USER:
What should be fixed in DiskAnalyzer?
ASSISTANT:
...Everything is tokenized into one large sequence:
[token1][token2][token3] ... [token N]The maximum number of tokens the model can process this way is its Context Window.
An easy analogy: weights are a brain that has been educated, while context is a desk covered with the documents currently open.
A larger desk lets you open more documents at once, but it does not mean the brain has been trained again.
What is strange about In-context Learning?
Suppose I have never known the language you invented:
Bloop = cat
Zarg = dog
Mep = runsThen you provide examples:
Bloop mep = The cat runs
Zarg mep = The dog runsand ask:
What does "Bloop" mean?The model can infer:
Bloop → cateven though its weights have not changed at all.
This is In-context learning.
What is interesting is that training equips a model with the ability to learn new patterns from context at the time of use.
Put another way:
Training does not teach only “answers”; it can create a machine that learns new patterns from a prompt.
Attention is the mechanism that lets it reuse information
Back to this example:
Honk = goose
Quack = duck
...many more messages...
What animal is Honk?When the model processes Honk in the question, Attention can link it to earlier information:
┌─────────────────────┐
↓ │
Honk = goose ... ... ... What is Honk?
███████████████████████████████
attentionThis does not mean it is making a database query like:
SELECT value
FROM conversation
WHERE key = 'Honk';It is neural-network computation.
What are Q, K, and V?
This is the heart of the Transformer, with a name that sounds like a three-boss gang 😄
Attention uses:
Query, Key, Value
The idea resembles a search system.
Suppose it has this information:
Honk = gooseIt creates representations that roughly serve as:
Key:
"Information about Honk"
Value:
"Information associated with goose"When it sees:
What animal is Honk?the current representation creates a Query:
Query:
"I need information about Honk"Attention then compares the Query with Keys:
Query
│
├── "TidySpace" 0.02
├── "Windows" 0.05
├── "Honk = goose" 0.91 ← 👀
└── "C#" 0.01In reality this is matrix/vector mathematics, not text or probabilities as in the example, but the intuition is useful.
The Transformer’s famous formula is:
Attention(Q,K,V) = softmax((QKᵀ) / √dₖ) VIn human terms:
Q = What am I looking for?
K = What is each piece of information about?
V = If this piece is relevant, what information should I use?Then:
Q × K
↓
Which things are most related?
↓
softmax
↓
Assign weights
↓
Blend in the relevant parts of VThis is one of the basic mechanisms that lets a Transformer handle context.
What is a KV Cache?
This relates directly to performance.
Suppose ChatGPT is generating:
The quick brown fox jumps over the lazy dog.
It does not create the whole sentence at once. It generates roughly:
The
↓
quick
↓
brown
↓
fox
↓
jumps
↓
...or, in reality, one token at a time.
When generating fox, the model must consider:
The quick brownIf it recalculated the representations of all old tokens every time it generated another token, it would be extremely wasteful.
So it stores the Keys and Values of prior tokens.
That is the:
KV Cache
Roughly:
Token Cached K/V
──────────────────────────────
The ✓
quick ✓
brown ✓
fox ✓
jumps ← being generatedIt can then reuse the old values for the next token.
This makes autoregressive generation more efficient.
Context Window and KV Cache are therefore related
If the context is very large:
1k tokens
10k
100k
1M
...the model must handle information about more tokens.
The KV cache can also use more memory.
So long context is not free.
It has costs in:
Memory
Compute
Latency
Attention
Retrieval qualityThere is another important point:
A large context does not mean the model will use every token equally well.
Putting a million tokens of source code in context and asking about one line does not guarantee it will find the bug.
This is why agent systems and coding assistants often use:
Search
Retrieval
File tools
grep
symbol search
RAG
summarizationinstead of stuffing an entire repository into context.
What about ChatGPT’s Memory?
That is a separate matter.
Conceptually:
LLM
│
┌───────┼────────┐
↓ ↓ ↓
Weights Context External memoryWeights
Knowledge/capabilities gained from training.
Fairly permanent until the model is trained or adjusted again.
Context
What the model is seeing in that inference run.
"We are currently discussing Transformers"External memory
An external system can store certain information and put it back into context in the future.
The LLM itself does not need to change its weights.
So the phrase “ChatGPT remembers me” should not be understood as:
GPT weights
↓
training...
↓
Add Suttisak’s information to the neural networkIt is more like:
Stored/retrieved information
↓
Select what is relevant
↓
Put it into context
↓
LLM
↓
ResponseRAG uses a similar principle
Suppose a company has 100,000 pages of documents.
Instead of retraining the LLM:
100,000 documents
↓
Train
↓
New modelwe can do:
User question
↓
Search / Retrieval
↓
Find relevant documents
↓
Take only important passages
↓
Context
↓
LLMFor example:
Which AWS region does HealthTAG production use?
The system searches architecture documents and finds:
Region: ap-southeast-1It then sends the LLM:
Context:
HealthTAG production region is ap-southeast-1.
Question:
Which AWS region does production use?The LLM answers Singapore.
This is not new knowledge trained into the model.
It is retrieval + inference.
Tools go further than RAG
An LLM alone has a world like this:
Text in
↓
Transformer
↓
Text outBut an agent can be:
LLM
│
┌────────┼────────┐
↓ ↓ ↓
Search Code Database
│ │ │
└────────┼────────┘
↓
Result
↓
LLM
↓
AnswerSuppose you ask:
How many open PRs does the repository have right now?
Expecting the weights to know this answer is wrong from the start, because the information changes constantly.
What should happen is:
LLM
↓
"I need to check GitHub"
↓
GitHub tool/API
↓
PR #42
PR #47
PR #51
↓
LLM
↓
"There are 3 PRs"This is where the picture of an Agent begins to emerge.
This lets us separate modern AI into several layers
The big picture is roughly:
┌──────────────────────────────────┐
│ AGENT │
│ │
│ Planning / Tool use / Loop │
│ │
│ ┌──────────────────────────┐ │
│ │ LLM │ │
│ │ │ │
│ │ Transformer + Weights │ │
│ └──────────────────────────┘ │
│ ↑ ↑ │
│ Context Memory │
│ ↑ │
│ Retrieval / Tools │
│ ↑ │
│ GitHub / Web / Files / APIs │
└──────────────────────────────────┘This matters because when you use a coding agent such as Codex and it:
reads AGENTS.md
↓
searches source
↓
opens files
↓
thinks
↓
edits code
↓
runs tests
↓
finds a test failure
↓
reads the error
↓
fixes it
↓
runs teststhat does not mean the LLM is retrained while it works.
What is happening is:
The same LLM + changing context + tools + a feedback loop
And that takes us to something one level more interesting:
Why does letting an LLM “think longer” make it better when its weights are unchanged?
This includes ideas such as reasoning models, test-time/inference-time compute, and agentic loops, and why the same model may solve a difficult problem better when given more compute.
It connects directly to what you see when using GPT-5.6 Sol across Instant, Medium, High, and Extra High thinking: why can simply giving the same model more time to compute make such a large difference? 🧠🔥
Reading notes
Note added to the conversation: The diagrams and comparisons of reasoning levels in this episode are conceptual explanations of inference-time compute. They do not describe ChatGPT’s internal architecture or establish that its reasoning levels differ only in time and compute. The name ChatGPT-5.6 Reasoning Instant identifies the conversation as recorded by the author.
See the publicly documented capabilities in OpenAI’s GPT-5.6 Sol documentation.
← Previous: How Can Predicting the Next Token Lead to Coding?
Next: Why More Thinking Can Improve an Answer →
Read the series
- Part 1: What Does Training an LLM Mean?
- Part 2: How Can Predicting the Next Token Lead to Coding?
- Part 3: Context, Attention and Memory During a Conversation (this episode)
- Part 4: Why More Thinking Can Improve an Answer
- Part 5: GPUs, VRAM and the Infrastructure Behind LLM Training
- Part 6: From a Base Model to an Assistant
- Part 7: Does an LLM Memorize, Learn or Guess?
- Part 8: Following a Token Through a Transformer
- Part 9: From an LLM to an AI System and Agent
- Part 10: More Compute for Training or for Answering?
- Part 11: From More Capable LLMs to the Question of AGI