Talking with AI, Part 7: Does an LLM Memorize, Learn or Guess?

Part 7 of 11 in Talking with AI: From LLM Training to AGI

A conversation between Dr. Suttisak Denduangchai and ChatGPT-5.6 Reasoning Instant, translated from the original Thai chat. ChatGPT’s explanations and examples are preserved, with headings, tables and mathematical notation adjusted for reading on the web.

Read the complete series

Me: Tell me more.

ChatGPT-5.6 Reasoning Instant:

Sure 🧠 This time, we are getting into the subject that causes many people to misunderstand LLMs the most:

Does an LLM “remember” what it read during training, or does it “learn”?

The answer is both, but they are different phenomena, and the boundary is not sharp.

1. Start with Memorization

Suppose the training data contains this sentence many times:

The capital of France is Paris.

After training, if we ask:

What is the capital of France?

the model answers:

Paris

But from the output alone, we cannot simply conclude that it contains a record:

France → Paris

It may have learned representations from a huge amount of related text:

France
 ├─ Paris
 ├─ Europe
 ├─ French
 ├─ Eiffel Tower
 └─ ...

But in some cases, a neural network can genuinely memorize specific sequences from training data, especially data that is repeated often or is distinctive.

2. How are Memorization and Generalization different?

Suppose the training data contains:

2 + 2 = 4
3 + 5 = 8
7 + 6 = 13

Then we ask:

438 + 217 = ?

If the model answers 655 because it has seen this exact text:

Memorization

But if it has never seen 438 + 217 and can still find the answer:

Generalization

This is what we want more of.

Training examples
       ↓
learn underlying pattern
       ↓
New unseen problem
       ↓
Correct answer

3. Overfitting is the other side of the story

Suppose I have a training set:

Question 1 → A
Question 2 → C
Question 3 → B
...

The model memorizes it all until training accuracy is:

100%

It looks incredibly capable 🧙

But then we give it a new exam:

40%

This is overfitting.

It is good at the training set but does not generalize well.

So we split data into, for example:

Dataset
  │
  ├── Training set
  │
  ├── Validation set
  │
  └── Test set

An important principle is that test data should not be used in training.

4. Then why do large LLMs not Overfit the entire Internet?

In reality, this is more complicated, but one reason is that the data is enormous and highly diverse.

Suppose there is text about gravity:

Newton described gravity...
F = Gm₁m₂/r²
Planets orbit...
The Moon...
Falling objects...
Orbital mechanics...

The useful way to reduce loss is not merely to remember every page; it is also to build representations that capture regularities among those pieces of data.

Something like this emerges:

             gravity
                │
       ┌────────┼────────┐
       ↓        ↓        ↓
     mass    distance   force
       │        │        │
       └────────┼────────┘
                ↓
           relationships

This is why an LLM can answer questions in forms that never appeared directly in the training data.

5. But LLMs really can memorize

We should not sugar-coat this too much.

Models can memorize some information from the training set, especially sequences that:

appear very frequently
are distinctive
have high predictability (see the clarification)
or have other conditions that cause them to be learned strongly

So model developers need to care about:

Deduplication
Privacy filtering
Copyright
Data governance
Memorization evaluation

Because saying:

“LLMs do not remember anything; they only learn patterns”

is also incorrect.

6. Data Deduplication is very important

Suppose an Internet dataset has the same article copied across 10,000 websites:

Article X
Article X
Article X
Article X
...
× 10,000

Without deduplication, this data has unnatural weight.

The model receives repeated signals that:

This text is extremely important.

So there are preprocessing pipelines such as:

Raw Web
   ↓
Parsing
   ↓
Language detection
   ↓
Quality filtering
   ↓
Safety / policy filtering
   ↓
Deduplication
   ↓
Dataset mixture
   ↓
Training

Building a dataset is therefore a very large engineering/research problem, not simply wget Internet and throw it into GPUs 😆

7. Data Mixture matters too

Suppose a dataset is:

99.9% Reddit comments
0.1% everything else

You can probably imagine the model’s personality 😅

Creators must decide how to mix:

Web
Books
Code
Math
Science
Reference
Multilingual
High-quality curated data
Synthetic data
...

and in what proportions.

So:

Data quality can matter as much as Data quantity.

One trillion tokens of data are not automatically better than 500 billion carefully selected tokens.

8. What is Synthetic Data?

It is very interesting for newer LLMs.

Instead of using only human text:

Human data
    ↓
Training

a model can be used to generate training examples:

Strong model
    ↓
Generate problems
Generate solutions
Generate explanations
Generate code
    ↓
Verify / filter
    ↓
Training data

For example, generate a programming problem:

Problem
 ↓
Candidate solution
 ↓
Compile
 ↓
Unit tests
 ↓
PASS
 ↓
Keep

If it fails:

🗑️

This is very good because there is a verifier.

9. But Synthetic Data can be toxic too ☣️

Suppose AI generates incorrect data:

Model
 ↓
Wrong answer
 ↓
Train next model
 ↓
Wrong pattern reinforced

Or synthetic data may have low diversity.

In the end, a model may learn too much from a world filtered through earlier models.

So it is not:

Synthetic = cheap unlimited intelligence

It needs:

Generation
+
Filtering
+
Verification
+
Diversity
+
Human / real-world grounding

10. What is Benchmark Contamination?

Suppose there is a benchmark:

Question:
If Alice has...

We use it to test how good a model is at reasoning.

But by chance, this question and its solution are in the training data.

Then the model answers correctly.

We might announce:

🎉 Reasoning score 98%!

But in reality, some of it may be:

Question
 ↓
seen during training
 ↓
memorized/reconstructed answer

It does not demonstrate as much generalization as we thought.

This is called data contamination.

11. Benchmarks therefore have an expiration date

Very famous benchmarks are posted on:

GitHub
Papers
Blogs
Reddit
Tutorials
Stack Overflow

A few years later, information about that benchmark fills the Internet.

Building new generations of training datasets must therefore be more careful.

This is one reason modern evaluation needs:

Private tests
New problems
Dynamic evaluations
Real-world tasks
Agent environments

rather than trusting a single leaderboard.

12. What causes Hallucination?

Now, everything we have learned fits together.

LLMs are trained to:

P(next token | context)

They are not fundamentally built to perform:

LookupTruth(question)

Those are enormously different.

Suppose we ask:

Who wrote the book The Quantum Duck of Bangkok?

If this book does not exist, the model still has to generate the next token.

It may generate:

“The book was written by Dr. ...”

because the pattern:

"Who wrote X?"
       ↓
"X was written by Y."

has linguistic probability.

Fluent language does not guarantee truth.

This is the heart of hallucination.

13. Why does it not always answer “I don’t know”?

Because the next-token objective does not have an internal variable like:

knowledge_confidence = 0.00
therefore say "I don't know"

as simply as that.

Post-training can teach a model to express uncertainty better, but the fundamental problem remains difficult:

The model must know what it does not know.

Which is harder than answering a question.

14. This is why RAG and Tools matter again

Instead of asking the weights:

Who is the CEO of company X right now?

we do:

Question
   ↓
Web/Search
   ↓
Current authoritative source
   ↓
Context
   ↓
LLM
   ↓
Answer + citation

This changes the system from:

"Do you remember?"

to:

"Go check."

For information that changes all the time, the latter is far superior.

15. Code is the same

Ask an LLM:

Is this function correct?

LLM:

“It looks correct.”

That is not enough.

It should:

Code
 ↓
Compiler
 ↓
Tests
 ↓
Static analysis
 ↓
Runtime
 ↓
Result

Then let the LLM analyze the evidence.

The broad principle is:

If truth can be checked against the outside world, do not rely on the LLM’s memory alone.

This is a very good principle for designing AI systems.

16. Where does “Knowledge” live in the weights?

Here we enter Mechanistic Interpretability.

Researchers want to know:

Which neuron does what?

At first, we might hope to find:

Neuron 19384
=
"Paris is capital of France"

Reality is not that simple.

Representations are often distributed.

Information about one concept may involve many activations:

      Neuron A
       ↙    ↘
Neuron B    Neuron C
    ↓          ↓
      Neuron D

And one neuron may respond to several concepts.

One aspect of this phenomenon is called superposition.

It is like compressing a vast number of meanings into a limited representation space.

17. But we can find some internal structures

Interpretability research has found that neural networks can contain features/circuits associated with some kinds of concepts or behavior.

For example, conceptually:

quotation
code syntax
geographic concepts
sentiment
entities
grammatical relationships
...

But we should not think that we already have a complete brain map of an LLM.

We are still far from:

open model
↓
point to every parameter
↓
explain completely what it does

Large models remain systems that we can build better than we can fully explain their internal mechanisms.

That is a fact that is both amazing and a little unsettling 👀

18. Now we have the full picture from data to answer

Internet / Books / Code / Data
             ↓
      Clean + Deduplicate
             ↓
         Tokenize
             ↓
══════════════════════════════
        PRE-TRAINING
══════════════════════════════
             ↓
       Learn patterns
             ↓
     Memorization + 
     Generalization
             ↓
         Base Model
             ↓
══════════════════════════════
        POST-TRAINING
══════════════════════════════
             ↓
     Instruction following
     Preference learning
     Reasoning / RL
     Tool use
     Safety
             ↓
       Assistant Model
             ↓
══════════════════════════════
          INFERENCE
══════════════════════════════
             ↓
Prompt + Context + Memory
             ↓
      Reasoning compute
             ↓
      Retrieval / Tools
             ↓
        Verification
             ↓
           Answer

And at this point there is a question that goes one level deeper:

If an LLM is a Transformer, then what exactly is the “model” we call GPT-5.6, Claude, Gemini, and so on made of?

Because in modern AI systems, what we call a “model” on a product page should not always be imagined as one neural network that receives a prompt and emits an answer.

Next, we can unpack Tokenizer → Embedding → Transformer Block → Attention → MLP → Residual Stream → Logits → Softmax → Sampling, then follow one token from the moment it enters the door until it becomes an answer on the screen 🔬🧠

Reading notes

Clarification added to the conversation: The phrase “high predictability” in the section about LLMs memorizing information should not be used on its own as evidence of memorizing training data. An easy-to-predict passage can receive a high probability because of general patterns the model has learned. The frequency of examples, rarity or specificity of sequences, model size and prompting context all relate to the extraction of memorized training text.

Original research: Extracting Training Data from Large Language Models and The Secret Sharer.

← Previous: From a Base Model to an Assistant

Next: Following a Token Through a Transformer →

Read the series

Back to Notes