Note / Essay

Talking with AI, Part 5: GPUs, VRAM and the Infrastructure Behind LLM Training

Where parameters live, why training consumes so much memory, and how GPUs divide the work, from numerical precision to mixture-of-experts models.

Part 5 of 11 in Talking with AI: From LLM Training to AGI

A conversation between Dr. Suttisak Denduangchai and ChatGPT-5.6 Reasoning Instant, translated from the original Thai chat. ChatGPT’s explanations and examples are preserved, with headings, tables and mathematical notation adjusted for reading on the web.

Read the complete series

Me: Tell me more.

ChatGPT-5.6 Reasoning Instant:

All right, this time let’s open the machine up and look at GPUs, VRAM and tensors ⚙️🧠

Our starting question is:

Why can training an LLM require thousands or tens of thousands of GPUs when the principle sounds as simple as “predict a token, then adjust the weights”?

Because adjusting those weights is the resource-intensive part.

1. What exactly is a parameter?

Suppose a tiny neural network uses the equation:

y = Wx

Here, W is a matrix:

W = [
  0.12   -0.73   0.44
  0.91    0.18  -0.31
  ...
]

Each number is a parameter the model learns.

An LLM contains many such matrices in its:

Embedding
Attention:
  Wq
  Wk
  Wv
  Wo

MLP:
  W1
  W2
  ...

× many Transformer layers

Together, these can add up to billions of parameters.

So when someone says:

“A 70B model”

they generally mean approximately:

70 billion parameters

That does not mean a 70 GB file.

2. How much RAM do 70B parameters need?

It depends on how many bits are used to store each parameter.

With FP32:

1 parameter = 32 bits
            = 4 bytes

That gives us:

70B × 4 ≈ 280 GB

The weights alone already occupy approximately 280 GB.

With BF16:

1 parameter = 2 bytes

The result is approximately:

70B × 2 = 140 GB

For inference, we may be able to quantize the weights further to 8-bit or 4-bit values.

Some rough examples:

Precision: FP32

  • 70B weights: ~280 GB

Precision: FP16/BF16

  • 70B weights: ~140 GB

Precision: INT8

  • 70B weights: ~70 GB

Precision: INT4

  • 70B weights: ~35 GB

These figures exclude other overhead.

This is why fitting a large model onto a single GPU is not straightforward.

3. Training is much more demanding than inference

For inference, the memory requirements are roughly:

Weights
+
Activations
+
KV Cache

Training adds some important items:

Weights
+
Activations
+
Gradients
+
Optimizer states

Suppose we use an Adam-like optimizer.

It needs to store additional information for each parameter to calculate its updates.

The picture becomes:

Parameter
   │
   ├── Weight
   │
   ├── Gradient
   │
   ├── Optimizer state #1
   │
   └── Optimizer state #2

So a model with 140 GB of weights cannot necessarily be trained with 140 GB of VRAM.

Its training footprint may be several times larger, depending on the precision, optimizer and techniques used.

4. What does the GPU actually do?

The core of a Transformer is full of this:

C = A × B

Matrix multiplication.

Vast amounts of it.

For example, in attention:

Q = XW_Q
K = XW_K
V = XW_V

Then:

Attention(Q,K,V) = softmax((QKᵀ) / √d) V

Matrix multiplication is everywhere.

GPUs are built to perform many computations in parallel.

A CPU is good at tasks such as:

Complex tasks
branch
OS
general-purpose workload

A GPU is good at:

Similar operations
on vast numbers of values
simultaneously

LLMs are therefore a particularly good fit for GPUs 🎯

5. Tensor cores

Modern GPUs designed for AI include hardware dedicated to matrix operations.

Conceptually:

Matrix A
   ×
Matrix B
   ↓
Tensor Core
   ↓
Matrix C

They can perform arithmetic at lower precisions such as:

FP16
BF16
FP8

very quickly.

Why not use FP64 for everything, as in some scientific computing workloads?

Because neural networks generally do not require that level of precision for every operation.

Lower precision can mean:

memory ↓
bandwidth ↓
compute ↑

Numerical stability still needs to be carefully controlled.

6. What is BF16?

A floating-point number can be roughly divided into:

Sign
Exponent
Mantissa

FP32:

[sign][ exponent ][      mantissa       ]
  1        8               23

BF16:

[sign][ exponent ][mantissa]
  1        8          7

BF16 retains an exponent range similar to FP32 while reducing the precision of the mantissa.

This suits neural networks because many operations do not need the fine numerical precision of FP32.

Modern training therefore often uses mixed precision:

Some operations → BF16 / FP8
Some operations → higher precision

The entire system does not have to use a single data type.

7. Why use multiple GPUs?

Suppose the model is too large to fit onto one GPU.

The simplest idea is to split it up:

GPU 1
[████████]

GPU 2
[████████]

GPU 3
[████████]

GPU 4
[████████]

In practice, things are more complicated because there are several ways to parallelize the work.

8. Data parallelism

Suppose we have 4 identical copies of the model:

GPU 1: Model → Batch A
GPU 2: Model → Batch B
GPU 3: Model → Batch C
GPU 4: Model → Batch D

Each GPU calculates its own gradients.

Then:

Gradient GPU1 ─┐
Gradient GPU2 ─┤
Gradient GPU3 ─┼→ synchronize
Gradient GPU4 ─┘
                 ↓
           Update model

The advantage is that we can process the training data faster.

But there is a catch:

Every GPU needs a copy of the model.

If the model cannot fit onto one GPU, this technique alone is not enough.

9. Tensor parallelism

Instead of copying the model, we split the same matrix across GPUs.

Suppose we have:

W =

████████████████
████████████████
████████████████
████████████████

We divide it up:

GPU 1
████

GPU 2
████

GPU 3
████

GPU 4
████

During computation, however, the GPUs need to exchange data.

That makes one thing particularly important:

Interconnect bandwidth.

A fast GPU is little use if it spends its time waiting for data from the GPU next door.

It is like having 10,000 chefs and a kitchen doorway only 70 cm wide 🍳

10. Pipeline parallelism

Another approach is to split the model by layer:

GPU 1
Layers 1–10
     ↓
GPU 2
Layers 11–20
     ↓
GPU 3
Layers 21–30
     ↓
GPU 4
Layers 31–40

Data moves through a pipeline.

To keep other GPUs from sitting idle, a batch can be divided into microbatches:

Microbatch 1 → GPU1 → GPU2 → GPU3 → GPU4
Microbatch 2 → GPU1 → GPU2 → GPU3 → GPU4
Microbatch 3 → ...

Like a factory production line.

11. Real systems combine several techniques

A large training cluster might look roughly like this:

                 Thousands of GPUs
                         │
        ┌────────────────┼────────────────┐
        │                │                │
   Data Parallel    Tensor Parallel   Pipeline
        │                │                │
        └────────────────┼────────────────┘
                         │
                  Distributed Training

It can also use techniques that shard:

Weights
Gradients
Optimizer states

to reduce duplication in memory.

This is where families of techniques such as FSDP and ZeRO come in.

12. The network matters as much as the GPUs

Suppose a GPU finishes its computation in:

10 ms

but then waits for synchronization for:

30 ms

That expensive GPU is indulging in a luxury activity:

Waiting. 🫠

A training cluster therefore needs a high-speed interconnect or network.

Roughly:

GPU ─── GPU
 │       │
GPU ─── GPU
 │       │
────────────
High-speed fabric
────────────
 │       │
GPU ─── GPU

This is why building AI infrastructure involves more than:

Buying lots of GPUs.

It is a full-scale distributed systems problem.

13. How does the dataset reach the GPUs?

The data pipeline is another part of the picture:

Storage
   ↓
Training dataset
   ↓
Tokenizer
   ↓
Token sequences
   ↓
Batch
   ↓
GPU cluster

If storage is slow to read:

GPU:
"Where is the data..." 🧍

More compute goes to waste.

So all of these need attention:

Storage throughput
Preprocessing
Tokenization
Batching
Caching
Networking

14. The actual training loop is coming into focus

We can now expand the original simplified description:

Predict a token
↓
Prediction error
↓
Adjust the weights

into:

Dataset
   ↓
Tokenization
   ↓
Create batch
   ↓
Distribute across GPU cluster
   ↓
Forward pass
   ↓
Predicted token probabilities
   ↓
Calculate loss
   ↓
Backward pass
   ↓
Gradients
   ↓
Synchronize across GPUs
   ↓
Optimizer step
   ↓
Update billions of parameters
   ↓
Next batch

        ↺

Repeated an enormous number of times.

15. How do the forward and backward passes differ?

Forward pass

Input
 ↓
Layer 1
 ↓
Layer 2
 ↓
...
 ↓
Prediction

This resembles what happens during inference.

Training adds a:

Backward pass

Loss
 ↑
Layer N
 ↑
...
 ↑
Layer 2
 ↑
Layer 1

to calculate:

∂Loss / ∂Weight

In plain language:

How much did this weight contribute to the prediction error, and in which direction should it move?

The optimizer then performs an update roughly like this:

W_new = W_old − η ∇L

Here, η is the learning rate.

This is the stage where the model actually learns.

16. This makes training expensive

It requires:

Forward
+
Backward
+
Optimizer
+
Communication

across:

billions of parameters
×
billions/trillions of tokens

The number of operations becomes enormous.

Training a frontier model is therefore not a matter of:

“Start Python and leave it running overnight.”

It is a major infrastructure project:

Dataset
     │
     ▼
Storage
     │
     ▼
Thousands of accelerators
     │
     ▼
High-speed network
     │
     ▼
Distributed training
     │
     ▼
Checkpoints
     │
     ▼
Evaluation

And if training crashes after a long run without a good checkpoint...

💸🔥

17. A checkpoint is an LLM’s saved game

The training state is saved periodically:

Step 100,000
   ↓
checkpoint

Step 110,000
   ↓
checkpoint

Step 120,000
   ↓
💥 node failure

You can return to:

checkpoint 110,000

instead of starting over.

A checkpoint may need to store both the model and the training state required to resume, so it can be very large.

18. What is MoE?

This is an interesting one.

Mixture of Experts.

Instead of sending every token through all the same parts of the neural network:

Token
 ↓
Huge network
 ↓
Output

we have multiple experts:

                 ┌→ Expert 1
                 │
Token → Router ──┼→ Expert 2
                 │
                 ├→ Expert 3
                 │
                 └→ Expert 4

A router selects a subset of experts for a particular token.

For example, conceptually:

Token X
 ↓
Expert 2 + Expert 7

Not every expert needs to be activated.

This allows:

A very large total parameter count.

while using:

Fewer active parameters per token.

This is called sparse activation.

19. Do not interpret “expert” this way

It does not necessarily mean:

Expert 1 = Doctor
Expert 2 = Programmer
Expert 3 = Thai
Expert 4 = Lawyer

❌ We should not take the term that literally.

Experts are parts of the neural network that learn specializations through optimization. Those specializations may not align with categories that humans can easily name.

20. MoE trades compute for complexity

The advantage:

Total model capacity ↑↑
Compute per token need not ↑ proportionally

But it introduces new challenges:

Routing
Load balancing
Communication
Expert placement
Memory
Training stability

Suppose every token heads for Expert #7:

Expert 1 😴
Expert 2 😴
Expert 3 😴

Expert 7
🔥🔥🔥🔥🔥🔥🔥

The system needs to work to balance the workload.

The bigger picture is coming together

What we casually call:

Training an LLM

actually looks like this underneath:

                 INTERNET / DATA
                       │
                       ▼
                DATA PIPELINE
                       │
                       ▼
                  TOKENIZER
                       │
                       ▼
                TRAINING BATCH
                       │
                       ▼
        ┌─────────────────────────────┐
        │     GPU/AI ACCELERATOR      │
        │          CLUSTER            │
        │                             │
        │ Tensor Parallel             │
        │ Pipeline Parallel           │
        │ Data Parallel               │
        │ Sharding                    │
        └──────────────┬──────────────┘
                       │
              Forward / Backward
                       │
                       ▼
                    LOSS
                       │
                       ▼
                  GRADIENTS
                       │
                       ▼
                  OPTIMIZER
                       │
                       ▼
              UPDATE PARAMETERS
                       │
                       │
                       └──────────↺

Repeat until we have:

Base LLM

But this is not yet a complete ChatGPT-like assistant.

It may have extensive knowledge, but it has not yet been adapted to follow instructions, use tools, give helpful answers or handle reasoning tasks in the desired way.

That leads naturally to the next important chapter:

Once pre-training is finished, what happens to the base model to turn it from an “expensive text prediction machine” into ChatGPT?

We will encounter SFT → RLHF → Reward Model → RL → reasoning training → tool-use training → safety training. We will also see why a system prompt is not training, two things people often confuse 🧠🔧

← Previous: Why More Thinking Can Improve an Answer

Next: From a Base Model to an Assistant →

Read the series

  1. Part 1: What Does Training an LLM Mean?
  2. Part 2: How Can Predicting the Next Token Lead to Coding?
  3. Part 3: Context, Attention and Memory During a Conversation
  4. Part 4: Why More Thinking Can Improve an Answer
  5. Part 5: GPUs, VRAM and the Infrastructure Behind LLM Training (this episode)
  6. Part 6: From a Base Model to an Assistant
  7. Part 7: Does an LLM Memorize, Learn or Guess?
  8. Part 8: Following a Token Through a Transformer
  9. Part 9: From an LLM to an AI System and Agent
  10. Part 10: More Compute for Training or for Answering?
  11. Part 11: From More Capable LLMs to the Question of AGI

Back to Notes