How do ChatGPT, Claude, and other LLMs actually generate text? This video breaks down the exact 5-step process that runs every time you hit send — from tokenization to the final word.
Most developers use LLMs daily without understanding what's happening under the hood. But knowing the mechanism changes how you prompt, debug, and build with these models.
In this video, you'll learn: → Why LLMs read tokens, not words (and why that matters for your API costs) → How embeddings create a "meaning space" where similar concepts cluster together → What attention actually does — with real examples you can visualize → Why the model genuinely doesn't know what it will say next → How temperature and top-p sampling shape your outputs
Timestamps:- 0:00 - The surprising truth about LLM generation 0:33 - The 5-step pipeline overview 0:58 - Step 1: Tokenization 2:04 - Step 2: Embeddings and meaning space 3:17 - Step 3: The Transformer and Attention 4:46 - Step 4: Probability distributions 5:46 - Step 5: Sampling, temperature, and top-p 07:05 - The autoregressive loop 08:05 - 3 practical insights you can use today
Every day, millions of people type prompts into chatGPT, Claude, or Grok and get responses
that feel almost human.
But most people don't realize?
The model has no idea what it's about to say.
Not the full sentence, not even the next word.
It's generating your response one piece at a time, and each piece is a probabilistic
guess from over a hundred thousand options.
In this video, we'll see exactly what happens from the moment you hit send to the moment
text appears, step by step.
So five things happen when you send a prompt.
1.
Tokenization.
Your text becomes pieces.
2.
Embeddings.
Those pieces become meaningful vectors.
3.
The transformer.
Context gets processed through attention.
4.
Probabilities.
Every possible next token gets a score.
5.
Sampling.
One token is selected.
Then it loops.
Let's look at each one in a bit more detail.
Step 1.
Tokenization.
LLMs don't read words.
They read tokens.
Here's OpenAI's tokenizer.
I type I love programming, it's awesome, and I get 7 tokens.
Notice most tokens are for the words, but there are separate tokens for the period.
This isn't random.
Tokenizers are trained on text data to find efficient patterns.
This happens before the model ever sees your input.
It's a pre-processing step, not the neural network deciding how to split.
Common words like the get one token.
Uncommon or long words get broken into sub-word pieces.
So indistinguishable, that's 4 tokens.
The just 1.
Why does this matter to you?
When an API says max 4096 tokens, that's not 4000 words.
It's roughly 3000 words of English.
Tokens are smaller units.
Every token gets a number, a token ID.
So I love programming, it's awesome becomes a sequence of 7 numbers, 7 integers.
That's what enters the model.
But numbers alone don't carry meaning.
That's step 2.
Embeddings.
A token ID is just a number.
The model needs to understand what it means.
So every token gets converted into a vector, a list of numbers representing its meaning.
These vectors have thousands of dimensions.
GPT-3 uses over 12,000 numbers per token.
And these aren't random numbers.
They're coordinates in a meaning space.
Think of it like this.
Words with similar meanings end up near each other.
King is near Queen.
Python the language is near JavaScript.
Python the snake is somewhere completely different.
There's a famous demonstration.
If you take the vector for king, subtract man, add woman, you land near Queen.
The model learned gender relationships just from text patterns.
Let me show you a more practical example.
Look at this embedding space for programming terms.
Function, method, procedure, clustered.
Variable, parameter, argument, clustered nearby.
Database, SQL, query, different cluster entirely.
This is how the model understands that JavaScript and Python are related.
Not because anyone told it, but because they appear in similar contexts.
These rich vectors now flow into the transformer.
Step three, the transformer.
Your embedding vectors enter a neural network with billions of parameters.
But I want to focus on the one mechanism that makes it all work, attention.
Imagine a spotlight operator at a concert.
The music shifts, the operator decides which musician to highlight.
During a guitar solo, spotlight on the guitarist.
During vocals, spotlight on the singer.
Attention works similarly.
When processing each token, the model decides which other tokens to focus on.
Take this sentence.
The cat sat on the mat because it was tired.
What does it refer to?
The cat, not the mat.
This is what attention does.
When the model processes it, it assigns high attention weight to cat and
low weight to mat, even though mat is closer in the sentence.
The model learned this from patterns across millions of examples.
It was tired, patterns match with animals, not objects.
This attention calculation happens multiple times in parallel,
through what are called attention heads.
Different heads can capture different relationships.
And then this whole layer repeats.
GPT-3 has 96 layers stacked.
LLAMA-3's 70 billion parameter model has 80.
Each layer refines the representation.
Each layer builds more abstract understanding.
What comes out?
Vectors that now encode not just individual token meanings, but
rich contextual information about the entire input.
Now we need to predict the next token.
Step four, probabilities.
The transformer has processed your input.
Now it needs to answer, what token comes next?
The final layer produces a score for
every token in the vocabulary, every single one.
LLAMA-3 has 128,000 tokens in its vocabulary.
Each gets a score.
These raw scores are called logits.
We apply a function called softmax to convert them into probabilities
that sum to one.
So for our input, we might get is 23% probability, really 14%,
the 9%, love 6%, and 127,996 more tokens with smaller probabilities.
This is the core reality of LLM generation.
The model doesn't decide what to say.
It produces a probability distribution over all possible next tokens.
Your final response is just one path through an enormous space of possibilities.
Now, how do we choose?
Step five, sampling.
This is where you have control.
The simplest approach, greedy decoding.
Pick the highest probability token every time.
Consistent?
Yes.
Boring?
Often.
That's where temperature comes in.
Temperature adjusts how confident the distribution is.
Same prompt, what is Python with different temperatures?
Low temperature sharpens the distribution.
Safe, predictable choices dominate.
High temperature flattens it.
Unlikely tokens get a real chance.
But push it too high and outputs often become incoherent.
That 1.5 example, already getting strange.
Then there's top p, also called nucleus sampling.
Top p says only sample from the smallest set of tokens
whose probabilities add up to p.
If top p is 0.9, you might be choosing from just 15 tokens or 500,
depending on how confident the model is.
Quick reference.
Writing code, temperature 0.2 to 0.4.
You want precision.
General tasks, temperature 0.7 to 1.0.
Balanced.
Creative writing, temperature 1.0 or higher.
Embrace variation.
When you set these parameters in an API call,
you're directly shaping this selection process.
One token selected.
Great, but we've only generated one token.
Last piece, the loop.
We generated one token.
Now we append it to the input and run the entire process again.
Tokenize, embed, transform, probabilities, sample for every single token.
What is Python?
First pass selects Python.
Now we have what is Python?
Python.
Second pass selects is.
Now we have what is Python?
Python is.
Third pass selects a.
And this continues until the model produces an end of sequence token
or hits a length limit.
This is why generation slows down for longer outputs.
Every new token requires attention over all previous tokens.
And this is why the model genuinely doesn't know what it will say in advance.
There's no hidden script, no planned sentence waiting to come out.
At token 10, token 50 hasn't been determined yet.
Each word is decided only when it's that word's turn,
based on everything that came before.
Now you understand what's actually happening.
Three insights you can use right away.
First, when LLMs hallucinate, they're not lying.
They're generating text that pattern matches
to what a confident, true-sounding response looks like.
The probability distribution doesn't know truth from plausibility.
The implication?
Always verify factual claims,
especially when the model sounds confident.
Second, temperature doesn't make models more creative.
It makes them more likely to select lower probability tokens.
Creativity is a human interpretation of that randomness.
The implication?
For deterministic tasks, coding, extraction, formatting,
use low temperature.
Don't leave it to chance.
Third, context limits aren't arbitrary product restrictions.
They're computational reality.
Attention has quadratic complexity.
Every token must attend to every other token.
The implication?
When you hit context limits,
it's not the company being stingy.
It's the architecture.
The next time you use an LLM, see what's happening.
Tokens in, meaning vectors, attention connecting context,
probabilities over a hundred thousand options,
one selection at a time.
It's not magic, it's mechanism.
Understanding the mechanism makes you a better builder.
If this helped, don't forget to like and share.
Thanks for watching.
See you in the next one.