How LLMs Generate Text

Watch generation happen one token at a time, and learn what the temperature setting actually controls.

Have you noticed that AI answers appear word by word, like someone typing fast? That's not a loading animation. It's literally how the model works.

When you send a prompt, the model runs a loop:

1. Tokenize the input (split it into small chunks called tokens; next lesson!)
2. Process all tokens through its neural network layers
3. Predict a probability for every possible next token
4. Sample one token from those probabilities
5. Append it to the text and go back to step 2, until a stop signal

This loop is called autoregressive generation: the model builds its answer one piece at a time, each new piece computed from everything before it, including the pieces it just wrote.

Step 3 is worth slowing down on. The model doesn't pick "the answer". It produces a probability distribution: a score for every token it could emit next.

Prompt: "The capital of France is"

Next-token probabilities (illustrative):
  " Paris"     92%
  " located"    3%
  " the"        2%
  " Lyon"     0.1%
  ...thousands of other tokens sharing the rest

Then step 4, sampling, picks one token from that distribution. Usually it picks a high-probability token, but "usually" is doing real work in that sentence, and there's a dial that controls it.

Multiple choice

At each step of generation, what does the model actually compute before a token is chosen?

AA probability distribution over every possible next token
BThe single correct next word, looked up from its training text
CThe full remaining answer all at once
DA list of documents that match the prompt

The dial is called temperature, and it controls the sampling step.

Low temperature (near 0): the model almost always picks the most likely token. Output is focused, predictable, and nearly the same on every run.
Higher temperature: lower-probability tokens get more of a chance. Output is more varied and creative, and also more random and error-prone.

Prompt: "Write a tagline for a coffee shop."

temperature ≈ 0:  "Great coffee, made fresh daily."   (safe, repeatable)
temperature ≈ 1:  "Where the espresso plots your
                   productivity."                      (surprising, riskier)

Rule of thumb: use low temperature for extraction, classification, code, and anything you'll parse with a program. Raise it only when you want variety, like brainstorming names or writing fiction.

Multiple choice

You're building a feature that extracts the invoice number and total from emails, and your code parses the model's output. Which temperature setting should you use?

ALow (near 0), so the output is consistent and reliably parseable
BHigh (near 1), so the model can think more creatively about the invoice
CAlternate between low and high on every request
DTemperature doesn't affect anything except response speed

Predict the output

You send the same prompt to the same model four times: twice at temperature 0, twice at temperature 1. Which pair of responses is more likely to differ noticeably from each other?

Run A1, A2: temperature = 0
Run B1, B2: temperature = 1

AB1 vs B2 (the temperature-1 pair)
BA1 vs A2 (the temperature-0 pair)
CBoth pairs differ equally, temperature only changes speed
DNeither: the same prompt always gives the same answer

One practical consequence of the token-by-token loop: output length drives latency and cost.

Every output token requires a full pass through the network. A 500-token answer takes roughly ten times as many generation steps as a 50-token answer. That's why:

Streaming UIs show tokens as they're generated, so users aren't staring at a blank screen
Asking for concise output ("answer in one paragraph") is a real performance optimization
The API (the web service you send prompts to and get responses back from) lets you set a maximum output length as a safety cap

Generation stops when the model emits a special stop token (its learned "I'm done" signal) or hits that maximum.

Multiple choice

Why does a 1,000-token response take noticeably longer to generate than a 100-token response to the same prompt?

AEach output token requires its own full pass through the network, so 10× the tokens means roughly 10× the generation steps
BLonger answers require the model to search more of its database
CThe model writes the whole answer instantly but reveals it slowly for readability
DIt doesn't: response time depends only on the input length

Recap 🧠

Generation is autoregressive: predict, sample, append, repeat, one token at a time until a stop token or length cap
At each step the model computes a probability distribution over every possible next token, then samples one from it
Temperature controls sampling: near 0 = focused and repeatable, higher = varied and riskier
Use low temperature for extraction, classification, and code; raise it only when you want variety
Output length drives latency and cost, because every output token is a full pass through the network

We keep saying "token". Time to define it properly, because tokens are also how you get billed.

Have you noticed that AI answers appear word by word, like someone typing fast? That's not a loading animation. It's literally how the model works.

When you send a prompt, the model runs a loop:

This loop is called autoregressive generation: the model builds its answer one piece at a time, each new piece computed from everything before it, including the pieces it just wrote.

Prompt: "The capital of France is" Next-token probabilities (illustrative): " Paris" 92% " located" 3% " the" 2% " Lyon" 0.1% ...thousands of other tokens sharing the rest

Prompt: "Write a tagline for a coffee shop." temperature ≈ 0: "Great coffee, made fresh daily." (safe, repeatable) temperature ≈ 1: "Where the espresso plots your productivity." (surprising, riskier)