Learngen-aiHow Chatbots Write

Lesson · 8 min

How Chatbots Write

Next-token prediction: the engine behind the magic trick.

A chatbot like ChatGPT or a local Qwen model generates text one small piece at a time. For each next piece it computes a probability over its whole vocabulary: "given everything so far, what likely comes next?" Pick, append, repeat. That’s the whole trick.

What this explains

  • Why answers sound human: training text WAS human writing.
  • Why answers can be wrong: high probability ≠ true. A confident wrong answer is called a hallucination.
  • Why phrasing changes answers: you change the pattern it is continuing.
  • Why it can’t "look things up" unless it’s given tools or sources.
Hallucinations are a side effect of predicting likely text rather than rare bugs. Verification is a core skill in this module for that reason.

Try it yourself

Try it — order

Assemble the next-token prediction loop.

1Pick one likely token
2Compute a probability over every possible next token
3Read everything so far in the conversation
4Repeat from the top until the answer is complete
5Append it to the text

Knowledge check

1

A chatbot confidently cites a research paper that doesn’t exist. This is called…

2

Why can rewording a prompt change the answer?