AI text isn't random — your decoding algorithm is secretly running the show

Most people think a language model writes text the way a person does — one word after another, guided by a coherent plan. It doesn’t. A model just spits out a vector of raw scores called logits for every possible next token. The actual text you see is the result of a decoding algorithm, a separate piece of engineering that turns those logits into a decision. This distinction matters because the algorithm shapes the model’s entire personality: whether it sounds boring, creative, or completely unhinged.

The simplest approach is greedy decoding. You always pick the token with the highest score. It’s deterministic, perfectly reproducible, and great for debugging. It’s also a creativity killer. A model doing greedy decoding will often trap itself in repetitive loops or reach for the most statistically common phrase every time, missing more interesting continuations. The chapter from the source material walks through building a greedy decoder from scratch using a small GPT-2 model from Hugging Face, showing that the logic is just an argmax call on the final logit vector.

To break that predictability, you introduce randomness via temperature sampling. Scaling the logits by a temperature parameter before applying softmax changes the shape of the probability distribution. A low temperature (close to 0) makes the model hyper-confident, effectively mimicking greedy decoding. A high temperature flattens the distribution so much that the model starts picking wildly improbable tokens. The trick is that temperature alone is a blunt instrument. That’s why practitioners layer on top-k sampling, which nukes all but the k most likely candidates, and nucleus (top-p) sampling, which dynamically keeps the smallest set of tokens whose cumulative probability hits a threshold like 0.9. This prevents the model from ever sampling the kind of complete nonsense token that lives in the long tail of a 100,000-word vocabulary.

The source explicitly connects these techniques to real tasks. Factual extraction wants a low temperature. Brainstorming wants a higher one. The combined sample_next_token function shown integrates temperature, top-k, and top-p filtering in sequence, and the order of operations matters because each step changes the distribution the next one sees. This is the practical wisdom that separates a janky chatbot from a reliable product. The chapter also covers repetition penalties, stop conditions, and structured output constraints for forcing JSON, but the core insight is philosophical: when you prompt an LLM, you’re not just talking to a neural network. You’re collaborating with a decoding strategy, and most of the magic (and the failure) lives there.