Temperature and sampling

In What AI actually is, you saw that a model writes by predicting the next word, one small piece at a time. This lesson zooms in on the moment of choice: how the model goes from "here are some likely next words" to "this is the word I am writing," and why that choice is where all the variety in AI answers comes from.

The model hands back a ranked list, not an answer

Here is the part most people never hear. When a model reaches the next spot in a sentence, it does not produce one word. It produces a ranked list of every word it knows, each with a chance attached. For "The sky over St. George tonight is," the list might put "clear" at the top, "beautiful" second, "orange" and "pink" a bit lower, and thousands of odd choices like "loud" way down at the bottom with tiny chances.

That list is the model's real output. Everything you read in a chat window is the result of a second, much simpler step that happens after the list is made: picking one word off it. That picking step is called sampling. Sampling just means choosing one item from the list in a way that respects the chances, so likely words get picked often and unlikely words get picked rarely, but not never.

Think of a raffle at a Hurricane church fundraiser. "Clear" bought forty tickets, "beautiful" bought twenty, "loud" bought one. Most draws go to the big ticket holders. Every so often, the one ticket wins. The model runs a raffle like that for every single word, then adds the winner to the sentence, builds a fresh list for the next spot, and runs another raffle. A three-paragraph answer is a few hundred raffles in a row.

Why not just take the top word every time? Because always taking the top word tends to produce flat, repetitive writing. The safest next word, chosen over and over, can loop on the same phrases or drift into bland filler. A little randomness keeps the writing varied and natural. That is the tradeoff the rest of this lesson is about.

Temperature: sharpen or flatten the list

Temperature is a setting that reshapes the ranked list before the raffle happens. It does not add new words or remove knowledge. It only changes how lopsided the chances are.

• Low temperature sharpens the list. The top word gets even more of the tickets and the long shots get almost none. The model becomes predictable and plays it safe.
• High temperature flattens the list. The top word loses some of its lead and the long shots get more tickets. The model becomes surprising, sometimes in a good way and sometimes in a nonsense way.

Try it below. Slide the dial and watch the bars change shape. Then hit "Sample 10 times" at each end and compare the tallies.

At the low end, "clear" wins almost every draw. At the high end, "stormy" and even "loud" start winning regularly. Notice that the order of the list never changes. "Clear" is always the favorite. Temperature only changes how much the favorite is favored.

One small word adds up fast. A slightly unusual pick early in an answer changes the list for every word after it, because each new list is built from everything written so far. Pick "stormy" instead of "clear" and the next sentence is now about monsoon season, not stargazing. That is why two answers to the same question can end up feeling completely different, not just a word or two apart.

The name comes loosely from physics, where hotter things jiggle around more. You do not need the physics. Hot means more jiggle, more variety. Cold means less jiggle, more of the same.

Top-p: cutting off the long tail

There is a second dial you will see mentioned next to temperature, called top-p (also called nucleus sampling). Instead of reshaping the chances, it trims the list. The model keeps the likeliest words, adding them from the top down until together they cover a set share of the total chance, and throws away everything below that line. With a tight top-p, only the few favorites stay in the raffle. With a loose one, more of the long shots stay in. The point is to stop a truly strange word from ever being drawn, even when temperature is turned up. OpenAI's own API documentation, as of October 2026, suggests adjusting temperature or top-p, but not both at once.

Why chat apps hide the dial

In ChatGPT, Claude, or Gemini, you generally do not set temperature yourself. The company running the app picks the setting for you and tunes it alongside everything else that shapes the answer, like the system prompt. If you want a more focused or more playful answer in a chat app, you ask for it in words: "give me the single most likely answer" or "give me ten wild ideas." The model reads that request and shifts its own choices.

The dial lives in the developer side of these products, the API (the direct line programmers use to send text to a model from their own software). Even there, it is fading. As of October 2026:

• OpenAI's API documents temperature as a setting from 0 to 2, with top-p as the alternative. Its guide for its newest models says to leave temperature and top-p out entirely when the model's reasoning (its "thinking" step) is turned on.
• Anthropic's API documents temperature from 0 to 1 with a default of 1, and marks temperature and top-p as deprecated (being phased out). Claude models released after Claude Opus 4.6 accept only the default temperature of 1 and reject other values.

The timing lines up with the rise of thinking models, the ones that write working notes before they answer. OpenAI ties its restriction directly to reasoning being turned on. The vendors do not spell out the full reason, but the pattern is clear. So the dial still explains how every model picks words, even when nobody outside the company is allowed to touch it.

Why the same prompt gives different answers, even at zero

Put the pieces together and the everyday puzzle solves itself. You send the same prompt twice. The model builds the same ranked list for the first word. Then it runs a raffle, and the raffle can land on a different word. From there each answer builds its own lists and runs its own raffles, and the two answers drift apart. Nothing is broken. That is sampling doing exactly what it was designed to do.

So surely turning temperature all the way to zero makes it repeat perfectly? At zero, the model is supposed to take the top word every time, no raffle at all. In practice it still is not fully repeatable. Anthropic's API documentation says so directly, as of October 2026: even at a temperature of 0, results "will not be fully deterministic," which is the technical way of saying the same input can still give a different output.

Why? The ranked list itself can shift by a hair from one run to the next. A model is served on big shared computers that handle many people's requests at once, grouped into batches. A September 2025 write-up from the AI research lab Thinking Machines traced most of this wobble to that grouping: how busy the server is changes how the work is split up, which changes the numbers by tiny rounding amounts. Usually that changes nothing. But when two words are nearly tied at the top, a hair is enough to flip which one wins, and from that word on, the answers go their separate ways.

The practical lesson: do not expect any AI tool to give you the exact same words twice, and do not build a habit around it. If you need a repeatable result, save the answer you liked rather than hoping to regenerate it.

Try this yourself

This takes a few minutes in any chat app you already use. Pick a question with one right answer and a question with many good answers, then ask each three times and compare. Use the regenerate or retry button, or start a fresh chat each time so the earlier answer does not influence the next one.

Question 1 (one right answer):
What is the capital of Utah? Answer in one word.

Question 2 (many good answers):
Write a one-sentence slogan for a landscaping company in Washington County, Utah.

Then look closely:

• The capital question should come back nearly identical every time. The top word on that list is so far ahead that the raffle almost never lands anywhere else.
• The slogan should change a lot. Many words are close together near the top of each list, so small draws send each run in its own direction.
• Now try a question you are not sure the model knows, like a detail about a small local business or event. If the three answers disagree with each other, treat that as a warning sign: the guesses are scattering. If they agree, that is not proof. As the hard part above showed, a model can land on the same wrong favorite every time, so check anything that matters against a real source.

That last check is one of the most useful habits you can take from this lesson. Asking twice costs you seconds and can tell you when not to trust the answer. For more ways to catch a confident wrong answer, see Trust and limits.