Attention: what the model looks at

You already know the model's main job from What AI actually is: predict the next word, over and over. This lesson is about the step that makes those predictions good. To guess the next word well, the model has to know which of the earlier words actually matter for it, and which ones it can mostly ignore. The part of the model that does that sorting is called attention.

The problem: which earlier words matter?

Read this sentence:

The trailer would not fit in the garage because it was too big.

What was too big? You did not even pause. The trailer. Now change one word:

The trailer would not fit in the garage because it was too small.

Now "it" is the garage. Same sentence, same "it," and the meaning flipped because of a word that came after it. Grammar alone does not settle this. Both "trailer" and "garage" are things, both came before "it," and either could be the one being talked about. You worked it out by connecting "big" and "small" to what you know about trailers and garages.

A model has the same problem on every single word. Some earlier words are crucial for understanding the current one, and most are not. The word "the" tells you very little. The word "trailer" tells you a lot. A model that treated every earlier word as equally important would drown in noise, especially in a long conversation. Attention is how it decides what to focus on.

Every word looks back

Here is the whole idea, with no math. When the model reads your message, each word gets a turn to look back at every word before it and ask, "How much do you matter to me?" It then gives each earlier word a share of its focus. A few words get a big share. Most get a sliver. That set of shares is the attention for that word.

Then each word borrows meaning from the words it focused on. If you read the embeddings lesson, every word starts as a point on a map of meaning. Attention nudges that point. After looking back, "big" in our sentence is no longer just "big" in general. It is "big, and this is about the trailer not fitting." Every word gets this treatment, so by the time the model predicts what comes next, each word carries a little of the context around it.

One detail matters for the trailer sentence. The chat models you use read left to right. A word can look back at everything before it, but never ahead. So when the model reaches "it," it has not seen "big" or "small" yet, and the honest answer at that moment is "could be either." The question gets settled a few words later, when "big" looks back and leans hard on "trailer" and on "it" together. Try it below.

A picture like this is called a heatmap: darker shading means more focus. Researchers draw them to peek inside real models. The one above is hand-made to show the idea, but the shape is the same. Tap "it" and the attention splits between the two candidates. Tap the last word and swap "big" for "small", and the strongest pull jumps from "trailer" to "garage."

Many heads, each watching for something different

You probably noticed the buttons for different heads in the explainer. An attention head is one separate pattern of looking back. A model does not run attention once. It runs many heads side by side, and each one learns to watch for a different kind of connection.

Think of a group of people reading the same page with different highlighters. One person highlights who is who: which "it" or "she" points to which person or thing. Another highlights grammar: which verb goes with which subject. Another mostly checks the word right next door. None of them sees the whole picture, but put their highlights together and the page is well understood.

Then the model stacks these rounds. The output of one round of attention feeds the next round, which looks back again with fresh eyes. Early rounds tend to pick up simple, local patterns. Later rounds can build on those to track bigger things, like the thread of an argument or who is speaking in a story. A large model repeats this many times before it settles on the next word.

The heads in the explainer have tidy names because we made them up. Real heads do not come with labels. Researchers have to study them to guess what each one tracks, and plenty of them do things nobody can name. Which brings us to the most surprising part.

Why long conversations cost more

Remember that every word looks back at every earlier word. That is what makes attention powerful, and it is also what makes it expensive. In Tokens, context, and system prompts you met the context window, the amount of text a model can hold at once. Attention is a big reason that window has a limit.

Picture a meeting where, before anyone speaks, they must first glance at everyone who already spoke. With five people that is quick. With five hundred, each new speaker has five hundred glances to make, and the room as a whole makes a staggering number. Double the length of a conversation and the looking back grows by much more than double, because there are more words doing the looking and more words to look at.

This explains a few things you may have noticed in chat apps. Very long chats can feel slower. Long documents use more of your usage limit. And a model can lose track of a detail buried deep in a huge conversation, because that detail is competing for focus with everything else. AI labs have found many clever ways to make long context cheaper and better, which is why context windows have grown so much, but the basic pressure never fully goes away. Keeping a chat focused, or starting a fresh one, still helps.

A short history: one paper changed everything

Attention had its big moment in June 2017. A team of eight researchers, mostly at Google, published a paper with a bold title: "Attention Is All You Need." They were working on translation between English and German, and English and French. Older translation systems read a sentence strictly one word at a time, carrying a running summary forward, which made them slow to train and prone to losing track in long sentences.

The paper threw out that word-by-word design and built a model out of attention, so every word could look directly at the others. They called the design the transformer. It used eight attention heads side by side, the same many-highlighters idea from above, and it trained in days on a handful of computer chips. The paper even included its own heatmaps, showing heads that appeared to work out which noun a word like "its" referred to.

Nearly every chat model you use today is a descendant of that design, scaled up enormously. When people say modern AI started in 2017, this is the paper they mean.

Try this yourself

You cannot see a chat model's attention directly, but you can watch it work. Open any chat app you already use and paste this:

Two sentences:
1. The trailer would not fit in the garage because it was too big.
2. The trailer would not fit in the garage because it was too small.

In each sentence, what does "it" refer to? Which word in the sentence decided it for you?

Notice that it gets both right and points to "big" and "small" as the deciding words. That is the same connection the heatmap showed. Then make it harder. Write your own ambiguous sentence, something like "The landscaper called the homeowner because she was running late," and ask who "she" is. Watch whether the model notices that the sentence is genuinely unclear, or picks one and sounds sure about it.

One more observation, for a long chat you already have open. Ask the model to quote back a small detail you gave it near the very start. Then start a fresh chat, give it only that detail, and ask again. Notice whether the answer in the long chat is a little fuzzier. That is attention being shared among a lot of words.

What to remember:

• Attention is each word looking back at earlier words and deciding how much each one matters.
• Many heads look back at once, each tracking a different kind of connection, and the model stacks many rounds of this.
• Those habits are learned in training, not written by anyone, which is why they are powerful and sometimes strange.
• Every word looking at every earlier word is why long context costs more and can get fuzzy.
• The 2017 paper "Attention Is All You Need" introduced the design behind today's chat models.