TALAMANA · THE AI LITERACY MAP FOR ARCHITECTURE AND DESIGN · Generative Mechanics · AGE 17—19 · FACTUAL · EVOLVING
Attention, the model's glance back
The model weighs every earlier token before each new one, and a page outweighs a line.
The idea
Before it picks each new token, the model looks back over everything in the window and works out how strongly each earlier token should bear on what comes next. This weighing is called attention. It is not one dial per token: it runs in many separate heads in every layer, and each head weighs the tokens differently. A designer skimming a brief is a fair analogy, not the mechanism. What gets weight steers the result. That is why a rule stated once can still shape the answer twenty lines later — or be passed over.
Why it matters
"It read carelessly" is the wrong diagnosis. The useful one is "attention did not land on that line". The fix is to put the important line where it will be weighed, and to phrase it plainly.
See it in the studio
Your brief says "brick, load-bearing, no concrete frame" in the second paragraph, then talks for a page about the garden. The result has a concrete frame. The brick line did not vanish. A page about the garden outweighed it.
Watch for this
Padding a prompt with more words to be safe. More words means more things competing for the same attention. Say the important thing plainly, near where it is needed.
Try it
Write a brief with one unusual constraint buried in the middle. Run it. Move the constraint to the last line and run again. Compare which run kept it.
Prove it
Explain, with no maths, why a detail twenty lines back can still shape the next word — or get ignored — using the picture of someone skimming a brief.
How it works
Inside the transformer, every token works out how strongly it should attend to every other token, in several parallel "heads" at every layer, and each head learns a different pattern of relevance. Information also travels through the residual stream and the feed-forward layers, so attention is one part of the machine, not the whole of it. The weights are learned in training, not written by hand. EVOLVING: attention is the mechanism inside the transformer, the architecture behind nearly all of today's large models. Alternatives that handle long inputs differently are under active research. If they take over, this card's mechanism changes even though its lesson does not.
What this idea builds on
What this idea opens up
Sources
Open this idea on the map · The complete map · Logika · RBDS AI Lab, India · revised every edition.