TALAMANA · THE AI LITERACY MAP FOR ARCHITECTURE AND DESIGN · Foundations · AGE 17—19 · FACTUAL · DURABLE
Attention becomes the architecture
Most chatbots and many image tools share one 2017 ancestor, and so share its limits.
The idea
In June 2017 eight researchers at Google published a paper called Attention Is All You Need. Attention itself was not new: translation systems had used it since 2014 to let a model look back at the parts of a sentence that mattered. What the paper did was make attention the whole architecture — the Transformer — and drop the step-by-step reading earlier designs relied on. The design trained efficiently on parallel hardware and grew with data and compute in a way earlier designs had not. Most large language models today, and many image and video generators, are built on it.
Why it matters
When most chatbots and many image tools share one ancestor, they share its habits and limits too. Knowing the ancestor explains why tools from rival companies fail in such similar ways.
See it in the studio
The weighing that lets a chatbot hold a long brief in view is what lets an image model connect "brick" to the right wall in your prompt. It is also what makes both lose the thread when the brief gets long. Same ancestor, same strength, same weakness.
Watch for this
"Attention arrived in 2017." It did not; the Transformer did. And treating "the Transformer" as the whole explanation. The design made scaling possible. The data and the compute did the scaling. This is a history card: it says what happened. The mechanics of attention live in Generative Mechanics.
Try it
Find out which model sits behind three tools you use — a chatbot, an image generator, a search assistant. Check whether each is described as a transformer or as built on one. Expect yes more often than not, and note the exceptions.
Prove it
Say what the 2017 paper proposed, in one sentence, say what was already there before it, and name two kinds of tool on this map that descend from it.
How it works
The title is a claim: attention alone, without the recurrent or convolutional structures earlier language systems relied on, was enough to reach the state of the art on translation, and faster. The paper itself says the best earlier models already used attention alongside recurrence; the novelty was removing everything else. Because attention can be computed for all positions at once, it suited graphics processors, and that is what let the design grow. Later work applied it to images by treating image patches like words, then to audio and video. The mechanics of the weighing, and what it costs when the input gets long, are taken up by the attention and context-window cards in Generative Mechanics.
Lineage
2012 put learning from examples in front. 2014 brought attention into translation. 2017 made it the whole design, and that design could scale. The years after supplied the data, the compute and the money. The boom is those three things building on each other.
What this idea builds on
What this idea opens up
Sources
Open this idea on the map · The complete map · Logika · RBDS AI Lab, India · revised every edition.