TALAMANA · THE AI LITERACY MAP FOR ARCHITECTURE AND DESIGN · Generative Mechanics · AGE 18—20 · FACTUAL · EVOLVING
One window, both modalities
Your image enters the window as tokens, and one attention stream weighs both.
The idea
In many current multimodal chat models, a vision encoder turns your uploaded image into patches, a small trained projector turns them into tokens the language model can accept, and they enter the same sequence as your words. One set of attention weights then runs over the mixed stream, weighing a patch of the plan against a word in your question. That is why such a model can answer a question needing picture and words together, and why it can get both wrong at once. This is the fourth number-space: a conversion into the language model's own tokens.
Why it matters
Everything you know about attention now applies to your image as well as your text. A large image is a lot of tokens competing with your question.
See it in the studio
You upload a ground-floor plan and ask "which rooms get morning light?" The model reads "north" from your title block, picks out windows from the patches, and reasons across both. When it is wrong, it is often because the patch it weighed was the wrong one, not because "the image side" failed.
Watch for this
Thinking in the translator picture: "it describes the image to itself, then answers." Some older systems did that, and some pipelines still do. The stronger models reason over the mixed stream directly. The difference shows on questions that need spatial detail.
Try it
Upload a plan and ask one question that needs only the text on it (what is the room labelled at top-left?), one that needs only the geometry (how many doors open onto the corridor?), and one that needs both. Note which fail.
Prove it
Explain, without using a "translator" picture, how a model reasons over an uploaded plan and a written question together — and tell apart a CLIP-style comparison space (SHARED SPACE) from image tokens entering a language model's window.
How it works
Visual instruction tuning (LLaVA was an early open example) runs the image through a vision encoder, projects the features into the language model's token space with a small trained layer, and trains on image–question–answer data. Newer models train on mixed streams from the start. EVOLVING: architectures differ — projectors, cross-attention, native mixed training — and which design dominates changes year to year. The lesson about attention over a mixed stream holds across all of them.
What this idea builds on
What this idea opens up
Sources
Open this idea on the map · The complete map · Logika · RBDS AI Lab, India · revised every edition.