TALAMANA · THE AI LITERACY MAP FOR ARCHITECTURE AND DESIGN · Generative Mechanics · AGE 17—19 · FACTUAL · EVOLVING

Why 'left of' fails

The captions the models learned from rarely say where things are.

The idea

Ask for "a stair left of the door" and many image models give you a stair and a door in any arrangement. The captions the models learned from rarely say where things are. People write "a stair and a door", not "a stair to the left of a door". Spatial words are rare in training data, so the model has had little practice. That is one cause, and it has been measured: spatially rich captions closed some of the gap. It is not the whole cause. How the model binds words to objects matters too.

Why it matters

If part of the failure is a data gap, you can work around it today with a control image or a sketch, and expect that part to shrink tomorrow. If you think it is random, you can do neither.

See it in the studio

"Courtyard at the centre, kitchen to the east, verandah facing the street" comes back as a handsome house with the kitchen anywhere. The words that describe a plan are exactly the words the model has seen least.

Watch for this

Writing longer spatial descriptions to fix a spatial failure. Using the rare words more often does not make them common to the model. Draw it instead.

Try it

Prompt "a stair" ten times, then "a stair left of the door" ten times. Score how often the second set is actually left. You now have a number for your tool.

Prove it

Explain why "a stair left of the door" is harder for a model than "a stair", and give one reason the same failure can improve with better training data.

How it works

The SPRIGHT work re-captioned millions of images with spatial detail and measured better spatial accuracy after training on them. CoMPaSS and others continue that line of work. EVOLVING for exactly that reason: the gap has been measured and it is moving. SPATIAL TEST on this strand gives you the procedure to measure it yourself.

What this idea builds on

What this idea opens up

Sources

Open this idea on the map · The complete map · Logika · RBDS AI Lab, India · revised every edition.

Age grows from 11 at the centre to 22 at the edge, and six sectors show the learning strands. Tab into the map and the arrow keys step from idea to idea, following the links where there is one. Enter opens the idea under the cursor, and E reads out its links and the reason recorded on each. Press slash for Search, question mark for the full key list, and Escape to leave. Open Ideas for the complete readable list, including what each idea builds on and what it opens up.

LOGIKA · RBDS AI LAB INDIA
ON-RAMP · AGE 11 · FIRST ENCOUNTERS, NOT GATES — IDEAS · — DEPENDENCIES
DONE
OPENS NEXT
SOLID — STANDS ON