TALAMANA · THE AI LITERACY MAP FOR ARCHITECTURE AND DESIGN · Foundations · AGE 16—18 · FACTUAL · DURABLE

Why did it do that?

The model's explanation is generated the same way as its answer.

The idea

Some systems can show their working: a rule fired, a factor weighed. Deep models mostly cannot. The "reason" is spread across millions of weights, and there is no explanation in sentences anywhere inside. Knowing a tool is a black box does not forbid using it. It changes how much you should lean on it before checking the result yourself.

Why it matters

Professional decisions need reasons that survive questioning: a jury, a client, a court. An answer with no findable reason can feed your thinking, but it cannot carry a decision. Something explainable must carry it.

See it in the studio

A model ranks your three options and prefers B. Ask it why, and it produces a fluent paragraph. That paragraph was generated the same way as the ranking. It is not a window into it. You still do not know why B. You only know what reasons for B tend to sound like.

Watch for this

Mistaking a generated explanation for the actual mechanism. The model explains its answers the way it does everything: by predicting plausible text.

Try it

Ask a model to choose between two options and explain why. Then swap the labels and ask again. If the explanation follows the labels rather than the content, you have seen the black box for yourself.

Prove it

Explain the difference between a system that can show its working and one that can only generate a story about its working.

How it works

A serious research argument says that where the stakes are high, you should prefer models that are interpretable by construction over explanations added after the fact to black boxes. Interpretability research is real and advancing, and honest about how far it has to go. For visual models there are tools that let you watch the process. That is not the same as reading the reasons.

What this idea builds on

What this idea opens up

Sources

Open this idea on the map · The complete map · Logika · RBDS AI Lab, India · revised every edition.

Age grows from 11 at the centre to 22 at the edge, and six sectors show the learning strands. Tab into the map and the arrow keys step from idea to idea, following the links where there is one. Enter opens the idea under the cursor, and E reads out its links and the reason recorded on each. Press slash for Search, question mark for the full key list, and Escape to leave. Open Ideas for the complete readable list, including what each idea builds on and what it opens up.

LOGIKA · RBDS AI LAB INDIA
ON-RAMP · AGE 11 · FIRST ENCOUNTERS, NOT GATES — IDEAS · — DEPENDENCIES
DONE
OPENS NEXT
SOLID — STANDS ON