TALAMANA · THE AI LITERACY MAP FOR ARCHITECTURE AND DESIGN · Foundations · AGE 15—17 · FACTUAL · DURABLE
Train, validate, test
A model judged on the data it learned from has already seen the exam.
The idea
A model is taught on one pile of data. It is tuned on a second pile: the builders try settings, look at the results on this pile, and adjust. Then it is judged on a third pile it has never met. The three piles must stay separate. If the judging pile leaks into the teaching pile, the model has seen the exam before it sits it. The score goes up, and the model has not got better. That is why a published score can be higher than the model deserves, and a studio test much lower.
Why it matters
Every score you will ever be shown was produced on some pile. Whether that pile was truly separate decides whether the number means anything.
See it in the studio
A plan-reading tool is sold with a high score on a set of floor plans. You try it on your studio's plans and it misreads half of them. One likely reason: the test plans came from the same archive as the training plans, so the exam was full of drawings the model had already seen. Your plans were not in either pile.
Watch for this
Checking your own work against the examples you learned from. You practise on ten past jury sheets and then judge yourself on the same ten. You will look good. That is the same leak, done by hand.
Try it
Take twenty reference photographs of one building type. Work out a rule for recognising the type from the first fourteen. Then test your rule on the remaining six, and separately on the same fourteen. Write both scores down. The gap between them is what the separate piles are for.
Prove it
Explain why a model must be judged on data it never learned from, and say what goes wrong with the score when the two piles mix.
How it works
The textbook names are training set, validation set and test set. Google's course suggests a rough split of about 70, 15 and 15 per cent, and warns that the more often you use the test pile to make decisions, the less it can tell you about new data. Two failures live here. Overfitting: the model memorises the teaching pile instead of learning a pattern from it, so it does well on what it has seen and badly on anything new. Leakage: items from the test pile, or near-copies of them, sit in the training pile, so the test is no longer a test. For large models trained on much of the public internet, leakage is hard to rule out, because a benchmark published online may already be in the training data. That is one reason the benchmarks idea on this map asks what a score actually measured.
What this idea builds on
- A starting idea.
What this idea opens up
Sources
Open this idea on the map · The complete map · Logika · RBDS AI Lab, India · revised every edition.