AI Explorers · lesson 12 of 24 · 25 min
Studying the exact answers to a practice quiz feels great until the real test has different questions. Models can fall into the same trap, so we test them carefully.
Train data vs. test data
If you test a model on the same examples it trained on, it might look perfect just because it memorized them. That tells you nothing about how it will do in the real world.
So we split the data. Most examples go into the training set. Some are held back in a test set the model never sees during training. After training, we check the model on the test set. That is like a fair exam with new questions.
Measuring accuracy
Accuracy is the share of test examples the model gets right.
Example: a model sorts 20 test photos into “apple” and “orange.” It gets 17 right and 3 wrong. Accuracy = 17 ÷ 20 = 85%.
But accuracy alone can fool you. Imagine a model that is supposed to spot a rare problem, like a broken part in a factory, and broken parts are only 1 out of every 100. A lazy model that always says “not broken” would be 99% accurate and completely useless. So we also look at which mistakes it makes.
Types of mistakes
A simple table shows what the model predicted vs. what was true. This is a small version of what data scientists call a confusion matrix.
| Model said “apple” | Model said “orange” | |
|---|---|---|
| Really apple | 9 (right) | 1 (wrong) |
| Really orange | 2 (wrong) | 8 (right) |
From this table, you can see the model mixes up oranges as apples more often than the other way around. That is a clue. Maybe the orange photos were darker, or some were green.
Some mistakes matter more than others. A spam filter that lets one junk email through is annoying. A spam filter that hides an important message from your coach is worse. People decide which mistakes are most important to avoid.
Overfitting
Overfitting is when a model learns the training examples too exactly, including random details, and then does badly on new data. It is like memorizing that question 4 is “C” instead of learning the math. Signs of overfitting: very high accuracy on training data, much lower on test data. Fixes include more varied data and simpler models.
Testing does not stop after a model is launched. The world changes. New phone cameras take different-looking photos, people start using new slang, and seasons change the lighting. A model that tested well last year can slowly get worse. Good teams keep checking their models with fresh examples and retrain them when the results start to slip.
Try it
Test the “model” in your head.
- Ask a friend or family member to be your model. Teach them a category with 8 examples, such as “foods that count as fruit” (include tricky ones like tomato and avocado if you like). Tell them the label for each.
- Prepare 10 new test items they have not heard. Write down the true label for each before you start.
- Read the test items one at a time. Record their answers.
- Calculate accuracy: number right ÷ 10.
- Fill in a 2 by 2 mistake table like the one in this lesson. Which kind of mistake did they make more often? What extra training example would fix it?
Check yourself
Why do we keep a test set separate from training data?
So we can check the model on examples it has never seen, which shows how it will work in real life.
A model gets 45 of 50 test examples right. What is its accuracy?
45 ÷ 50 = 90%.
What is overfitting?
When a model memorizes training examples too closely and then does poorly on new data.
Key words
- Training set: examples used to train a model.
- Test set: examples held back to check the model.
- Accuracy: the share of predictions that are correct.
- Confusion matrix: a table of what the model predicted vs. what was true.
- Overfitting: learning the training data too exactly to handle new data.
Previous lesson: Training data and labels
Back to AI Explorers
Next lesson: Meet Teachable Machine