AI Explorers · lesson 15 of 24 · 25 min
Today you get to break things on purpose. You will train a model with bad data and watch what happens. It is the fastest way to understand why data matters so much.
Why break a model?
Scientists learn by changing one thing and watching the result. In this lesson, you will run experiments where you change only the training data. Each experiment copies a real problem that happens in AI systems used by millions of people.
You can use an Image Project or an Audio Project in Teachable Machine. In an Audio Project, the tool asks you to first record background noise for a “Background Noise” class. Follow its on-screen instructions. Then each other class gets short clips, like a clap, a snap, or a word you say.
Experiment 1: The lopsided model
Train two classes, but give one class many more samples than the other. For example, 80 samples of “clap” and only 10 of “snap.”
Predict first: what will the model do when you snap?
What often happens: the model leans toward the bigger class. In the real world, this is what can happen when a dataset has lots of examples of some people or situations and very few of others.
Experiment 2: The shortcut model
For an image model, train “pencil” samples while wearing a red shirt and “pen” samples while wearing a blue shirt, with the shirt visible in the frame. Then switch shirts and test.
Predict first: what will the model use to decide?
What often happens: the model learns the shirt color instead of the object. This is the “hidden shortcut” from lesson 11, and you just created one.
Experiment 3: The mislabeled model
Put a few wrong samples in a class on purpose. For example, slip several “pen” samples into the “pencil” class. Train and test.
What often happens: confidence drops and the model mixes up the two classes more. Real datasets labeled by busy people can have errors like this.
What this means in real life
| Your experiment | Real-world version |
|---|---|
| Lopsided classes | A voice tool trained mostly on some accents works worse for others |
| Shortcut | A model learns the background or the camera, not the actual thing |
| Mislabeled data | Errors in labels quietly teach the model wrong answers |
The lesson: a model does exactly what its data teaches it, including the mistakes. When an AI system treats some people unfairly, look at the data first. You will build on this in Unit 7.
Try it
- Choose Image or Audio Project. Train a fair, balanced two-class model first and test it 10 times. Record accuracy as your baseline.
- Pick one of the three experiments above. Write your prediction before you train.
- Change only the training data, train, and test the same 10 times. Record accuracy.
- Compare to your baseline. Was your prediction right?
- Write three sentences: what went wrong, why, and one real-world system where this kind of data problem could hurt people.
Check yourself
Why should you write your prediction before running an experiment?
It makes you think about cause and effect and lets you check whether your understanding was right.
In the shortcut experiment, what did the model really learn?
The shirt color, not the object, because that was the easiest pattern to separate the classes.
If an AI system works worse for one group of people, where is a good first place to look?
The training data, to see if that group was missing, underrepresented or mislabeled.
Key words
- Baseline: the result you compare against before changing something.
- Experiment: a test where you change one thing and observe what happens.
- Underrepresented: having too few examples in the data.
- Mislabeled: tagged with the wrong label.
Previous lesson: Build and test an image model
Back to AI Explorers
Next lesson: How chatbots predict the next word