AI Explorers · lesson 23 of 24 · 25 min
Plans are done. Today your AI helper comes to life. Then you’ll do what too many real companies skip: test it carefully, and check it’s fair.
Step 3: build your prototype
Open Teachable Machine and pick the project type that matches your plan: image, audio or pose. Remember, you can train without signing in.
Follow your data plan:
- Create your classes, including a “nothing” or background class.
- Collect balanced samples with the variety you planned.
- Avoid the hidden shortcuts you listed.
- Keep your test items aside. Don’t train on them.
Train the model. Do a quick check in Preview to make sure it’s working at all before the real test.
Step 4: test it properly
Use your held-back test items. Run at least 5 tests per class, and record every result in your project journal:
| Test | True class | Model said | Confidence | Right? |
|---|---|---|---|---|
| 1 | ||||
| 2 |
Then calculate accuracy and make a mistake table (like the one from lesson 12). Write down your helper’s weakest spot. Is there a class it confuses a lot? A condition, like dim light or noise, that trips it up?
Make at least one improvement to the data and retrain. Rerun the same tests and record the new accuracy.
Step 5: the fairness check
Ask: who will use this helper, and does it work equally well for all of them?
- List the different users or situations your helper will meet. For a sound helper: different smoke alarms and doorbells, quiet and loud homes. For a hand-position helper: different skin tones, hand sizes, left- and right-handed players. For a sorting helper: different brands, crushed and uncrushed items.
- Test each group or situation separately, with at least 3 tests each.
- Compare accuracy across groups. A gap is a warning sign.
- If there’s a gap, figure out why (usually missing data) and fix it.
- Write down any group you could not test, and say so honestly in your pitch.
Also run a harm check: what is the worst thing that could happen if your helper is wrong? If a recycling helper is wrong, a can goes in the trash. If a smoke alarm helper misses an alarm, that is serious. High-stakes helpers need backup plans, like keeping the regular alarm sound on too.
Expect your first results to be messy. That is normal and it is part of the work. Professional teams go through many rounds of collecting data, training and testing before a system is ready. What matters is that you record what you tried, what happened, and what you changed. Your project journal is the proof of your thinking.
Try it
- Build your prototype in Teachable Machine using your data plan.
- Run at least 5 tests per class and calculate accuracy.
- Make one data improvement, retrain, and retest. Record before and after.
- Run the fairness check for at least two groups or situations and compare results.
- Write your harm check: the worst mistake your helper could make and your backup plan.
Check yourself
Why should you test each group separately?
Overall accuracy can hide that the helper works much worse for one group or situation.
What should you do if one group has lower accuracy?
Look for missing or poor data for that group, add better samples, retrain and retest.
What is a harm check?
Thinking about the worst thing that could happen if the AI is wrong, and planning a backup for it.
Key words
- Fairness check: testing whether a system works equally well for different groups.
- Harm check: thinking through what happens when the AI is wrong.
- Backup plan: a safe fallback if the AI fails.
- Weak spot: a situation where the model often makes mistakes.
Previous lesson: Capstone part 1: pick a problem and plan your data
Back to AI Explorers
Next lesson: Capstone part 3: pitch your AI helper