AI Explorers · lesson 19 of 24 · 25 min
Imagine a soap dispenser that works for your friend’s hand but not yours. Or a photo app that can’t find your face in a group picture. When technology works better for some people than others, that’s a fairness problem.
What is AI bias?
Bias in AI means a system’s results are unfair or less accurate for certain groups of people. It is usually not because someone typed “be unfair.” It creeps in through choices and data.
Common sources of bias:
- Training data that leaves people out: if a face system learns mostly from some faces, it works worse on others. You saw this in your lopsided-model experiment.
- Data that reflects past unfairness: if a hiring tool learns from a company’s past hires, and that company mostly hired one kind of person, the tool may copy that pattern.
- Labels from people: human opinions in labels can carry stereotypes.
- Features that act as stand-ins: something like a zip code can quietly stand in for race or income.
- Who is in the room: if the team building a product does not include people from many backgrounds, they may never test it on situations they don’t experience.
A real example: Gender Shades
In 2018, researcher Joy Buolamwini, working with Timnit Gebru, published a study called Gender Shades. They tested commercial face analysis systems from major tech companies that tried to guess whether a face was male or female.
They found the systems were much less accurate for darker-skinned women than for lighter-skinned men. The study showed that the test datasets commonly used to measure these systems did not include enough people with darker skin. After the research got attention, some of the companies updated their systems.
Joy Buolamwini founded the Algorithmic Justice League, an organization that works to raise awareness about the harms of biased AI. Her work is one example of how Black technologists and researchers are shaping the fight for fair AI, and why it matters who gets to build and question these tools.
What does “fair” mean?
Fairness is not always simple. Should a system have the same accuracy for every group? Make the same number of mistakes for every group? Treat everyone the same way, or adjust for past unfairness? People still debate these questions. What everyone agrees on: builders should test their systems on many kinds of people and fix problems before the system affects real lives.
Ways to reduce bias:
- Collect diverse, representative training data.
- Test accuracy separately for different groups, not just overall.
- Include people from different backgrounds on the team and in testing.
- Let people affected by a system give feedback and appeal decisions.
- Don’t use AI for high-stakes decisions without human review.
Try it
- Pick an AI system from your earlier lists, like a face filter, voice assistant or recommendation app.
- List three groups of people who might use it differently: for example, people with different skin tones, accents, ages, or who speak more than one language.
- For each group, write one way the system could work worse for them and why (think about the training data).
- Design a fairness test: describe what examples you would collect and how you would measure if it works equally well for everyone.
- Write one sentence you would say to the company that built it.
Check yourself
What is AI bias?
When an AI system’s results are unfair or less accurate for certain groups of people.
What did the Gender Shades study find?
Commercial face analysis systems were much less accurate for darker-skinned women than for lighter-skinned men.
Why test accuracy separately for different groups?
A high overall accuracy can hide that the system works much worse for one group.
Key words
- Bias: unfair or uneven results for certain groups.
- Representative data: data that includes all the kinds of people or cases the system will meet.
- Fairness: treating people justly and evenly.
- Audit: a careful check of how a system performs.
Previous lesson: Prompting basics and staying safe
Back to AI Explorers
Next lesson: Privacy, deepfakes and trust