Aim for FiveComputer Science Principles

Unit 5 · Topic 5.3

5.3 Computing Bias

Programs make decisions about loans, jobs, school admissions and even who gets stopped at an airport. This topic covers how human bias ends up inside those programs, where it can enter, and what programmers can do to reduce it.

Key terms

  • bias
  • algorithmic bias
  • biased data
  • reducing bias

What computing bias is

Bias is an unfair tilt toward or against a person or group. People have biases, often without noticing, and the programs people build can end up with the same ones.

A program isn't neutral just because a computer runs it. It follows rules that people wrote and learns from data that people collected. If the rules or the data are slanted, the results will be too, and they'll be applied to thousands of people very quickly.

Two ways bias gets in

Bias written into the algorithm. A programmer chooses what the program considers and how much each thing counts. If a rule gives extra points for something that's really a stand-in for income, race or gender, the program treats groups unequally, even if it never mentions those groups. A ZIP code, for example, can act as a stand-in for income or race because neighborhoods often differ in both.

Bias in the data. Many programs learn from examples. A hiring tool trained on ten years of a company's past hires learns to prefer whatever those hires had in common. If the company mostly hired men, the tool may learn to rank women lower. A face recognition system trained mostly on lighter-skinned faces can make more mistakes on darker-skinned faces; a 2018 study led by researcher Joy Buolamwini found that commercial programs that guess a person's gender from a photo of their face made far more errors on darker-skinned women than on lighter-skinned men.

Bias can enter at every stage

  • Deciding the goal: what counts as a "good" applicant or a "risky" customer.
  • Collecting data: who is included, who is missing, and whose past treatment is recorded.
  • Choosing inputs and rules: which factors the program uses and how they're weighted.
  • Testing: whether the program was checked on people unlike the developers.
  • Using the results: whether anyone reviews the program's decisions or they're accepted automatically.

Reducing bias

Programmers have a responsibility to look for bias and reduce it, because a biased program can spread an old unfairness to far more people than any one person could.

Practical steps include testing the program with data from many different groups and comparing the results group by group, gathering training data that represents everyone the program will affect, removing inputs that act as stand-ins for protected traits, getting feedback from diverse users and teammates, and keeping a human able to review and override decisions.

Diverse teams help here too. People with different backgrounds are more likely to notice a group the program is missing, which ties back to collaboration in topic 1.1.

Worked examples

Try each one yourself first, then open the solution.

  1. Example 1

    Tracing a biased rule

    A scholarship program uses this procedure, and applicants with a score of 75 or more get an interview.PROCEDURE scholarshipScore(gpa, hasHomeComputer) { score ← gpa * 20 IF (hasHomeComputer) { score ← score + 10 } RETURN(score) }Two students both have a 3.5 GPA. One has a computer at home and one doesn't. Trace the procedure for each, and explain why the rule is biased.

    Show the solution
    1. Step 1: Student with a home computer: score ← 3.5 * 20, so score is 70. hasHomeComputer is true, so score ← 70 + 10, which is 80. 80 ≥ 75, so this student gets an interview.
    2. Step 2: Student without one: score ← 3.5 * 20, so score is 70. hasHomeComputer is false, so the IF body is skipped and score stays 70. 70 is less than 75, so no interview.
    3. Step 3: Same grades, different outcomes. The only difference is owning a computer, which mostly reflects family income and the digital divide, not merit.
    4. Step 4: It's even possible for a lower GPA to beat a higher one: a 3.2 with a computer scores 74, while a 3.6 without one scores 72.
    5. Step 5: This is bias written into the algorithm. Removing the home-computer bonus (or replacing it with something that measures the student's own work) reduces it.

    Answer: Scores are 80 and 70, so only the student with a home computer gets an interview despite equal GPAs; the bonus favors wealthier families, so the bias is in the algorithm.

  2. Example 2

    Data or algorithm?

    A voice assistant understands most users well but often misunderstands people with strong regional accents. The developers trained it on recordings collected mainly from people in two large cities. Where did the bias come from, and what would reduce it?

    Show the solution
    1. Step 1: The rules for turning sound into words weren't written to exclude anyone, so look at the data.
    2. Step 2: The training recordings came mostly from two cities, so accents from elsewhere were barely represented. The program learned the patterns it saw most.
    3. Step 3: That makes it bias in the data.
    4. Step 4: To reduce it, collect recordings from speakers across many regions, then test accuracy separately for each accent group to check the gap has closed.

    Answer: The bias comes from unrepresentative training data; collecting recordings from many regions and testing accuracy for each group would reduce it.

Common mistakes

  • Thinking a computer can't be biased because it's just following instructions. It follows instructions and data that people chose.
  • Assuming bias is gone once race or gender is removed as an input. Other inputs, like ZIP code, can stand in for them.
  • Mixing up the two sources. Ask whether the unfairness comes from a rule someone wrote (algorithm) or from the examples the program learned from (data).

On the exam

  • Questions often describe a program that treats some group unfairly and ask for the most likely cause, or for the action most likely to reduce the bias. Good answers involve broader data, testing across groups or changing a biased rule.
  • Be ready to say that bias can enter at any stage of development, not only when data is collected.

Connected topics

Videos

Check yourself

4 questions on 5.3 Computing Bias. Pick an answer to see if you got it, and why.

Question 1 of 4

A company builds a program to screen job applications. It trains the program on ten years of the company's past hiring decisions, during which most people hired were men. Which is the most likely problem?

Question 2 of 4

A face-recognition program makes far more mistakes on people with darker skin than on people with lighter skin. Its developers find that most of the photos used to train it showed lighter-skinned faces. What is the best way to reduce this bias?

Question 3 of 4

A bank's loan program lowers an applicant's score if the applicant lives in certain ZIP codes. Which statement best describes the bias risk?

Question 4 of 4

At which stages of building software can bias be introduced?

0 of 4 answered