AP® Statistics review sheet from Aim for Five (aimforfive.com/stats/units/1/1-12)
Unit 1 · Topic 1.12
1.12 Potential Problems with Sampling
Bias is a flaw in how data are collected that pushes results in one direction. This topic covers the main kinds (voluntary response, undercoverage, nonresponse and response bias) and why convenience samples are a problem. A bigger sample doesn't fix any of them.
Key terms
- bias
- voluntary response bias
- undercoverage
- nonresponse bias
- response bias
- convenience sample
What bias means
A sampling method is biased if it tends to give statistics that are too high, or too low, compared with the parameter. The key word is tends: it's a repeated, one-directional error built into the method, not bad luck in a single sample.
Random chance makes samples vary around the truth. Bias makes them miss in the same direction again and again. Taking a larger sample reduces chance variation but does nothing about bias.
Sources of bias in choosing the sample
These come from who ends up in the sample:
- Voluntary response bias: people choose themselves, as with call-in or online polls. People with strong feelings, often negative ones, are more likely to respond.
- Convenience sampling: choosing whoever is easiest to reach, like friends or shoppers at one mall. They may be different from the population in important ways.
- Undercoverage: part of the population is left out of the sampling frame or is less likely to be chosen, like a phone survey that only calls landlines.
Sources of bias in getting answers
These come from what happens after units are chosen:
- Nonresponse bias: some chosen people can't be reached or refuse, and they differ from those who answer. A survey about free time will miss busy people who don't have time to reply.
- Response bias: answers or measurements tend to be off in one direction. Causes include leading or confusing question wording, the interviewer's presence, sensitive topics where people give socially acceptable answers, and self-reported measurements (people tend to overstate height and understate weight).
Quick reference
| Type | Where it comes from | Example |
|---|---|---|
| Voluntary response | People choose to join | Online poll on a news site |
| Convenience | Easiest units chosen | Asking your own friends |
| Undercoverage | Part of population can't be chosen | Email survey that can't reach people without email |
| Nonresponse | Chosen units don't respond | Busy people ignore a mailed survey |
| Response | Answers are off in one direction | Leading question, self-reported weight |
Explaining bias well
Naming the type isn't enough. A full explanation says which group is over- or underrepresented, how that group likely differs on the variable being measured, and which direction the estimate will probably be off.
Example (convenience sampling): "Surveying students in the library at 7 a.m. will overrepresent students who study a lot. They likely report more study hours than typical students, so the sample mean will probably overestimate the mean study time for all students."
Worked examples
Try each one yourself first, then open the solution.
- Example 1Calculator allowed
Name the bias and its direction
A radio host asks listeners to call in and say whether the city should build a new sports stadium. Of 900 callers, 72% say no. Identify the bias and its likely effect.
Show the solutionHide the solution
- Step 1: Listeners chose whether to call, so this is a voluntary response sample.
- Step 2: People who feel strongly, especially those upset about spending on a stadium, are more likely to call.
- Step 3: So 72% probably overestimates the proportion of all city residents who oppose the stadium. Only this host's listeners could call, too, which is undercoverage.
- Step 4: 900 callers is a lot, but size doesn't fix bias.
Answer: Voluntary response bias (plus undercoverage of people who don't listen). The 72% likely overstates opposition among all city residents.
- Example 2Calculator allowed
Trap: wording bias
A survey asks, "Given the dangers of speeding near schools, do you support more speed cameras?" Another version asks, "Do you support more speed cameras?" Which version would likely show higher support, and what kind of bias is it?
Show the solutionHide the solution
- Step 1: The first version opens with a reason to say yes, which nudges respondents.
- Step 2: That's response bias caused by leading question wording.
- Step 3: The first version would likely overestimate support compared with the neutral question.
Answer: The first, leading version would likely give higher support. That's response bias from question wording, which overestimates support.
Common mistakes
- Saying a bigger sample would fix bias. It reduces chance variation only.
- Confusing nonresponse (chosen people don't answer) with voluntary response (people choose themselves to answer).
- Naming a type of bias without saying who's over- or underrepresented and which direction the estimate moves.
- Calling any mistake "bias." Chance variation between random samples isn't bias.
On the exam
- Free-response questions often ask you to explain how a sampling method could lead to bias. Include the group, how it differs on the response variable, and the direction (overestimate or underestimate), in context.
- Multiple-choice questions often test whether a larger sample fixes bias. It doesn't.
Connected topics
Videos
Check yourself
4 questions on 1.12 Potential Problems with Sampling. Pick an answer to see if you got it, and why.
A news website posts the question "Should the city build a new stadium?" and 8,300 readers vote, with 71% saying no. Why might this result be misleading?
A polling company surveys a random sample of 1,500 people by calling landline phone numbers. Which type of bias is most likely?
A random sample of 2,000 residents is mailed a survey about local parks. Only 340 return it. What is the main concern?
A survey asks, "Given the dangerous condition of our roads, do you support a small tax increase to fix them?" Which type of bias does this wording most likely create?
0 of 4 answered