AP® United States Government and Politics review sheet from Aim for Five (aimforfive.com/gov/units/4/4-6)
Unit 4 · Topic 4.6
4.6 Evaluating Public Opinion Data
Not every poll deserves the same trust. To judge one, check who was sampled and how, how big the sample was, how the questions were worded, and whether the conclusions actually match the data. Polls have missed before, as in the 1980 and 2016 presidential elections, and how much public opinion shapes policy depends on the issue and the moment.
Key terms
- reliability
- credibility
- sample size
- sampling error
Reliability and credibility
- Reliability: a poll is reliable if the same method gives consistent results when repeated. If two polls taken the same week with the same method give wildly different numbers, something is wrong.
- Credibility (or validity): a poll is credible if it accurately measures what it claims to measure and can be trusted. That depends on a representative sample, neutral wording, a transparent method and a sponsor without a motive to skew results.
- A poll can be reliable but not credible: a consistently biased method gives the same wrong answer every time.
Questions to ask about any poll
- Who was sampled? All adults, registered voters or likely voters? For elections, likely voters are usually most useful but hardest to identify.
- How were they chosen? A random sample is credible; a self-selected online sample isn't.
- How many? A sample of 1,000 has a margin of error around ±3 points; a sample of 200 is much less precise. Subgroups (like young voters within a poll) have larger margins of error than the whole sample.
- How were questions worded and ordered? Look for leading or loaded wording.
- Who paid for it? Polls released by campaigns or advocacy groups deserve extra scrutiny.
- Do the conclusions match the data? A headline may claim a lead that's within the margin of error.
Sampling error and other errors
Sampling error is the random difference between a sample and the population that comes from surveying only part of it. The margin of error measures it. But polls can also be wrong for reasons the margin of error doesn't capture: some groups are less likely to respond (nonresponse), some people aren't reachable at all (coverage), people may give socially acceptable answers instead of true ones, and pollsters may guess wrong about who will actually vote.
Famous polling misses
Each miss taught pollsters something. One lasting change: after 2016, many pollsters began weighting their samples by education, since voters without college degrees had been underrepresented.
| Election | What happened | Main lesson |
|---|---|---|
| 1936 | The Literary Digest magazine mailed millions of ballots, mostly to people from phone and car registration lists, and predicted Franklin Roosevelt would lose. He won in a landslide. | A huge but unrepresentative sample is worse than a small random one |
| 1948 | Major polls stopped surveying weeks before the election and predicted Thomas Dewey would beat Harry Truman. Truman won. | Opinion can shift late; keep polling |
| 1980 | Final polls showed a close race, but Ronald Reagan won by about 10 points, as many voters decided late, after the only debate between Reagan and President Jimmy Carter, a week before the election. | Late deciders can swing results beyond what polls capture |
| 2016 | National polls came fairly close to the final popular-vote margin, but polls in several key Upper Midwest states underestimated the support of the candidate who won the Electoral College. | State polls matter in the Electoral College; many polls didn't weight for education, and late deciders broke one way |
How much does public opinion matter?
Officials watch polls, and public opinion influences which issues get attention and how politicians frame them. But its influence varies. It matters most on issues that are highly visible and that voters care strongly about, and near elections. On technical or low-attention issues, organized interest groups may have more influence than broad opinion. Some officials act as trustees (2.3), following their own judgment even against polls. Polls can also shape campaigns themselves, affecting fundraising, media coverage and who qualifies for debates.
Worked examples
Try each one yourself first, then open the solution.
- Example 1
Judging which poll to trust
Poll A surveyed 1,200 randomly selected likely voters by phone and online panels, weighted to match the state's population, with a margin of error of ±3. Poll B collected 15,000 responses through a link posted on a campaign's social media page. They disagree. Which is more credible? Explain.
Show the solutionHide the solution
- Step 1: Compare sampling methods: A uses random selection and weighting; B is self-selected and drawn from a campaign's followers.
- Step 2: Recognize the trap: B's larger size doesn't fix its bias, much like the Literary Digest's huge sample in 1936.
- Step 3: Consider the sponsor: B comes from a campaign with an interest in a favorable result.
- Step 4: Conclude based on method, not size.
Answer: Poll A. Its random, weighted sample of likely voters makes it representative, while Poll B's self-selected sample from a campaign's followers is biased no matter how large.
Common mistakes
- Trusting a poll because it has a large sample. Representativeness matters more than size.
- Saying all 2016 polls were wrong. National polls were fairly close on the popular vote; the larger misses were in key state polls.
- Assuming the margin of error covers every kind of error. It measures only random sampling error, not bias from wording, nonresponse or bad turnout guesses.
On the exam
- Expect questions asking you to evaluate a poll's method or identify why a poll might be inaccurate.
- In quantitative analysis, check whether a stated conclusion is supported by the numbers and the margin of error.
Connected topics
Videos
Check yourself
4 questions on 4.6 Evaluating Public Opinion Data. Pick an answer to see if you got it, and why.
| Poll | Who was surveyed | Sample size | Candidate X (%) | Candidate Y (%) | Margin of error |
|---|---|---|---|---|---|
| Poll A | Likely voters | 800 | 47 | 44 | ± 3.5 points |
| Poll B | Registered voters | 1,500 | 45 | 46 | ± 2.5 points |
Hypothetical polls of the same Senate race taken the same week
Which conclusion is best supported by the data?
Which difference between the polls most likely helps explain why their results differ?
Before trusting either poll, a careful reader would most want to know
After the 2016 presidential election, polling experts found that many state polls had underestimated the winner's support, partly because they didn't adjust their samples for voters' education levels. This finding best shows that
0 of 4 answered