AP® Statistics review sheet from Aim for Five (aimforfive.com/stats/units/3/3-8)
Unit 3 · Topic 3.8
3.8 Potential Errors When Performing Tests
Any test can reach the wrong decision. A Type I error rejects a true H₀; a Type II error fails to reject a false one. This topic covers their probabilities (α and 1 − power), what increases power and how to describe each error and its consequences in context.
Key terms
- Type I error
- Type II error
- power
- significance level (α)
The two errors
A Type I error finds convincing evidence for Hₐ when Hₐ is actually false: a false alarm. A Type II error misses a real effect: Hₐ is true, but the data weren't convincing.
| Decision | H₀ actually true | H₀ actually false |
|---|---|---|
| Reject H₀ | Type I error | Correct (this is power) |
| Fail to reject H₀ | Correct | Type II error |
Probabilities
P(Type I error) = α. If you reject whenever the p-value ≤ 0.05, then when H₀ is true you'll wrongly reject 5% of the time.
Power is the probability of correctly rejecting H₀ when it's false. P(Type II error) = 1 − power. Researchers often aim for power of at least 0.80, so P(Type II error) is at most 0.20.
What increases power
Power goes up (and P(Type II error) goes down) when, with everything else unchanged:
- The sample size increases.
- The standard error decreases (a bigger sample is one way; less variable data is another).
- The real value of the parameter is farther from p₀, since a bigger effect is easier to detect.
- The significance level α increases. But that also raises P(Type I error), so there's a trade-off.
Consequences decide the trade-off
Which error is worse depends on the situation, and you should think about it before collecting data. If a Type I error is costly, use a smaller α like 0.01. If a Type II error is costly, increase the sample size to raise power (or use a larger α).
Describe errors in context: name the decision and what's actually true. "A Type I error would be concluding the new drug lowers blood pressure more than the old drug when it actually doesn't. Patients might switch to a more expensive drug with no extra benefit."
Why not make α tiny?
Lowering α makes false alarms rarer, but it also makes it harder to reject H₀ when it really is false. With n fixed, a smaller α means less power and more Type II errors. The only way to lower both error rates at once is to collect more data.
Picture two sampling distributions of p̂: one centered at p₀ (if H₀ is true) and one centered at the real p (if it isn't). The rejection cutoff sits between them. The area of the H₀ curve beyond the cutoff is α. The area of the real curve beyond the cutoff is the power. A larger n narrows both curves, so they overlap less and power rises.
Worked examples
Try each one yourself first, then open the solution.
- Example 1Calculator allowed
Describe both errors and consequences
A factory tests H₀: p = 0.02 vs. Hₐ: p > 0.02, where p is the true proportion of defective parts made on a machine today. If the test rejects H₀, the machine is shut down for repair. Describe a Type I and a Type II error and a consequence of each.
Show the solutionHide the solution
- Step 1: Type I: rejecting a true H₀. Concluding the defect rate is above 2% when it actually is 2%. Consequence: an unnecessary shutdown, losing production time and money.
- Step 2: Type II: failing to reject a false H₀. Not finding convincing evidence that the defect rate is above 2% when it actually is. Consequence: the machine keeps running and more defective parts reach customers.
Answer: Type I: shutting down a machine that's fine (lost production). Type II: leaving a faulty machine running (defective parts shipped).
- Example 2Calculator allowed
Probabilities of errors and power
A test uses α = 0.05. If the true proportion is actually 0.04, the test's power is 0.72. Find P(Type I error) and, when p = 0.04, P(Type II error). Name two ways to raise the power.
Show the solutionHide the solution
- Step 1: P(Type I error) = α = 0.05.
- Step 2: P(Type II error) = 1 − power = 1 − 0.72 = 0.28.
- Step 3: Raise power with a larger sample size or a larger α (which also increases the Type I risk).
Answer: P(Type I) = 0.05; P(Type II) = 0.28. Increase n or increase α to raise power.
- Example 3Calculator allowed
Trap: Type II error after rejecting
A test rejects H₀ with a p-value of 0.003. A student asks, "Could we have made a Type II error?" Answer.
Show the solutionHide the solution
- Step 1: A Type II error can only happen when you fail to reject H₀.
- Step 2: Since H₀ was rejected, the only possible error is Type I (if H₀ is actually true).
Answer: No. After rejecting H₀, the only possible error is a Type I error.
Common mistakes
- Mixing up the two errors. Type I: reject a true H₀. Type II: fail to reject a false H₀.
- Saying the p-value is the probability of a Type I error. α is.
- Describing an error without context or without saying what's actually true.
- Thinking you can lower both error probabilities by changing α. To lower both, increase the sample size.
On the exam
- Expect "describe a Type II error in context" and "what's a consequence?" Name the decision, what's actually true and a real-world result.
- If the test rejected H₀, the possible error is Type I; if it failed to reject, Type II. Questions often hinge on this.
Connected topics
Videos
Check yourself
4 questions on 3.8 Potential Errors When Performing Tests. Pick an answer to see if you got it, and why.
A standard treatment for a condition has a 30% success rate. Researchers test a new treatment on a random sample of patients, using H₀: p = 0.30 and Hₐ: p > 0.30, where p is the new treatment's success rate.
Described study
Which describes a Type I error in this context?
What is a possible consequence of a Type II error in this context?
Which change would increase the power of the test, with everything else unchanged?
The researchers use α = 0.05. If the new treatment's true success rate really is 30%, what is the probability that they make a Type I error?
0 of 4 answered