In this chapter, we will introduce the idea of the confidence interval. This is a complementary concept to null hypothesis significance testing.
Remember, our sample is only interesting to us insofar as it gives us an indication of the population parameters — particularly the population mean, because we want to know if that mean is different from the population mean that the null hypothesis predicts. The confidence interval quantifies how precisely our sample estimates the population mean by giving a range of likely values.
A confidence interval consists of two features. There’s the point estimate, which is our best guess of the population parameter. Specifically, the point estimate is our sample mean, because you might remember we have said that the sample mean is an unbiased statistic: on average, sample means are equal to the population mean. So if we just have a single sample mean, that is our best guess at the mean of the population from which that sample was drawn.
However, we know that sample means, though they might be unbiased on average, are not individually perfect estimates of the population mean. So the confidence interval includes a margin of error — a range either side of the point estimate that indicates the amount of uncertainty surrounding that estimate of the population mean. That range covers a stated proportion of the distribution. For a 95% confidence interval, the region in the middle of the distribution is 95% of the distribution. You could have a 99% confidence interval, which covers 99% of the distribution, or an 80% confidence interval, which covers only 80% of the distribution.
Calculating the boundaries of the confidence interval isn’t too complicated; it just involves a little algebra. In the hypothesis testing chapter, we used the \(z\) equation to calculate \(z\), giving us the exact \(z\) score for our observed sample mean based on a population mean we specified from our null hypothesis. In contrast, the confidence interval is not based on that null hypothesis. Instead, we want to calculate \(\mu\) — the potential population means which define the upper and lower bounds of the confidence interval. So instead of specifying \(\mu\) from the null hypothesis and solving for \(z\), we specify \(z\) and we solve for \(\mu\).
\[ z = \dfrac{M - \mu}{\sigma_M} \]
\[ \mu = M \pm z * \sigma_M \]
Specifically, the \(z\) values we specify are the \(z\) values which correspond to the boundaries of the confidence interval, and we find them the same way we find our critical cutoff values for alpha — the most extreme 5% (or whatever) of the distribution. So if we specify a 95% confidence interval, we’re saying we want to split the distribution into the 95% in the middle, leaving 5% in the tails, or 2.5% at either end — which is the same as when we specified alpha saying we wanted 5% in the tails, leaving 95% in the middle. Again, like alpha, we can specify different values. We might want a 99% confidence interval or an 80% confidence interval, and that will give us different \(z\) scores for the boundaries. But a 95% confidence is fairly standard, the same way that an alpha of 0.05 is.
We find the \(z\) values which correspond to those regions of the normal distribution, multiply by the standard error of the mean, and add to (and subtract from) the sample mean to find the corresponding \(\mu\) values. Visually, the equation looks like this: the sample mean \(M\) sits at the center as the point estimate, the boundaries of the interval are the potential population means \(\mu\), and the distance from the center to either boundary is \(z \times \sigma_M\).
So let’s calculate a confidence interval for some real data: our class’s reaction times. Like the samples we imagined in the hypothesis testing chapter, our class of \(n = 23\) students took the humanbenchmark.com reaction time test, for which the population of test-takers has a mean of \(\mu = 284\) milliseconds and a standard deviation of \(\sigma = 50\). The class’s average came out at \(M = 322.59\) milliseconds. That sample mean is our point estimate — but how precisely does it estimate the mean of the population that our class represents? Since we know the population standard deviation, the standard error of the mean is \(\sigma_M = 50 / \sqrt{23} = 10.43\).
The first step is to specify the \(z\) values for the boundaries of the confidence interval. That is, we need to decide on our desired confidence and find the \(z\) scores which divide the distribution up in that way. Say we want a 95% confidence interval. We need the \(z\) scores that cut off the lowest and highest 2.5% of the normal distribution, leaving 95% in the middle. Those should look familiar from the hypothesis testing chapter: they’re \(\pm 1.96\), the same values that mark out the two-tailed critical regions for \(\alpha = .05\).
Now we can put those values into the equation with our sample mean and standard error of the mean. You should get two answers corresponding to the population means of the two boundaries; allowing for rounding error, you should have gotten something like 302 and 343.
\[\mu = M \pm z \times \sigma_M = 322.59 \pm 1.96 \times 10.43 = [302.2,\ 343.0]\]
Calculate the CI
A sample mean is 322.59, the standard error of the mean is 10.43, and the 95% boundary z value is 1.96. What is the approximate confidence interval?
The interval is 322.59 +/- 1.96 * 10.43, or about [302, 343].
The margin of error is the boundary z value multiplied by the standard error of the mean.
It’s important to be clear about what the confidence interval does and does not tell us. Again, it primarily indicates the precision of a parameter estimate. It does not say we are 95% sure the true population mean is within this range. It’s easy to get confused about this because it would be so natural to take it as an indication that the true population mean must be within the confidence interval. But like alpha and the Type I error, we’re admitting that 5% of the time (or whatever we choose as our criterion for “quite unlikely”), the unlikely thing will happen. For the most part, the true population mean is something we can just never have concrete knowledge of. As Jerzy Neyman, the statistician who developed confidence intervals, put it: “The parameter is an unknown constant and no probability statement concerning its value may be made.”
A deck of playing cards is a known population consisting of the numbers 1 through 13 repeated 4 times, once for each suit. The average of this population (\(\mu\)) is 7.00, and the standard deviation (\(\sigma\)) is 3.74. The deck is a convenient example of a known population, but generally speaking we don’t know the true population parameter — that’s the whole reason for taking samples and applying statistics! Sampling from a population we do know, though, lets us check every confidence interval against the truth.
In this visualization, you can take samples from the deck repeatedly. Because this is a known population, the dashed line can mark the true population mean, \(\mu = 7\) — the value each sample will estimate.
Each sample produces an estimate of the population parameter with a margin of uncertainty: this is the confidence interval for the sample. These five cards give M = ..., with a 95% confidence interval of [..., ...].
The visualization really is sampling from a fair deck with a mean of 7. But as you’ll see, some of the confidence intervals don’t include the true population mean. Sample 7 is one: it just so happened that we obtained a sample whose confidence interval didn’t contain \(\mu\).
The confidence interval is a statement about repeated, long-run probabilities: if we sample from a population repeatedly, the proportion of confidence intervals which include the true population mean will be equal to the specified level of confidence. After ... samples here, that proportion is ...%. A common mistake is to think that the confidence interval tells us, with the given degree of confidence, that the true population parameter is a value somewhere within the stated range. This sounds intuitive but it’s not quite true.
For example, if we compute an 80% CI for each sample, around 80% of those CIs will include the population mean — here, ...%. With a lower confidence like 80%, we’re specifying a narrower range, but we are correspondingly less confident.
Maybe that sounds a bit disappointing. Surely the whole point of taking a sample is to be able to say something about the population it came from? Well, yes — this is where confidence intervals connect to null hypothesis significance testing. Let’s say we suspect that someone has removed some cards from our deck and so its population mean is not, in fact, 7. The null hypothesis, that our deck hasn’t been tampered with, would be that \(\mu = 7\): if the null value lies outside our confidence interval, we reject the null hypothesis; if it’s inside, we don’t reject it. This deck really is fair, so each of the ... rejections here would be a mistake — a Type I error.
Sample size affects the precision of the confidence interval as well: larger samples result in a smaller standard error of the mean, which results in a narrower interval. With 30 cards per sample, every interval narrows — but the proportion including \(\mu\) still matches the specified confidence: ...%.
With the dashed line hidden, this is what real research looks like: the true population mean is something we can just never have concrete knowledge of. Under the frequentist approach, the population parameter is a fixed constant, and we can’t make probabilistic statements about constants. So strictly speaking, the CI for a single sample reflects the uncertainty in that sample’s estimate of the parameter, rather than uncertainty about the true value of the parameter. Open the controls and keep drawing.
It’s worth thinking about the factors that affect the width of a confidence interval. To keep things concrete, each comparison below starts from our class reaction time interval: \(M = 322.59\), \(\sigma_M = 10.43\), and a 95% interval of \([302.2,\ 343.0]\).
The most obvious factor is the confidence level desired — for example, 80%, 95%, or 99%. More confidence (for example, choosing 99% rather than 95%) increases the width of the confidence interval. Less confidence (choosing an 80% confidence interval) decreases the interval width. This is a little bit counterintuitive, because by increasing our confidence from, say, 95% to 99%, we’re actually lowering the precision of our estimate. Another way to think about this is that we could have a 100% confidence interval: we’d say that 100% of sample means will be between minus infinity and plus infinity, but obviously that wouldn’t be a very useful confidence interval. With a lower confidence like 80%, we’re specifying a narrower range, but we are correspondingly less confident. For the class data, an 80% interval uses \(z = \pm 1.28\) instead of \(\pm 1.96\), shrinking the interval from about 41 milliseconds wide to about 27.
Sample size will affect the precision of our confidence interval as well. As you know, by and large, larger samples result in a smaller standard error of the mean, which will result in a narrower interval. On the other hand, a smaller sample means a larger standard error, meaning less precision. If our class had \(n = 100\) students instead of 23, the standard error would drop from \(50 / \sqrt{23} = 10.43\) to \(50 / \sqrt{100} = 5.00\), and the 95% interval would narrow from about 41 milliseconds wide to under 20 — a more precise estimate at the same level of confidence. Notice that the whole sampling distribution gets narrower, and the interval narrows with it.
Finally, the variability of the population matters too. The standard error of the mean depends on the population standard deviation, \(\sigma_M = \sigma / \sqrt{n}\), so a less variable population produces a smaller standard error and a narrower interval from the same sample size. If reaction times had a standard deviation of 25 milliseconds rather than 50, the standard error for our class of 23 would be 5.21 instead of 10.43, and the interval would be about half as wide. Unlike the confidence level and the sample size, though, this isn’t something the researcher gets to choose: the population’s variability is whatever it is.
Finally, if you really want a deep understanding of null hypothesis significance testing and confidence intervals, you’ll spend some time trying to grasp how they correspond to one another. It can be tricky to get your head around, but they each apply the same basic logic from different perspectives.
The \(p\) value and the confidence interval will always agree about statistical significance if the confidence interval is \(1 - \alpha\). For example, if alpha is 0.05 and you specify a 95% confidence interval: if \(p < \alpha\) for the hypothesis test, then the confidence interval will not contain the null hypothesis expected mean. Or to put it another way, if the confidence interval range does not contain the null hypothesis value, then you know that the results are statistically significant. In this sense, as the card-deck visualization above showed, the interpretation of any single confidence interval is most closely related to the binary decision about the null hypothesis.
Maybe you already kind of realize why this is the case. Both the alpha level and the confidence level define a distance from a mean to a limit. When alpha corresponds to confidence — when we have alpha .05 and 95% confidence — the distances in both cases are exactly the same. The only difference is that the null hypothesis looks for the distance from the null hypothesis mean to the alpha limit, while the confidence interval looks for the distance from the point estimate to the confidence limit.
This is hard to grasp, so let’s look at it visually, using our class reaction time data one more time. The null hypothesis perspective is to focus on the distance from the mean of the null distribution to the edge of the critical region. With an alpha of 0.05, this spans 47.5% of the distribution if we’re just looking in one direction. If the sample mean is beyond that boundary, we declare a statistically significant result. The confidence interval perspective is to focus on the distance from the point estimate (which is the sample mean) to the edge of the confidence interval. With a 95% confidence interval, that once again corresponds to 47.5% of the distribution if we’re looking in just one direction.
Start with the familiar hypothesis-testing picture. The red curve is the distribution of sample means the null hypothesis predicts: centered on \(\mu = 284\), with standard error \(\sigma_M = 10.43\) for samples of 23. The shaded tails are the critical regions for \(\alpha = .05\). Our class mean — the blue dot at \(M = 322.59\) — falls in the upper critical region, so the result is significant: p = ....
Significance is really a statement about distance. The arrow marks the span from the null mean to the critical boundary: \(1.96 \times \sigma_M\), or ... ms. Any sample mean farther than that from 284 is significant, and \(M\) is farther than that.
Now the confidence interval perspective on the very same data. The blue distribution is centered on the point estimate \(M\), and the shaded middle 95% is the confidence interval: [..., ...]. The arrow from \(M\) to the interval’s edge is \(1.96 \times \sigma_M\) again — exactly the same distance as before, measured from the other mean. The null value (the red dot at 284) sits outside the interval.
Put both pictures together and the correspondence is clear: the two arrows are the same length. So the blue dot lies beyond the red boundary exactly when the red dot lies beyond the blue boundary. Rejecting the null because \(M\) is in the critical region, and rejecting it because \(\mu\) is outside the confidence interval, are the same decision seen from two sides.
Here’s what a non-significant result looks like. If the class mean had been 300 ms, it would sit inside the critical boundaries (p = ...), and — correspondingly — the confidence interval [..., ...] now includes the null hypothesis mean of 284. The two perspectives always agree.
Open the controls and drag the sample mean back and forth. Watch for the moment \(M\) crosses the critical boundary: the \(p\) value crosses \(\alpha = .05\) at exactly the moment the confidence interval slides off the null mean. Changing the sample size or the confidence level moves both boundaries in lockstep, so the two decisions can never disagree.
So we reject the null because the sample mean is far enough away from the null population mean — it’s one of the most extreme means for that distribution, one of the 5% of least common means — and correspondingly, the null hypothesis mean is not contained within the 95% confidence interval. It is one of the most extreme means for the population that produced the confidence interval, one of the 5% of least common means. This is a subtle idea and it’s hard to intuitively grasp. But if you really want to understand this, you’ll spend some time dwelling on it.
Learning Checks
On average across many studies, what value of z would you expect if the null hypothesis is true each time?
0
If the null is true, the expected difference in the numerator is zero.
Which combination is most likely to produce a significant z statistic?
Large mean difference, small variability, large n
A larger numerator and smaller standard error of the mean produce a larger absolute z value.
In a single study, what absolute value of z is most expected if the null hypothesis is true?
A value near 0
Random samples vary, but values near zero are most common under the null.