10  Confidence Intervals

Ten confidence intervals around point estimates: nine cross a central dashed line, while the final red interval misses it.

In this chapter, we will finish up our initial exploration of Null Hypothesis Significance Testing, the \(z\)-test, and related ideas, by introducing the confidence interval. Like statistical power, this is a complementary concept often used in the context of a hypothesis test. But rather than focusing on a binary decision about whether to reject the null hypothesis, a confidence interval attempts to quantify the uncertainty in our estimate of the population mean.

What can one sample tell us?

As we’ve worked our way through the past couple of chapters, we’ve been dealing with situations where we know a population’s variability, \(\sigma\), and the null hypothesis specifies a population mean, \(\mu_0\). We take a sample, calculate its mean, and calculate the corresponding \(z\)-score to determine whether it is sufficiently unlikely under the null hypothesis.

In a sense, this approach centers the null hypothesis. We are asking how well the sample fits with what the null would lead us to expect. But what if we let the sample take center stage instead, and ask what population means look plausible given the sample mean we observed?

To put this in context, consider another population with known characteristics: a deck of cards. A complete deck consists of the numbers 1 through 13 (Ace = 1; Jack = 11; Queen = 12; King = 13), repeated 4 times, once for each suit. So that’s 52 cards total, and we can compute the characteristics of the full deck, like its standard deviation, \(\sigma = 3.74\). We could also compute its mean, but let’s hold off on that, and use a few samples to try to estimate it instead.1

Sample A: three cards

Sample A: 5 of clubs, 4 of spades, Queen of clubs. Three cards with a mean of 7.

\(n = 3\), \(M = 7\).

Your prediction

Using only these three cards, what would you estimate the population mean to be? Do you have any reservations about giving just one number? What if I ask you to suggest a range of values you would consider plausible? How confident can you be in your suggested range? (No need for formulas or calculations here; just go with your gut.)

Sample B: six cards

Sample B: Ace, Queen, 7, 3, 8, Jack. Six cards with a mean of 7.

\(n = 6\), \(M = 7\).

Your prediction

These six cards have the same mean as sample A. Would you keep your best estimate? Would you make your tentative range wider, narrower, or leave it unchanged? Explain what the extra cards change.

Sample C: another six cards

Sample C: Queen, King, 8, 8, Queen, 10. Six cards with a mean of 10.5.

\(n = 6\), \(M = 10.5\).

Your prediction

What estimate and tentative range would you give now? Should your range move, change width, or both? Could a sample like this come from the same deck?

Your guesses bring out two different questions: what value should we estimate, and how much room should we leave for sampling error? I wasn’t necessarily expecting precise numerical answers, though maybe you did feel a general hunch about how wide or narrow of a range to consider. If not, that’s okay too. The tool that allows us to be precise about all this is the confidence interval.

Confidence interval features

A confidence interval is made from two pieces.

A point estimate is a single number used to estimate a population parameter. Here, our point estimate is the sample mean, \(M\). In Chapter 4 we demonstrated that, under independent random sampling, the sample mean is an unbiased estimator of the population mean: across repeated samples, its average equals \(\mu\). We know that an individual mean can differ from the population mean due to sampling error, but the point estimate is the best single number we have for the population mean.

We do need to acknowledge the potential error, though. The second piece is a margin of error, a distance either side of the point estimate that indicates the amount of uncertainty surrounding that estimate of the population mean.

Together, the point estimate and the margin of error form an interval estimate. We can represent it visually with a dot for the point estimate and error bars extending one margin of error either side of it. The two ends show the lower and upper limits of the confidence interval; each half of the bar is one margin of error.

Figure 10.1: The features of a confidence interval: a point estimate, lower and upper limits, and a margin of error extending equally on either side of the point estimate.
A point estimate with an interval extending equally to its left and right. The distance from the point to either endpoint is the margin of error.

How big should the margin of error be? To decide, we need to choose a confidence level. This is usually expressed as a percentage, describing how often confidence intervals will include the true population mean within their range across repeated samples. In other words, if we set our confidence level at 95%, we’re saying that if we drew sample after sample from a fixed population and computed a confidence interval for each sample, in the long run 95% of the intervals would include the population’s true \(\mu\). If that sounds a bit abstract and confusing, don’t worry. We’ll return to this point later in the chapter.

A higher confidence level produces a wider interval. This might sound slightly counterintuitive, but it makes sense if you think about it: if we want to be more confident that our interval captures the true population mean, we need to leave more room for the variability that comes from sampling error. A lower confidence level allows us to be less conservative, and thus we can report a narrower interval.

By analogy, say I want to guess the average height of the students taking my statistics course next semester. I could guess very confidently that the average will be between 2’0” and 10’0”, but I can only be highly confident because my guess covers pretty much all possibilities. I might also guess something relatively precise, like between 5’6” and 5’8”, but I ought to be correspondingly less confident: there are many more ways I can be wrong.2

Constructing a confidence interval

As you’ve seen over the past couple of chapters, in the context of NHST, our sample is only interesting to us insofar as it gives us clues about the population it came from. One way of reasoning about what the sample is telling us is to construct a sampling distribution around the null \(\mu_0\), and place our sample mean within that distribution. Then we can ask, is the sample mean far enough away from \(\mu_0\) to reject that null? The idea of the confidence interval is to apply the same kind of reasoning but from the perspective of the sample mean: what values of \(\mu\) would we fail to reject given the sample mean we observed?

This isn’t an entirely new idea. On our journey through NHST, we have already been constructing a kind of interval. For our two-tailed \(z\) test with \(\alpha = .05\), we divided the null sampling distribution into rejection and non-rejection regions, bounded by the critical values we looked up in a unit normal table: \(\pm 1.96\) standard errors either side of the null mean, \(\mu_0\). An observed mean inside that range would not lead us to reject that null. This is a range of sample means under a specified population mean.

Figure 10.2: The middle 95% of a normal null sampling distribution lies within \(1.96\,\sigma_M\) of \(\mu_0\). Sample means in this range do not lead to rejection in a two-tailed \(z\) test with \(\alpha = .05\).
A normal null curve centered on μ₀, with two shaded 2.5% rejection regions and an indicator spanning the non-rejection range from μ₀ − 1.96 σ_M to μ₀ + 1.96 σ_M.

The confidence interval turns this around: given the sample mean we observed, which population means would it not be unusually far from? The distance is the same: if \(M\) is within \(1.96\,\sigma_M\) of a proposed \(\mu_0\), then that \(\mu_0\) is within \(1.96\,\sigma_M\) of \(M\).

The mechanics and math of the confidence interval are a direct extrapolation from the null test, though the confidence interval can exist as its own logical thing: no null value is needed to calculate the interval itself.

To show this logic in action, let’s revisit the cards we were dealt earlier. For our null hypothesis, we’ll use \(\mu_0 = 7\), the midpoint of the card values from 1 to 13, along with the known deck standard deviation, \(\sigma = \sqrt{14} \approx 3.74\). (The curve is the normal approximation to the sampling distribution under this null; the approximation is imperfect, particularly with three cards. The matching test and interval use that same approximation.)

Activity 10.1: From the null range to an interval estimate
The original card samples reappear. A bracket marks the null non-rejection range centered at 7, then moves below the curve to center on the observed sample mean. Sample A has interval [2.77,11.23], B [4.01,9.99], and C [7.51,13.49]. Only C excludes 7.

Here are the same three cards from sample A. The curve describes sample means we might observe under the null hypothesis \(\mu_0 = 7\). The dot below marks what we actually observed: \(M = 7\). The curve is a sampling distribution—a model of repeated sample means. The point is the estimate from this one sample.

The space between the lower and upper critical regions is a kind of interval: the range of sample means that would not lead us to reject \(\mu_0 = 7\). It’s not a confidence interval, but it’s related.

We can move a copy of that range down so it is centered on \(M\). Why does that work? Any proposed population mean within \(1.96\,\sigma_M\) of our sample mean would put this sample within its own non-rejection range. The new interval collects those proposed population means. Here \(M\) happens to equal 7, so the move is straight down.

Sample B had the same mean, \(M = 7\), but was made of six cards. The larger sample size results in a smaller standard error of the mean. That is reflected in the null curve becoming narrower, and its interval of nonsignificant means covering a smaller range. The confidence interval becomes narrower too, centered on the same point estimate, but more precise at the same confidence level.

Sample C also has six cards, so the null curve and its bracket stay the same as B’s. But its mean, \(M = 10.5\), lies outside that bracket. Before moving on, where should the confidence interval be centered? Should its width change?

Now the bracket moves down and across to center on \(M = 10.5\), keeping the same width. It no longer reaches 7. The sample mean was too far from 7 to fit inside that null’s bracket; equally, 7 is too far from the sample mean to fit inside its confidence interval.

Compare these with your opening predictions. A and B have the same estimate, but B’s larger sample makes its interval narrower. B and C have equal precision but different estimates: C’s interval moves without changing width, even though its cards look more alike. We used the deck’s known variability, not the spread of each sample, to calculate these margins. Two intervals include 7; one does not.

A hypothesis test asks about one proposed population mean. Here, a confidence interval collects the whole range of proposed means that would not be rejected by the matching test.3

This also explains where the 95% comes from. Under the normal model, 95% of the distribution of sample means falls within \(1.96\,\sigma_M\) of the true mean. For exactly those samples, the interval around \(M\) reaches far enough to include that true mean. We’ll see this happen across repeated samples later in the chapter.

Calculating the confidence interval

So the confidence interval is a slightly new way of looking at things, but mathematically, it only uses tools we’ve already acquired.

It starts with the familiar \(z\)-test formula:

\[ z = \frac{M - \mu_0}{\sigma_M} \]

In Chapter 9, to work out statistical power we rearranged this formula to convert the critical values back into raw sample means, \(\mu_0 \pm z \times \sigma_M\). There, the null mean was fixed, and we solved for the sample means at the edges of the non-rejection range.

Now we turn that around. We know the sample mean, and we want the edges of the range of population means that would not be rejected.

\[ \text{confidence interval for }\mu: \quad M \pm z \times \sigma_M \]

Note that here, \(z\) isn’t a test statistic calculated from our data. It’s the critical value matching our chosen confidence level: the positive \(z\)-score that cuts off the middle portion of the normal distribution from its two tails. For 95% confidence, that’s \(z = 1.96\), leaving 2.5% in each tail, just like the critical values for a two-tailed test with \(\alpha = .05\). For 80% confidence it would be \(z = 1.28\), and for 99%, \(z = 2.58\).

So the lower and upper limits of the interval are:

\[ \text{lower limit} = M - z \times \sigma_M \qquad \text{upper limit} = M + z \times \sigma_M \]

The amount we subtract and add, \(z \times \sigma_M\), is the margin of error. Choosing a different confidence level changes \(z\), and so the width of the interval; the point estimate and the standard error stay the same.

Figure 10.3: A confidence interval for the population mean extends from \(M - z\,\sigma_M\) to \(M + z\,\sigma_M\), with margin of error \(z\,\sigma_M\).
A point at M, with lower limit M minus z times sigma-M and upper limit M plus z times sigma-M. One half of the bar is labelled z times sigma-M.

Example: How much do students sleep?

Let’s practice calculating a confidence interval for some real data: a sample of \(n = 32\) students in my statistics class who reported how many hours of sleep they got the previous night. Their average was \(M = 7.31\) hours.

That mean becomes our point estimate of the population mean. What population mean are we estimating? We should be careful not to overclaim by suggesting that this is some universal average sleep duration or something. Rather, we’re estimating something like the average amount of sleep of students like these. The confidence interval will tell us how precise that estimate is.

To calculate it, we just need three ingredients: the sample mean, the standard error of the mean, and the \(z\) value for our chosen confidence level.

  1. The sample mean we already have: \(M = 7.31\).

  2. The standard error of the mean. We’ll assume that students’ hours of sleep are normally distributed, with a known standard deviation of \(\sigma = 1.5\) hours. (Why? I just made it up, that’s why.) So the standard error is \[ \sigma_M = \frac{\sigma}{\sqrt{n}} = \frac{1.5}{\sqrt{32}} = 0.265 \]

  3. The \(z\) value. Say we want a 95% confidence interval. As we saw above, that means \(z = 1.96\): the \(z\)-score that leaves 2.5% of the normal distribution in each tail, and 95% in the middle.

Now we can plug them into the equations to find the lower and upper limits. The margin of error is \(z \times \sigma_M = 1.96 \times 0.265 = 0.52\), and we subtract it from, and add it to, the sample mean:

\[ \text{lower limit} = 7.31 - 1.96 \times 0.265 = 6.79 \qquad \text{upper limit} = 7.31 + 1.96 \times 0.265 = 7.83 \]

So the 95% confidence interval runs from about 6.8 to 7.8 hours; this gives us a range of plausible values for the average amount of sleep in the population of students like these, acknowledging the uncertainty in our estimate. Notice that we didn’t need a null hypothesis to calculate it: just the data, the known \(\sigma\), and our chosen confidence level.

Figure 10.4: The sample mean of 7.31 hours of sleep and its 95% confidence interval, from 6.79 to 7.83 hours, assuming a known population standard deviation of 1.5 hours.
The class mean is 7.31 hours; its 95% interval extends from approximately 6.79 to 7.83 hours.

This interval expresses sampling uncertainty under our assumed model. It does not establish that one class’s convenience sample represents all the students whose sleep we might want to study.

Calculate the CI

Now calculate a 99% confidence interval for students’ mean hours of sleep. The sample mean is still \(M = 7.31\) hours and the standard error is still \(\sigma_M = 0.265\) hours, but for 99% confidence, \(z = 2.58\). What is the lower limit of the interval?

What is the upper limit of the 99% confidence interval?

How does the 99% interval compare to the 95% interval of about \([6.79, 7.83]\)? Why?

Factors affecting CI width

Having calculated one realistic confidence interval for students’ sleep, it’s worth thinking about the factors that affect the width of a confidence interval. To keep things concrete, each comparison below starts from our sleep interval: \(M = 7.31\), \(\sigma_M = 0.265\), and a 95% interval of \([6.79,\ 7.83]\), and explores how changing one factor affects the width of the confidence interval.

Activity 10.2: What changes the interval’s width?
An interval around 7.31 hours widens or narrows as sliders change the confidence level, sample size, and population standard deviation. The point estimate and axis stay fixed.

The most obvious factor is the confidence level we choose — for example, 80%, 95%, or 99%. As we saw earlier, more confidence (choosing 99% rather than 95%) widens the interval, and less confidence (choosing 80%) narrows it. Changing the confidence level changes only \(z\); it does not change the sample mean or its standard error. Move the confidence slider and watch the interval.

With a lower confidence like 80%, we’re specifying a narrower range, but we are correspondingly less confident. For the sleep data, an 80% interval uses \(z = 1.28\) instead of \(1.96\), shrinking the interval from about 1.04 hours wide to about 0.68.

Sample size will affect the precision of our confidence interval as well. Larger samples result in a smaller standard error of the mean, which will result in a narrower interval. On the other hand, a smaller sample means a larger standard error, meaning less precision. Move the sample size slider and watch the interval.

If the mean of \(7.31\) had come from a class of \(n = 128\) students instead of 32, the standard error would drop from \(1.5 / \sqrt{32} = 0.265\) to \(1.5 / \sqrt{128} = 0.133\). As a result, the 95% interval would narrow from about 1.04 to 0.52 hours wide. Notice that it took four times as many students to make the interval half as wide, because the standard error depends on the square root of \(n\).

Finally, the variability of the population matters too. The standard error of the mean depends on the population standard deviation, \(\sigma_M = \sigma / \sqrt{n}\), so a less variable population produces a smaller standard error and a narrower interval from the same sample size. Move the population standard deviation slider and watch the interval.

If students’ sleep had a standard deviation of 0.75 hours rather than 1.5, the standard error for our class of 32 would be 0.133 instead of 0.265, and the interval would be half as wide. It scales linearly with the population standard deviation, unlike sample size, which scales with the square root.

Notice that these are some of the same factors that affected statistical power in Chapter 9. More participants, or less population variability, give us a smaller standard error, which in turn produces narrower intervals and more power to detect a specified difference. Raising the confidence level is different. A 99% interval is wider than a 95% interval. Its matching test would use \(\alpha = .01\) rather than \(.05\). With everything else held fixed, that stricter test has less power.

Interpreting confidence intervals

So each sample can offer us an estimate of the population parameter with a margin of uncertainty: that’s the confidence interval for the sample. But we haven’t said how to interpret it, exactly. At least, not to my satisfaction. The answer is a bit subtle and tricky. We’ve touched on some of the issues along the way. We usually have just one sample, while the frequentist logic describes what happens over many repeated samples. So what does that long-run tell us about the interval we actually have?

The frequentist interpretation: long-run coverage

First, let’s make sure we understand the idea of long-run coverage. We mentioned it as the definition of confidence level. If we sample from a population repeatedly and construct a 95% confidence interval from each sample, about 95% of those intervals will include the true population mean. With 80% intervals, about 80% will, and so on. (These coverage rates are exact under the normal, known-\(\sigma\) model; for other populations, like our deck, they’re approximate.)

The confidence level describes the method used to construct the interval. It doesn’t mean that the single interval we’ve calculated has a 95% probability of containing the true population mean. Under the frequentist approach, the population parameter is a fixed constant. Any one confidence interval either contains it or it doesn’t. Usually we don’t know which. As the originator of the confidence interval concept, Jerzy Neyman, put it, “The parameter … is an unknown constant and no probability statement concerning its value may be made” (Neyman, 1937, p. 349).4

To illustrate this more concretely, let’s return to our deck of playing cards, a known population of 52 values (the numbers 1 through 13 repeated four times). Its mean is \(\mu = 7\), and its standard deviation is \(\sigma \approx 3.74\). Usually when we’re doing research, we don’t know the true population parameter—that’s the whole reason for taking samples and applying statistics. But sampling from this population we do know lets us check how often our intervals capture it.

Activity 10.3: Explore confidence interval coverage
Interactive demonstration of confidence-interval coverage: 100 samples of six playing cards each produce a 95% confidence interval, plotted against the deck's true mean of 7. Roughly 95 of the 100 intervals include 7. Recomputed as 80% intervals, roughly 80 do. Samples of 30 cards give narrower 95% intervals, and still roughly 95 of 100 include 7.

We’ll draw a fresh set of samples from the deck, six cards at a time, and calculate a 95% confidence interval for each one. Because this is a known population, the dashed line can mark the true population mean, \(\mu = 7\). Each interval will either include that line or miss it. The axis has room for 100 samples.

Here is our first sample. Its mean is M = ..., giving a 95% confidence interval of [..., ...]. It includes 7.

A second sample gives a different mean. Its width is the same as the first, because the sample size, \(\sigma\), and confidence level haven’t changed. But this interval extends from ... to ..., centered at M = ....

Now we’ll do this for 100 samples. Most intervals include 7, but every so often one doesn’t, shown in red. Those are the samples which would have resulted in rejecting the null hypothesis. After ... samples, ...% of the intervals include \(\mu\), just under the expected 95%. A different set of 100 samples would land a little differently again, but over the long run of repeated samples, the proportion settles near the confidence level.

We could calculate an 80% interval for those same 100 samples instead. The means don’t move but the margins around them shrink, from \(z = 1.96\) to \(z = 1.28\) standard errors. The narrower intervals miss \(\mu\) more often: now ... of the 100, or ...%, include it within their range. The confidence level is the long-run capture rate we chose for the method, but a run of 100 samples can land above or below it.

Switching back to 95% CIs, we’ll draw 100 new samples but with 30 cards instead of six. A larger sample has a smaller standard error, so each interval is narrower, a more precise estimate. Yet the capture rate stays close to the confidence level: ...% of these intervals include \(\mu\). Sample size can increase precision, whereas the confidence level sets how often the method captures \(\mu\).

In real research we usually don’t get 100 samples, we just get one. And we don’t know the true population \(\mu\), we just have our hypothesis. So the line now marks \(H_0: \mu = 7\). Suppose the one sample we happen to actually collect is sample ..., with interval [..., ...]. It includes 7, so the matching null test would not reject \(H_0\). If \(H_0\) is true, that is the correct decision. If \(H_0\) is false, it is a Type II error. Neither the null test nor the confidence interval can tell us which is the case.

Now suppose our one study had been sample ... instead, with interval [..., ...]. It excludes 7, so we would reject \(H_0\). If \(H_0\) is true, that is a Type I error; if it is false, a correct decision. Again, the researcher looking at this one interval can’t tell which.

Because we built the deck, we know \(H_0\) is true here, so each of the ... rejections is a Type I error. This is the same long-run logic as \(\alpha\) in the previous chapters. \(\alpha = .05\) is the Type I error rate of the testing procedure across many studies where \(H_0\) is true; 95% confidence is the capture rate of the interval procedure across many samples. Neither is the probability that a particular decision is right or a particular interval contains \(\mu\). They tell us how often the method goes wrong, not whether it went wrong this time. Open the controls to draw more samples, or change the sample size and confidence level.

What changes the capture rate?

We repeatedly take independent random samples of 25 observations from a normal population with a known standard deviation and calculate a 95% confidence interval from each sample. In the long run, about what percentage of intervals will contain the true population mean?

80%

95%

25%

100%

For the same population and sampling method, we now increase each sample’s size from 25 to 100 but keep calculating 95% confidence intervals. In the long run, about what percentage of intervals will contain the true population mean?

80%

95%

99%

100%

Now suppose we switch from 95% to 80% confidence intervals, and also increase sample size again, up to \(n = 1000\). In the long run, about what percentage of intervals will contain the true population mean?

80%

Somewhere between 80% and 95%

95%

100%

Confidence intervals and hypothesis testing

Showing that the long-run coverage idea works out is all well and good, but the question remains: if the confidence interval for a single sample doesn’t let us put a probability on the true population mean, what does it tell us? Surely the whole point of taking a sample is to be able to say something about the population it came from?

That limitation was part of the idea from the start. Jerzy Neyman developed confidence intervals in the 1930s (Neyman, 1937), having developed a long-run approach to hypothesis testing with Egon Pearson a few years earlier. Their goal was modest; they didn’t expect their frequentist approach to establish whether any particular hypothesis was true or false. Instead, they aimed to find rules which would ensure that “in the long run of experience, we shall not be too often wrong” (Neyman & Pearson, 1933, p. 291). Confidence intervals make the same kind of promise: how often the method works across repeated samples, not whether it worked for the one sample we have.

This is what we saw at the end of Activity 10.3. The long-run coverage didn’t make any individual confidence interval conclusive. One that included 7 within its limits would have corresponded to a nonsignificant test; one that didn’t would have rejected the null.

Stated generally, if \(\mu_0\) lies outside the confidence interval, the matching two-tailed test rejects the null hypothesis; if \(\mu_0\) lies inside the confidence interval, the matching test fails to reject. By “the matching two-tailed test,” I mean that the test’s \(\alpha\) level complements the confidence level: a 95% CI matches a two-tailed \(\alpha = .05\); a 99% interval matches \(\alpha = .01\), an 80% interval a test at \(\alpha = .20\), and so on. When that’s the case, both procedures depend on the same distance: how many standard errors separate our sample mean from the null hypothesis mean.

To take another example, our 95% confidence interval for the class’s sleep data was \([6.79,\ 7.83]\). Using the same assumed population standard deviation, we can test the null hypothesis that students like these average \(\mu_0 = 8\) hours:

\[ z = \frac{M - \mu_0}{\sigma_M} = \frac{7.31 - 8}{0.265} = -2.60 \]

For a two-tailed test with \(\alpha = .05\), the critical values are \(\pm 1.96\). Our \(z\) falls below the lower critical value, so we reject the null hypothesis. The confidence interval leads to the same decision: 8 lies outside its limits, so it’s a significant result.

From this point of view, confidence intervals don’t add much over and above the null test. If all we do is check whether \(\mu_0\) is inside the interval, we’ve simply performed the same significance test with error bars instead of critical regions. The logic and math behind the decision rule is the same.

What does the interval add?

From another point of view, though, the interval does have something more to offer than the binary significant-or-nonsignificant verdict. It bundles a point estimate and its precision into a range of plausible population means, rather than simply arbitrating a single hypothetical value. Getting a sense of that range can be more informative than only knowing whether \(p < \alpha\).

For example, recall the idea I floated in Chapter 9 of giving SAT-takers a banana before completing the exam. I made up two studies that gave the same sample mean of 501, a one point improvement from the null mean of 500, but with different sample sizes. Assuming the same known \(\sigma = 100\), we can now calculate the confidence intervals for those outcomes.

Table 10.1: Confidence intervals and matching two-tailed test decisions for two hypothetical banana studies with the same sample mean, a null mean of 500, and known \(\sigma = 100\).
Sample size \(M\) 95% CI Decision (\(\alpha = .05\))
50,000 501 \([500.12,\ 501.88]\) Reject \(H_0\)
1,000 501 \([495.20,\ 507.20]\) Fail to reject \(H_0\)

The point estimates are identical, but they come with different degrees of precision. The larger study is significant, yet even the upper end of its confidence interval suggests a benefit of less than two points (501.88 - 500 = 1.88 points). The small study is nonsignificant, but its interval stretches from a five-point decrease to a seven-point improvement. It doesn’t establish that the banana does nothing; rather it fails to reject the null and leaves quite a lot of uncertainty about the true effect. From this point of view, the interval adds a way of quantifying that uncertainty which the significance decision alone neglects.

Summary

Where does this leave us? I’ve been deliberately ambivalent about the advantages and limitations of confidence intervals, but arguments about all this have been going on for quite a while. In 1986, influential statisticians Martin Gardner and Douglas Altman argued for the use of “Confidence intervals rather than P values: estimation rather than hypothesis testing” in medical research (Gardner & Altman, 1986). Some social scientists have likewise called for a “new statistics” focusing on tools like confidence intervals instead of null tests (Cumming, 2014).

A binary decision rule can be useful: it tells us when to reject the null and controls how often the procedure rejects a true null in the long run. But it’s a limited summary of the evidence, focusing on a probabilistic interpretation of the outcome given the assumptions of the null model. And there’s nothing magical about \(\alpha = .05\). Results with \(p = .049\) and \(p = .051\) lead to different decisions at that threshold, despite the evidence being close to identical. And as we saw in Chapter 9, the decision alone doesn’t tell us how big an effect might be, or whether a study could reliably detect it.

There’s nothing magical about the edges of a confidence interval either. Values just inside and just outside its limits are barely different from one another. If we only ask whether the interval includes the null value, we’re back to the same binary decision.

Nor does a confidence interval account for every source of uncertainty. Increasing the number of students who reported their sleep could lead to a narrower confidence interval, but it wouldn’t change the fact that they’re a convenience sample, reporting one night’s sleep from memory. The calculation accounts for sampling error under the assumed model; it doesn’t settle whether that model is appropriate.

In 2016, the American Statistical Association cautioned that “no single index should substitute for scientific reasoning” (Wasserstein & Lazar, 2016, p. 132). Whatever analysis we subject data to, we always need to think carefully about how the research was designed, what was measured, who was sampled, and how big of an effect would actually matter. Confidence intervals, like hypothesis tests, are tools for that kind of thinking, not replacements for it.

Looking ahead

We’ve covered a lot in this and the previous couple of chapters: the foundations of Null Hypothesis Significance Testing, and its nuances, extensions, and related ideas. There’s effect size, statistical power, confidence intervals, and the probabilistic thinking around our chances of making correct inferences, and the spectre of the kinds of inferential errors we can make.

We’ll be using these same principles in the coming chapters, but we’ll use them in slightly new ways depending on the knowledge we have about the population. Did you raise an eyebrow when I said we “assumed” a known population standard deviation for reaction times or students’ sleep? How could we ever know something like that? I admitted to just making it up so that we could explore these concepts in the context of the \(z\)-test. But in real research, we usually don’t know the population standard deviation. Thankfully, it’s not too much of a problem, but it will require us to revise our calculations slightly and introduce a new distribution, the \(t\)-distribution.

Learning Checks

Under the normal, known-\(\sigma\) model, what does the 95% in a 95% confidence interval describe?

The proportion of individual scores inside the interval

The probability that the calculated interval contains the population mean

The long-run proportion of intervals that contain the population mean

The proportion of future sample means inside the calculated interval

An independent random sample of 25 test scores has a mean of 52. Assume a normally distributed population with a known standard deviation of 10. Calculate a 95% confidence interval for the population mean using \(z = 1.96\), and state the margin of error. Show your working.

A two-sided 95% confidence interval for a population mean is [6.8, 7.8]. What would the matching two-sided test of \(H_0: \mu = 8\) conclude at \(\alpha = .05\)?

Fail to reject, because 8 is close to the upper limit.

Reject, because 8 is outside the interval.

Fail to reject, because the interval does not include zero.

We need a different sample to make the decision.

Two studies estimate a treatment’s benefit. Their 95% intervals are [-6, 8] and [-1, 1] points. Both matching tests fail to reject a null difference of zero. Suppose a benefit of at least 5 points would make the treatment worthwhile. Do the two studies tell us the same thing? Why?

A two-sided test has 80% power to detect a specified alternative mean at \(\alpha = .05\). If that alternative is true, about what proportion of repeated samples would produce matching 95% intervals that exclude the null value?

5%

20%

80%

95%

We collect sleep reports from a very large convenience sample of students at one university and obtain a narrow confidence interval. Does that resolve the problem of estimating mean sleep for all university students?

  1. We’ll suppose that to produce these samples we sample with replacement, putting each card back and shuffling before drawing the next one. So the same card can appear more than once, and each draw is independent.↩︎

  2. Another way to think about it is that I could propose with 100% confidence that the mean would be between minus infinity and plus infinity. I’m guaranteed to be right, but it’s not a very useful guess, all things considered.↩︎

  3. Just FYI, we’re using a simple method here for a symmetric \(z\) interval, with known \(\sigma\). Other methods can be used to produce asymmetric intervals. With the card CIs we’re also leaving the confidence interval untruncated: Sample C’s upper limit exceeds the maximum possible card value of 13. That reflects the normal approximation used by this simple method. More sophisticated methods can produce truncated intervals that respect the population’s bounds. That’s all beyond the scope of this book, but I wanted you to know it’s out there.↩︎

  4. That’s a feature of the frequentist framework we are using throughout this book, not a law of nature. Bayesian approaches to probability, for example, do treat parameters as having probability distributions, by integrating both the data and prior assumptions about which values are plausible. Bayesian analyses are becoming more common in the social sciences, but that’s beyond the scope of this book.↩︎