11 The \(t\) test

Animated family of t distribution curves.

We’ve introduced the general idea of null hypothesis significance testing in the past few chapters, and we used the \(z\) statistic to complete some hypothesis tests. Here we’ll introduce the \(t\) statistic. We’ll see that it’s not very different from the \(z\) statistic; it’s just used in slightly different circumstances. We’ll revisit sample variance and degrees of freedom, because they will be newly relevant in the context of the \(t\) distribution, then work through a single-sample \(t\) test.

Known unknowns

We started the previous chapter by taking some samples from a full deck of cards. Since we knew the composition of the deck, we could figure out the population standard deviation, \(\sigma\). Knowing \(\sigma\) let us calculate the standard error of the mean, \(\sigma_M\), for our confidence interval and the denominator of the corresponding \(z\) test.

Now let’s take some samples from a new deck. But maybe this isn’t a regular, complete deck. Maybe I took some cards out. Which ones? Maybe I didn’t even look! So the characteristics of the population are now a mystery: we don’t know \(\mu\), or the true standard deviation, \(\sigma\). What are we to do? Let’s compare two potential samples and think it through.

Same mean, different uncertainty?

Each of these possible samples has four cards and a mean of 8. (We put each card back and shuffle before drawing the next one.)

Sample A: \(n = 4\), \(M = 8\)

Sample A: 6, 7, 8, Jack. Four cards with mean 8.

Sample B: \(n = 4\), \(M = 8\)

Sample B: 2, 5, Queen, King. Four cards with mean 8.

Which sample has more variability, and so would have the larger standard deviation if you calculated it?

Sample A

Sample B

They have the same standard deviation because their means are equal.

We don’t know the deck’s standard deviation. Suppose we use each sample’s standard deviation as an estimate of it, then divide by the square root of sample size, as before. Which sample would give the larger estimated standard error of the mean?

Sample A

Sample B

Both would give the same estimate.

Suppose we want to test a null hypothesis of \(\mu_0 = 7\) (the mean of an untampered deck, since we don’t know this deck’s actual mean). Both sample means are 8, so both differ from \(\mu_0\) by 1. If we divide that difference by each sample’s estimated standard error, which sample gives a test statistic farther from zero?

Sample A

Sample B

Both give the same test statistic.

Sample A’s cards are more similar to one another. Does that necessarily mean their average is closer to the deck’s true mean? What worries you about judging a mean difference using variability estimated from just four cards? What might help? Jot down your thoughts; you don’t need a new formula.

Quiz 11.1

We find ourselves in a more typical situation now. It’s rare that we know the population standard deviation when we’re doing real research, and so we can’t use the known-\(\sigma\) \(z\)-test approach very often in practice. Thankfully, we don’t have to throw out everything we’ve learned. The procedures from the past few chapters provide the foundations. We just let our sample do a little more work: we use its variability to estimate the population variability. This changes our math slightly, and it means we have to account for the fact that our estimate of uncertainty is itself uncertain. Doing so brings us to the \(t\) statistic.

From \(z\) to \(t\)

When I say it changes our math slightly, I mean very slightly. We just use the sample standard deviation, \(s\), instead of a known population \(\sigma\). That’s all. We divide \(s\) by \(\sqrt{n}\) to calculate the denominator of the \(t\) statistic, which we call the estimated standard error of the mean, \(s_M\).

\[ s_M = \dfrac{s}{\sqrt{n}} \tag{11.1}\]

It’s estimated because it’s calculated from \(s\) rather than the true \(\sigma\). (You’ll remember that Greek symbols are usually used to refer to population parameters and regular letters for sample statistics; the estimated standard error gets denoted as \(s_M\), because it’s based on a sample statistic.)

Then we use that estimated standard error as the denominator of the \(t\) statistic, just as we used the standard error in the denominator of the \(z\) statistic.

\[ t = \dfrac{M - \mu_0}{s_M} \tag{11.2}\]

Here’s a side-by-side comparison of the \(z\) and \(t\) statistics.

Table 11.1: Standard error and test statistic equations for a single-sample mean comparison. In both tests, \(\mu_0\) is the mean specified by the null hypothesis.

	Population variability known	Population variability estimated
Standard error	\(\sigma_M = \dfrac{\sigma}{\sqrt{n}}\)	\(s_M = \dfrac{s}{\sqrt{n}}\)
Test statistic	\(z = \dfrac{M - \mu_0}{\sigma_M}\)	\(t = \dfrac{M - \mu_0}{s_M}\)

So the numerator is the same. The denominator still describes the variability of the sampling distribution, but now we calculate it from an estimate of the population standard deviation. You might say the \(t\) test is just the \(z\) test in a different pair of pants.

For example, back in Chapter 8 we performed a \(z\) test to see whether radioactive spider bites seemed to have affected the reaction times of a sample of three people who we imagined had taken the humanbenchmark.com reaction time test. The calculations relied on assuming a population standard deviation of 50 milliseconds, but that wasn’t really a fact we knew about the population; I just made it up.

Best not to just make things up, though, at least not when we don’t have to. The \(t\) statistic lets us use basically the same hypothesis-testing logic without knowing (or needing to invent) the population standard deviation.

But the change has an important consequence. Hypothesis testing is about imagining the distribution of test statistics we would expect to see if the null hypothesis were true and we sampled repeatedly from the population it describes. In the context of the \(z\) test, each sample’s numerator would differ because of the different possible sample means, but the denominators would always be the same because of the known \(\sigma\) (and fixed sample size). With independent observations from a normal population, the resulting \(z\) statistics form a standard normal distribution; with a sufficiently large sample from a nonnormal population, this can be a useful approximation. But if we compute a \(t\) statistic for each sample, both the numerator and denominator vary from sample to sample, since each sample provides its own estimate of the population’s variability, in addition to the mean. We need to account for that additional uncertainty when figuring out the distribution of the \(t\) statistics.

Student’s \(t\)

The statistical ideas behind this were developed by William Sealy Gosset. Unlike many of the figures who developed the statistical procedures we still rely on today, Gosset wasn’t an academic statistician. In fact, he spent his career working for the Guinness brewing company, eventually becoming Head Brewer. Around the turn of the 20th century, Gosset helped the company bring science and statistics to bear on questions of beer production.

Guinness sent Gosset to study for a year under pioneering statistician Karl Pearson. Gosset realized that the data he was working with at Guinness were often more limited than the big datasets Pearson had access to. Whereas Pearson could develop procedures based on datasets with thousands of observations, Gosset might be working with just four farms, each growing a few varieties of barley (Box, 1987). His key insight was that a small experiment’s results could look more convincing than they really were if a sample just happened to show little variability—like Sample A in Quiz 11.1.

To show this, Gosset did something a lot like our little playing card demonstration at the start of the chapter. He wrote values (real heights and finger length measurements from an earlier dataset of 3,000 prisoners) on pieces of cardboard, shuffled them, and divided them into 750 samples of four (Student, 1908). Each sample would have not only a different mean, but also a different standard deviation. He was able to show that statistical procedures developed around large sample sizes and the normal curve do not work equally well with small samples: “as we decrease the number of experiments, the value of the standard deviation found from the sample of experiments becomes itself subject to an increasing error, until judgments reached in this way may become altogether misleading” (Student, 1908, p. 2).

So Gosset developed a way to judge mean differences that allowed for uncertainty in the sample’s estimate of variability. He published his solution in 1908 under the pseudonym “Student,” because Guinness didn’t allow its scientists to publish under their real names (Box, 1987). Another statistician, Ronald Fisher (whose 1925 book gave us the .05 convention in Chapter 8), later rescaled the resulting statistic, named it \(t\), and tabled it by degrees of freedom (Fisher, 1925). The \(t\) statistic is still often referred to as “Student’s \(t\),” combining Gosset’s pseudonym and Fisher’s notation.

The \(t\) distribution(s)

The solution was to ditch the standard normal distribution and compare \(t\) statistics with a different reference distribution: the \(t\) distribution.

Compared with the standard normal curve, the \(t\) curve is more spread out: lower in the center and higher in the far tails.

Figure 11.1: The standard normal curve and the \(t\) distribution for samples of four (\(df = 3\)).

The standard normal curve compared with a t distribution with 3 degrees of freedom. The t curve is lower in the center and higher in both far tails.

Why? Math, basically. The \(t\) statistic is a ratio. Dividing by a denominator that is too small inflates the ratio (pushing \(t\) farther from zero) more than a denominator equally too big shrinks it.

To illustrate, think back to the two hands of cards from Quiz 11.1. Both had four cards and a mean of 8. Now suppose I lied: I didn’t actually mess with the deck. Both hands came from a full deck, with \(\mu = 7\) and \(\sigma = 3.74\). Both hands have the same numerator, \(M-\mu_0 = 8-7 = 1\), but different denominators:

Table 11.2: Two samples with the same mean but different standard deviations. Percentages show changes relative to the known-\(\sigma\) calculation, using unrounded values.

	Known \(\sigma\)	Sample A: 6, 7, 8, 11	Sample B: 2, 5, 12, 13
Standard deviation	\(\sigma=3.74\)	\(s=2.16 \, (-42\%)\)	\(s=5.35 \, (+43\%)\)
Test statistic	\(z=0.53\)	\(t=0.93 \, (+73\%)\)	\(t=0.37 \, (-30\%)\)

Both hands were wrong about the true standard deviation by about the same amount, in opposite directions. But compared with the \(z\) statistic, which used the true \(\sigma\), Sample A’s underestimate inflates the statistic by almost 75%, while Sample B’s overestimate shrinks it by only about 30%. In other words, making the denominator too small has a bigger effect than making it too large by the same amount.¹

When independent observations come from a normal population, the sample mean and standard deviation vary independently; a too-small denominator doesn’t necessarily come with a smaller numerator to compensate. So each sample might underestimate or overestimate population variability, but the errors aren’t equally consequential. The result is a distribution with heavier tails—more extreme values than we’d expect if the denominator were fixed at the true population standard deviation.²

How much this changes the distribution depends on degrees of freedom—the number of observations free to vary in the estimate of the standard deviation. So unlike the single standard normal curve we used for \(z\), there’s a different \(t\) curve for each value of degrees of freedom.

In the single-sample situation we’re focusing on in this chapter, \(df = n - 1\), the same as when we computed the variance and standard deviation in Chapter 4. (Recall that once the sample mean is fixed, the deviations from it must sum to zero: if we know \(n-1\) deviations, the last is determined. That leaves \(n-1\) deviations free to vary when estimating variability.)

The smaller the sample, the less reliable its standard deviation, and the more the distribution is pushed out into the tails. As sample size increases, the randomness in the denominator is reduced and the \(t\) distribution approaches the standard normal curve. But the \(t\) distribution depends on the sample’s degrees of freedom rather than \(n\) directly because different designs can have different degrees of freedom in their estimates of standard deviation, as we’ll see in the next chapters.

Activity 11.1 compares \(t\) curves of different degrees of freedom with the standard normal curve. Each curve shows the distribution of test statistics across repeated samples when the null hypothesis is true, assuming independent observations from a normal population.

Activity 11.1: Explore the \(t\) distribution

Interactive comparison of t and standard normal distributions, with fixed boundaries at −1.96 and +1.96. About 8.2% of t statistics with df = 9 fall beyond these boundaries, compared with 5% of standard normal scores.

Here’s the \(t\) curve for samples of ten, \(df = 9\), with the standard normal curve for comparison. Look at the far tails, where the \(t\) curve sits above the normal curve, and the peak in the center, where the \(t\) curve is lower than the normal curve. Visually, the difference is subtle. In the next step we’ll quantify the difference more precisely.

The labeled values \(\pm 1.96\) are the familiar cutoffs for a two-tailed \(z\) test at \(\alpha = .05\); 5% of the normal curve lies beyond them. But this \(t\) curve contains about 8.2% in the same regions. If we used the normal cutoff for these \(t\) statistics, we’d reject a true null hypothesis more often than our intended 5%.

The difference is more pronounced with smaller samples. Our hands of four cards (and Gosset’s groups of four) have \(df=3\). With fewer observations, the estimate of population standard deviation is less reliable. Now about 14.5% lies beyond \(\pm1.96\). We’d reject a true null hypothesis much more often than our intended 5%.

At \(df=30\) the estimate of population standard deviation is more reliable and the two curves are nearly indistinguishable visually, but there’s still a difference. About 5.9% lies beyond \(\pm1.96\), so we’d still reject a true null a little too often if we used the normal cutoff.

Use the slider to explore. Notice that the curve changes very noticeably at small \(df\), but the differences become increasingly subtle as the \(t\) distribution approaches the standard normal curve at higher values of \(df\).

So to keep our two-tailed rejection rate at 5%, we need cutoffs farther from zero than \(\pm1.96\). How much farther depends on degrees of freedom. That’s what the \(t\) table tells us.

The \(t\) table

In the unit normal table, we looked up a \(z\) score to find the area beyond it. The \(t\) table generally works in the other direction: choose a tail probability and degrees of freedom, and it gives you the critical \(t\) value.

In Table 11.3, use the column heading for your chosen tail and alpha: one-tailed .05 or two-tailed .05, for example. Then find the row for your degrees of freedom, which for a single-sample \(t\) test is one less than your sample size.

Table 11.3: Critical \(t\) values by degrees of freedom and tail probability. For a two-tailed test, use the positive value and its negative counterpart.

Static table of t critical values.

To take an example, say we have a sample of \(n = 23\) scores. That gives us \(df = 22\). Figure 11.2 shows the critical regions for two- and one-tailed tests. If we are doing a two-tailed test with \(\alpha = .05\), we look in the two-tailed .05 column and find the critical value of \(t = \pm 2.074\). If we were doing a one-tailed test with \(\alpha = .05\), we would look in the one-tailed .05 column and find the critical value of \(t = +1.717\) (or \(-1.717\) if our alternative hypothesis were that the population mean is less than the reference value).

Figure 11.2: Critical regions for \(\alpha = .05\) with \(df = 22\).

Null t distribution with df = 22 and two critical regions shaded, 2.5% below t = −2.07 and 2.5% above t = +2.07.

Null t distribution with df = 22 and the highest 5% shaded as a single critical region above t = 1.717.

Null t distribution with df = 22 and the lowest 5% shaded as a single critical region below t = -1.717.

For a \(z\) test at \(\alpha = .05\), our critical values wouldn’t depend on \(n\) or \(df\): we’d use \(\pm 1.96\) for a two-tailed test and \(+1.645\) or \(-1.645\) for a one-tailed test. Notice that the \(t\) critical values are more extreme than the corresponding \(z\) values. With the \(t\) distribution, we have to go farther from zero to leave the same proportion in the critical regions because of its heavier tails.

Suppose we had a sample of \(n = 60\), with \(df = 59\). For a two-tailed \(t\) test at \(\alpha = .05\), the critical values would be \(t = \pm 2.001\), down from \(\pm 2.074\) at \(df = 22\). The larger sample size and corresponding \(df\) mean that the \(t\) distribution is closer to the normal distribution, and so the critical values are less extreme, closer to the \(z = \pm 1.96\) we would have used if we knew the population standard deviation.

Find the critical values

Use the table above. For a single-sample \(t\) test with \(n = 15\) and a two-tailed \(\alpha = .05\), what critical values should you use?

\(\pm 2.145\)

\(\pm 1.761\)

\(\pm 1.960\)

\(\pm 2.131\)

Now use \(df = 15\) for a one-tailed test with \(\alpha = .01\). The alternative says the population mean is greater than the reference value. Where does the critical region begin?

\(t = +2.602\)

\(t = -2.602\)

\(t = +2.947\)

\(t = +1.753\)

A \(t\) test, step by step

Let’s work through an example hypothesis test using our new tool, the \(t\) test. We’ll use data from the humanbenchmark.com reaction time test we’ve imagined giving to a sample of Spider-People in previous chapters. But this time we’ll use some actual data from students in a statistics class. (None of them were bitten by radioactive spiders, as far as I know.)

In Chapter 8, we used the test’s reported average of \(\mu = 284\) milliseconds for our null hypothesis. This time, rather than making up a value for \(\sigma\), we’ll use the class’s standard deviation to estimate the population standard deviation, calculate the standard error from it, and compare the mean with the same website reference of 284 milliseconds.

Step 1: State the hypotheses

We’ll use a nondirectional alternative hypothesis: the population mean differs from the reference in either direction. Correspondingly, the null hypothesis says that there is no difference; the population mean is equal to the reference value. (Like I said in Chapter 8, nondirectional hypotheses are generally the default, unless a directional test is justified and chosen before inspecting the data.)

\[ H_0: \mu = 284\text{ ms} \]

\[ H_1: \mu \ne 284\text{ ms} \]

Figure 11.3 shows the logic of the hypothesis test using the same kind of diagram as in Chapter 8, but adjusted to show the revised logic. Like the single-sample \(z\) test, a single-sample \(t\) test compares a population mean with a specified reference value. Unlike the \(z\) test, we no longer claim knowledge of the reference population’s standard deviation. Instead, we estimate it from the sample’s standard deviation. The sample’s mean and size are still known, and we can still use them to make an inference about the treated population’s mean. Here the “treatment” is just being a student in a statistics class—not something anyone was assigned to, as we’ll see when we interpret the results.

Figure 11.3: A diagram representing the logic of the single-sample \(t\) test. We still have a reference mean, but population variability is now unknown. The sample supplies \(s\) as well as its mean and size.

Reference population with specified μ₀ and unknown σ; treated sample supplies M, n, and s; inference to the treated population's unknown μ; does it have the same μ as the reference?

Step 2: Set the decision criteria

We’ll use \(\alpha = .05\). With 65 scores, \(df = 64\). Looking in the two-tailed .05 column of the \(t\) table (Table 11.3), as we did earlier, gives critical values of \(\pm 1.998\). We reject the null if \(t < -1.998\) or \(t > 1.998\).

Table 11.4: An excerpt of the \(t\) table, with the \(df = 64\) row highlighted. The two-tailed \(.05\) heading puts \(.025\) in each tail.

t table excerpt for 62 through 66 degrees of freedom, highlighting the two-tailed .05 critical value of 1.998 at df = 64.

Step 3: Collect and analyze the data

Here are the scores, in milliseconds:

Figure 11.4: The 65 class reaction times, in milliseconds, sorted from lowest to highest.

The 65 class reaction times in milliseconds, sorted from 159 to 426 ms.

If you count them up, you’ll find \(n = 65\) scores, giving us the 64 degrees of freedom used above. If you add up all the values you get \(\Sigma X = 19{,}382\). So the sample mean is \(M = \frac{\Sigma X}{n} = \frac{19{,}382}{65} \approx 298.18\) ms.

To calculate our sample standard deviation, we divide the sum of squared deviations from the sample mean by \(n-1\) to get the sample variance, then take its square root to get \(s\)—exactly the procedure explored in Chapter 4. That gives us the following quantities:

Table 11.5: Summary statistics and null reference for the class reaction time test.

Quantity	Symbol	Class data
Sample size	\(n\)	65
Sample mean	\(M\)	298.18 ms
Sample standard deviation	\(s\)	47.76 ms
Null reference mean	\(\mu_0\)	284 ms
Degrees of freedom	\(df = n-1\)	64

Using those values, we can compute the estimated standard error of the mean:

\[ s_M = \frac{s}{\sqrt{n}} = \frac{47.76}{\sqrt{65}} \approx 5.923\text{ ms} \]

Then we can put that estimated standard error into the denominator of our \(t\) statistic:

\[ t = \frac{M-\mu_0}{s_M} \approx \frac{298.18-284}{5.923} \approx 2.39 \]

Step 4: Make a decision

The sample’s \(t\) statistic exceeds the upper critical value of 1.998, so we reject the null hypothesis. The class’s mean reaction time was significantly slower than the 284 ms null hypothesis reference.

Figure 11.5: The observed \(t = 2.39\) falls in the upper critical region of the null \(t\) distribution with \(df = 64\).

Null t distribution with 64 degrees of freedom. The two rejection regions lie beyond ±1.998; the observed t of 2.39 is in the right rejection region.

The corresponding two-tailed \(p\)-value is approximately \(.020\). As in Chapter 8, this includes outcomes at least as extreme in either direction. (Our critical-value table locates the critical region boundary, not this exact \(p\)-value; for that we usually use statistical software.)

Step 5: Estimate the effect, interval, and power

I didn’t include these in our original 4-step hypothesis-testing procedure in Chapter 8, but we’ve acquired some additional tools since then that we can also adapt to this new context.

Effect size

We can use a version of Cohen’s \(d\) for the single-sample \(t\) test, just as we did with the \(z\) test. The only difference here is that we use the sample standard deviation (\(s\)) in place of the unknown population standard deviation (\(\sigma\)).

\[ d = \frac{M-\mu_0}{s} = \frac{298.18-284}{47.76} \approx 0.30 \]

The numerator shows that the class mean is a little more than 14 milliseconds slower than the reference value of 284 ms. The denominator puts this raw difference in units of the standard deviation, estimated from the sample itself, for a Cohen’s \(d\) of about 0.30.

Interpreting this effect size is the same as before. It depends on context, expectations, and our judgment about what constitutes a practically important difference.

Confidence interval

Likewise, we can compute a confidence interval for the population mean. Once again, we just tweak the formula to use the estimated standard error and a \(t\) critical value. Here \(t\) is the positive critical value for our chosen confidence level and \(df\), not the \(t = 2.39\) we calculated from the data. For 95% confidence and \(df = 64\), \(t \approx 1.998\), the same value as our two-tailed test at \(\alpha = .05\).

\[ M \pm t \times s_M \approx 298.18 \pm 1.998(5.923) \approx [286.35,\ 310.02]\text{ ms} \]

Figure 11.6: The 95% \(t\) interval excludes the reference mean of 284 ms, agreeing with the two-tailed test at \(\alpha = .05\).

95% t confidence interval for the mean: 286.35 to 310.02 ms, with point estimate 298.18 ms. The reference mean of 284 ms lies below the entire interval.

The meaning of 95% confidence is the same as in Chapter 10: under the model’s assumptions, this procedure would produce intervals containing the true population mean in 95% of repeated samples. It doesn’t account for selection bias or measurement problems. And at the same numerical standard error, the larger \(t\) critical value makes the interval a little wider than one using \(z = 1.96\).

Statistical power

Remember the two sampling distributions from Chapter 9: one under the null hypothesis and one under a specified alternative. We can use the same idea to think about power for a \(t\) test. Suppose the true population mean really is 298.18 ms—the mean we observed in our class. Suppose also that the population standard deviation is 47.76 ms, using our sample’s \(s\) as a planning estimate. With samples of 65 and a two-tailed \(\alpha = .05\) test, how often would we reject the null of 284 ms?

Figure 11.7 illustrates the logic by placing two \(t\) curves at those hypothetical population means. The purple area represents the chance of rejecting that null hypothesis if that alternative were true.³

Figure 11.7: An illustration of statistical power. The rejection regions come from the null model, but power is an area under the alternative. These schematic \(t\) curves illustrate that logic; they aren’t an exact \(t\)-power calculation.

Schematic null and alternative t curves (df = 64) centered on 284 and 298.18 ms. Power is the shaded area under the alternative beyond the null rejection boundaries, about .654; beta is the area between them, about .346.

Bear in mind that plugging the mean we observed back into the power calculation is called observed power. It doesn’t tell us the true population mean, or whether this result is trustworthy. In fact, in a test like this one, with the same sample size and two-tailed alpha, any result with \(p \approx .02\) will come out with an observed power of about .65. Observed power isn’t adding new information; it’s just a restatement of the result we already have (Hoenig & Heisey, 2001).

As we said in Chapter 9, power is more useful for planning. In this sample we saw a difference between the sample mean and null of around 14 ms. Say we want to be able to detect a difference of 10 ms or more, in either direction. The next time we run the study, we could plan the sample size around that difference to aim for power of .80 under our planning assumptions.

We would use the sample-size equation from Chapter 9 (Equation 9.2):

\[ n \approx \frac{\left(z_{1-\alpha/2} + z_{1-\beta}\right)^2}{d^2} \]

Here, \(d\) is the hypothesized difference in standard deviation units, so we need a value for the population standard deviation. We still don’t know it, but our class gives us an estimate, \(s = 47.76\) ms. (Using these data to estimate variability can help with planning, though \(s\) is uncertain too; the problem with observed power was treating the observed effect as the true effect.) A difference of 10 ms is then \(d = 10/47.76 \approx 0.209\). For a two-tailed test with \(\alpha = .05\) and power of .80, we use the same values as in Chapter 9:

\[ n \approx \frac{(1.96 + 0.84)^2}{0.209^2} \approx 179.5 \]

Rounding up, that’s 180 students.

“Wait, why are we using \(z\) values?” you ask. Shouldn’t we be using \(t\) values? Well, yes; but remember, we’re trying to figure out sample size here. The \(t\) critical value depends on \(df = n - 1\), and \(n\) is exactly what we’re trying to find. So using the \(z\) formula gives us a first approximation. It can be slightly too low, especially when the projected \(n\) is small, because the actual \(t\) critical value will be greater than the corresponding \(z\) value. But the difference is small when the sample size is large, so the approximation is very close.⁴

The bigger source of uncertainty is the standard deviation. If the population standard deviation is larger than our class’s 47.76 ms, a study with 180 students would have less power than we planned for, so it can be prudent to aim for more than the calculations would suggest anyway, just in case.

The same factors affect power as before. Holding the others fixed, a larger true mean difference, less population variability, or a larger sample makes an effect easier to detect. Increasing alpha also increases power, but at the cost of a greater chance of a Type I error.

Reporting and interpreting the results

Reporting results

There’s a fairly standard way of reporting a hypothesis test. Here’s an example based on our class reaction time data:

A two-tailed single-sample \(t\) test comparing the class reaction times (\(M = 298.18\) ms, 95% CI [286.35, 310.02], \(SD = 47.76\) ms) with a reference of 284 ms was statistically significant, \(t(64) = 2.39\), \(p < .05\), \(d = 0.30\).

There’s a lot of information packed into this fairly short, dry, technical statement. You should report the kind of hypothesis test, the null reference, the descriptive statistics with a confidence interval for the mean, the test statistic and its degrees of freedom in parentheses, whether \(p\) fell below \(\alpha\), and the effect size.⁵

Interpreting the results

Performing the analysis, computing the effect size and confidence interval, and reporting the results are all fairly mechanical once you get the hang of it. But interpreting the results requires more thought.

So what does it mean, that we rejected our null hypothesis here? Our test provides evidence that the population mean differs from the 284 ms benchmark, under the assumptions we’ve made. It doesn’t tell us what caused the difference.

We could be tempted to turn our reaction time findings into a headline, something like “study shows that studying statistics makes your brain work slower.” But this class wasn’t randomly assigned to study statistics, the reference value isn’t from a randomized comparison group, and our convenience sample of statistics students may not be representative of the broader population. Maybe a lot of people on humanbenchmark.com take the test a bunch of times and get faster with practice, so our class was just unpracticed at this task rather than slower in general. Differences in devices or browsers could also affect the measured times. Or maybe we’re making a Type I error. This is a good lesson on why a “statistically significant” result doesn’t by itself explain the difference or establish how widely it generalizes.

Assumptions of the single-sample \(t\) test

Interpreting the outcome of a test fairly also depends on whether the assumptions of the test are reasonable. The exact \(t\) reference distribution assumes independent observations from a normal population. With larger samples, the procedure can tolerate some departures from normality, but remember there is no automatic cutoff (at \(n=30\) or anywhere else). Strong skew and extreme observations deserve attention, especially in small samples.

Part of checking the extent to which these assumptions are met involves something I skipped over in my haste to work through the hypothesis test: looking at the data. Before running any tests, researchers should generally see how the data were distributed, check for any concerning outliers, and things like that. Figure 11.8 shows a histogram of the class reaction times.

Figure 11.8: A grouped histogram, with a bin width of 20 ms, representing the 65 reaction times recorded for a class of students who took the humanbenchmark.com reaction time test.

Grouped histogram of class reaction times.

The sample’s distribution looks a bit lumpy and uneven, but forms a normal-ish shape: a central cluster with a few scores farther out on either side, roughly symmetrically. Perhaps the low score of 159 ms and high scores of 405 and 426 ms deserve a closer look. In a real analysis, we’d check for recording errors or problems completing the task, rather than discard scores just because they seem unusual at a glance.

And remember, this is a histogram of the scores in the sample. It doesn’t establish the shape of the population, or the distribution of sample means it would produce for samples of this size. Still, with \(n = 65\) and no huge departure from normality evident in the sample, the \(t\) test is a reasonable choice for this practice analysis.

The assumption of independence can’t be established by looking at the data; it’s more an issue of methodology. In this context it means that one person’s score shouldn’t depend on another person’s score. For a class of statistics students taking a reaction time test, that’s probably a safe assumption; sitting through the same lectures is unlikely to make their reaction times resemble one another’s. A test of their understanding of statistical concepts would be more questionable. Students in the same course share an instructor, lectures, readings, and so on, and those shared influences could push their scores in the same direction. The \(t\) test would treat those scores as more separate pieces of information than they really are. Another kind of dependence would be repeated attempts by the same person. That wouldn’t usually count as independent observations, since having performed the task once can affect the outcome of subsequent attempts. As a feature of study design and recruitment, we would want to know that the scores are from separate students rather than treating repeated attempts as new, independent observations.

Looking ahead

So with just a null hypothesis reference \(\mu_0\) and a suitable sample, the \(t\) test allows us to use a sample’s mean and standard deviation to test the null hypothesis (assuming the sample is a suitable representation of the population, the test’s statistical assumptions are reasonably met, and all the usual caveats). The basic logic of the test doesn’t change, but the additional uncertainty in the \(t\) statistic is reflected in the greater spread of the \(t\) distribution, and its dependence on \(df\).

But what if we don’t have a reference mean to use for our null hypothesis either? Say we had a hunch that radioactive spider bites, or studying statistics, or whatever, might affect reaction times, but we didn’t even have a reference of 284 ms to compare against. How would we proceed? In the next chapter, we’ll extend the \(t\) test to accommodate research designed for exactly these situations.

Learning Checks

In a single-sample test, why can’t we simply replace the unknown population standard deviation with the sample standard deviation and keep using the usual \(z\) cutoffs? Why does the problem become smaller with larger samples?

An independent random sample of 16 scores from a normally distributed population has \(M = 46\) and \(s = 8\). Test \(H_0: \mu = 50\) against a two-tailed alternative at \(\alpha = .05\). Calculate the estimated standard error, \(t\), and degrees of freedom, then state your decision. The two-tailed .05 critical values are \(\pm 2.131\). Show your working.

A single-sample \(t\) test is reported as \(t(21) = 2.38, p < .05\). How many people were in the sample?

A single-sample test gives \(t(399) = 4.00\), \(p < .001\), and \(d = 0.20\). What does \(d = 0.20\) tell us about the observed mean difference?

It is 0.20 sample standard deviations above the null reference.

It is 0.20 estimated standard errors above the null reference.

It is 0.20 points above the null reference on the original scale.

A study reports a 95% \(t\) confidence interval for the population mean of [46, 54]. For the matching two-tailed test of \(H_0: \mu = 50\) at \(\alpha = .05\), using the same data and assumptions, what is the decision?

Fail to reject, because the interval includes 50.

Reject, because the interval excludes zero.

Reject, because the interval includes values other than 50.

A sample of students who play video games every day takes the humanbenchmark.com reaction time test. A single-sample \(t\) test finds that their mean is significantly faster than the website’s reference of 284 ms. Which conclusion is best supported?

There’s evidence that their population mean differs from 284 ms.

Playing video games makes people’s reaction times faster.

Nothing, because the students weren’t randomly assigned to play video games.

A researcher records ten reaction-time attempts from each of six students and analyzes all 60 attempts as independent observations in a single-sample \(t\) test. Which assumption is most directly in doubt?

Independence of observations.

A normally distributed population.

Random selection from the population.

A researcher plans a study to detect a true difference of 10 ms, using an estimated population SD of 48 ms. The sample size is chosen to aim for power of .80. If the true SD is actually 60 ms, with the true difference, sample size, and alpha unchanged, how will power compare with the planned value?

Lower.

About the same.

Higher.

A researcher plugs the observed mean difference into a power calculation after completing a study. The resulting observed power is .65. What does this tell us about the result beyond its \(p\)-value?

It adds no independent information about the result.

It gives a 65% probability that the alternative hypothesis is true.

It tells us the true effect is large enough for 65% of repeat studies to be significant.

The card example illustrates the arithmetic, but a deck’s values aren’t normally distributed, so these four-card samples don’t give an exact \(t\) reference distribution.↩︎
The mathematical lopsidedness is compounded by another factor: under normal sampling, samples underestimate variability more often than they overestimate it. We saw in Chapter 4 that dividing \(SS\) by \(n-1\) makes the sample variance unbiased—correct on average. But the errors aren’t evenly scattered around the true value. More samples underestimate variance, balanced by fewer, larger overestimates. The sample standard deviation, derived by taking the square root of variance, also tends to underestimate \(\sigma\). So underestimates are both more common and mathematically more influential in the ratio.↩︎
I’m simplifying the illustration and computation of power here by using a single \(t\) curve shifted over to the alternative mean. Statistical software will give a slightly different answer than my approximation. For these hypothetical values, software gives power of about .655, compared with the figure’s approximation of .654. The difference can be greater with smaller samples. The curves aren’t literal sampling distributions of the mean: in a real \(t\) test, the rejection boundaries in milliseconds change with each sample’s estimated standard error. An exact power calculation accounts for this using a noncentral \(t\) distribution. That’s beyond the scope of this book; for us, grasping the general logic is enough.↩︎
Statistical software can do the exact calculation for a \(t\) test, using the noncentral \(t\) distribution I mentioned in an earlier footnote. Here it gives 181 students.↩︎
Statistical software would also give the exact \(p\)-value (about .020 here, as we saw in Step 4). If you have it, you can report it in place of \(p < .05\): \(t(64) = 2.39\), \(p = .020\).↩︎