11 The \(t\)-test

Animated family of t distribution curves.

Introduction

We’ve introduced the general idea of null hypothesis significance testing in the past few chapters, and we used the \(z\) statistic to complete some hypothesis tests. Here we’ll introduce a new test statistic, the \(t\) test. We’ll see how it differs from a \(z\) statistic–which is not very much at all, thankfully. It’s just used in slightly different circumstances. We’ll reintroduce the ideas of sample variance and degrees of freedom because they will be newly relevant thanks to this \(t\) statistic. And we’ll practice hypothesis testing with the \(t\) test.

Recap: The \(z\) test

To begin with, let’s quickly recap the idea of hypothesis testing using the \(z\) test. As we learned, the \(z\) test is an appropriate statistic to use if and only if we know the population mean and standard deviation of the original population.[^And we know that the sampling distribution will be normally distributed, but that’s more of an assumption than a requiement] The null hypothesis significance test assumes a treated population that is no different to the original population. We determine the location of our actual sample mean in the sampling distribution of the mean for that assumed null hypothesis distribution, and we reject that null hypothesis if the sample looks sufficiently extreme and therefore unlikely to have come from that distribution.

\[ z = \frac{M - \mu}{\sigma_M} \]

It’s the denominator here, the standard error of the mean, that describes the variability of the sampling distribution, and therefore determines the probability of a given outcome. The standard error quantifies how much difference is reasonable to expect between the sample mean and its population mean. To determine that standard error, we need to know the population standard deviation, \(\sigma\), and the sample size, \(n\).

\[ \sigma_M = \frac{\sigma}{\sqrt{n}} \]

So to use this \(z\) test, we have to know the population standard deviation, \(\sigma\), which allows us to calculate this standard error. But like I mentioned when we introduced the \(z\) test, it isn’t actually used very often. Most often in real research we don’t know the standard deviation, and if we don’t know the population standard deviation, we can’t use it to calculate the standard error, and so our \(z\) test statistic is missing its denominator.

Thankfully, this isn’t just a dead end. There is a very simple solution.

From \(z\) to \(t\): The Estimated Standard Error

Which brings us to the \(t\) test. The solution to not knowing the population standard deviation is simply to use our sample’s standard deviation to estimate the population variability. Instead of our denominator being the standard error of the mean, \(\sigma_M\), we call it the estimated standard error of the mean, \(s_M\). You’ll remember that Greek symbols are usually used to refer to population parameters and letters for sample statistics. So we’re still referring to a standard deviation here, but we’ve swapped out the \(\sigma\) for the lowercase \(s\).

So this is the fundamental difference between the \(z\) statistic and the \(t\) statistic. To use the \(z\) statistic we must know the population standard deviation. If we don’t, we use estimated standard error, \(s_M\), in place of the unknown population standard error.

Formulas for Standard Error and Estimated Standard Error

Remember that standard error, \(\sigma_M\), is \(\sigma\) over the square root of \(n\), the sample size, or the square root of variance over the square root of \(n\), or the square root of variance over \(n\). These are all algebraically equivalent. The estimated standard error, \(s_M\), is \(s\), the sample standard deviation, over the square root of \(n\), or the square root of \(s^2\) (sample variance) over the square root of \(n\), or the square root of sample variance over \(n\). Again, these are all equivalent.

Now, you could be forgiven for thinking at first that this just seems like a cheap trick, or not a real difference, maybe a distinction without a difference. The only difference here is the symbol and the name for the denominator of the test statistic. In every other way this is just the \(z\) test. You might say the \(t\) test is just the \(z\) test in a different pair of pants. And that’s true in a way. But hopefully your next thought should take you back to why we took such care when we learned about variance and standard deviation to point out that samples give a biased underestimate of population variability and that we needed to correct for that by dividing by the degrees of freedom rather than by \(n\).

Degrees of Freedom Revisited

So this brings us back to this idea of degrees of freedom. It’s going to be relevant to us now once again, now that we’re dealing with the sample variance and the \(t\) statistic. We didn’t need to think about it with the \(z\) test because the \(z\) test relied on us just possessing knowledge somehow of the population variance. And remember, when we learned to calculate population variance, we just divided by \(N\), the number of scores in the population. No degrees of freedom required. But to calculate sample variance and standard deviation, we had to divide by degrees of freedom.

Again, this was because we lose a degree of freedom in the process of calculating sample standard deviation, because we first have to calculate the sample mean in order to calculate the deviation scores — how far each individual score deviates from that mean. And calculating that statistic, the mean, lost us a degree of freedom because one score was then constrained. It was no longer free to vary. And because we’re using the sample variability as part of the \(t\) statistic here, we have to be aware of the degrees of freedom for the test.

Specifically, the kind of \(t\) test we’ll introduce in this lecture is called the single-sample \(t\) test. And it’s the closest analog of the \(z\) test that we used, where we know the original population mean. Here we just no longer claim to know the original population variability. And for this single-sample \(t\) test, \(df = n - 1\), just as it was when we calculated a sample’s variance or standard deviation.

But it’s worth flagging at this point that whether degrees of freedom is \(n - 1\) or something else depends on what kind of \(t\) test we’re doing, and we’ll introduce other kinds of \(t\) tests in the coming lectures, which will have different degrees of freedom.

The \(t\) Distribution

So here’s the familiar normal curve. This is what we were dealing with to determine probabilities for the \(z\) test. Again, we didn’t have to think about degrees of freedom because there is just one mathematically defined normal curve. The probabilities were always the same, no matter what our sample size was.

Now that we have to consider degrees of freedom, the picture is slightly more complicated. And that’s because there’s actually no single \(t\) curve. The \(t\) distribution is actually a family of distributions. There’s a slightly different \(t\) curve, with slightly different probabilities, for each possible value of degrees of freedom. It just depends on our sample size.

But you can begin to see the trend here. With very low values of degrees of freedom — in other words, with small sample sizes — the \(t\) curve is noticeably lower at the peak and higher in the tails. This means it has greater variability than the true normal distribution. The scores are more spread out here. As we increase the degrees of freedom for the curve, its peak gets higher and the tails get lower. The shape of the curve gets closer and closer to the shape of the normal curve. But it never quite matches it.

And this is once again to do with the fact that samples inherently underestimate population variability. The \(t\) distributions acknowledge and compensate for this by having more variability than the normal curve. And by scaling that difference with degrees of freedom, the lower the sample size, the lower the degrees of freedom, and the more variability in the sampling distribution as compared to the true normal curve that we use when we just know the population parameters and we don’t have to estimate anything based on our sample.

How the \(t\) Distribution Changes with Degrees of Freedom

The graph below shows how the \(t\) distribution changes along with degrees of freedom, with the true normal distribution shown for reference. I only go up to degrees of freedom of 30 here. But in reality, we could have larger sample sizes, and degrees of freedom can theoretically go up to infinity. But again, you can see the trend. The shape of the curve changes very quickly as we increase degrees of freedom from very small values to slightly larger values. But the change slows down, the higher we get.

Explore the \(t\) distribution

Interactive t-distribution explorer controlled by degrees of freedom.

With one degree of freedom, the \(t\) curve is much flatter than the normal curve and more of the probability is out in the tails.

Increasing to two degrees of freedom already pulls the center upward and the tails downward.

By five degrees of freedom, the curve is still visibly wider than the normal curve, but the difference is smaller.

At ten degrees of freedom, the \(t\) distribution is approaching the normal distribution.

By thirty degrees of freedom, the difference is subtle. Use the control to try values in between.

In other words, there is a big difference between the distributions for degrees of freedom of one and two, or four and five, but a comparatively small difference between the distributions for twenty-eight and twenty-nine or twenty-nine and thirty degrees of freedom.

Finding Probabilities with the \(t\) Distribution

So now that we know about the properties of the distribution, particularly how it depends on degrees of freedom, we’re ready to start thinking about probabilities. With the normal distribution and the \(z\) test, we could determine the area under the curve for, say, the most extreme five percent of the distribution. And we want to do exactly the same thing for the \(t\) distribution. The only difference is that, again, there is more than one \(t\) distribution. And so the specific probabilities depend on the degrees of freedom.

The \(t\) Table

So instead of referring to the unit normal table like we’ve done in the past, we refer to a \(t\) table. Like the unit normal table, this will give us the \(t\) values associated with different regions of the distribution. But unlike the unit normal table, the \(t\) table must include values for different degrees of freedom.

Static table of t critical values.

So typically the table looks something like this. There’s a column for degrees of freedom, followed by columns for different alpha values of .1, .05, and so on. And often it’ll give these values for both one-tailed and two-tailed hypotheses. In other words, if you had a one-tailed hypothesis, an alpha .05, you’d look in the second column here. If your hypothesis was two-tailed with \(\alpha = .05\), you’d look in the third column. Then you find the right row based on your degrees of freedom, which for a single-sample \(t\) test is one less than your sample size.

Connect the table to the critical regions

Interactive t critical-value table.

t distribution curve with the selected critical region shaded.

Determine Probabilities

For df = 22 and a two-tailed alpha of .05, what critical t values should you use?

+/-2.07

+/-1.00

+/-1.64

+/-3.50

Using R

Or, of course, we can use R instead. We used the pnorm() and qnorm() functions when we were dealing with the normal distribution. For the \(t\) distribution there are analogous functions, pt() and qt(). The main difference is that, like we said, to find a particular \(t\) distribution, we need to state our degrees of freedom. If we tried to run the function without it, R will just give us an error message. So we specify df as an argument within the function.

When to Use the Single-Sample T Test

Here’s the familiar inferential process diagram again.

Inferential process diagram for a single-sample t test.

And so again, the situation that leads us to use a single-sample \(t\) test is one in which we only partially know the characteristics of the original untreated population. Specifically, we know the population mean, but we do not know the population standard deviation. That means we have to use the sample standard deviation along with sample size to determine the estimated standard error of the mean for the sampling distribution.

And as with the \(z\) test, we assume, according to our null hypothesis, that the treated population mean is the same as the untreated population mean. So we locate our sample mean in the sampling distribution for the null population. And we decide whether it’s sufficiently unlikely to have come from that unchanged population. If so, we reject the null hypothesis and admit that the treatment might have caused the population mean to change.

Example: Reaction Time Data

So now we’ll work through an example hypothesis test using our new tool, the \(t\) test. For the purposes of the example, we’ll use some data from that reaction time test you all took a few lectures ago. Here are the scores, in milliseconds (two people didn’t record a score, hence the NAs):

327 335 359 430 275.4
272 350 343.2 278 354
303 328 371 312 346
359 NA 259 313.6 258
244 374.4 NA 338 290

And if you count them up, you’ll find that there are 23 valid scores. This is our sample size, and it’ll be important in a moment for determining the degrees of freedom for the \(t\) distribution.

And I constructed a group frequency table and a grouped histogram for the scores. It’s worth reflecting for a moment on why I made these the way I did; why I selected this interval width of 20 for the table and histogram, and why I used the histogram rather than a bar chart or something else.

Grouped frequency table of class reaction times with interval width 20.

Grouped histogram of class reaction times. The population mean of 284 ms sits toward the lower end of the sample distribution.

Why?

Why might we use a grouped frequency table and histogram for the class reaction-time scores?

I’ve also put a red dashed line on the graph representing the population mean, according to that website, which you might remember was 284 milliseconds. In the past when we did our silly Spiderman \(z\) test example, I just guessed at the population standard deviation. But now let’s not pretend to know that population standard deviation. If we want to know whether the class’s sample average is significantly different from that population average, you can see that this is a situation for the \(t\) test. We know the population mean, but not the variability. We’ll have to estimate the standard error using our sample’s variability.

Now, the distribution is a slightly weird shape. It’s certainly not normal. And since our sample size is a bit less than 30, this actually throws off some of the statistical assumptions. But let’s just run with it anyway. From just eyeballing the distribution, you might guess that the class’s sample average is higher than the population average.

So how much higher? And is the difference sufficiently unlikely, given the variability in the data, that we could call the difference statistically significant? Well, let’s conduct a formal hypothesis test. Remember, the four steps we’ll follow: we need to state hypotheses, locate the critical region, calculate the test statistic, and then make a decision about the null hypothesis.

Step 1: State the Hypotheses

So first, our hypotheses. What do you think they’ll be? There’s room for disagreement, but a nondirectional hypothesis seems most appropriate here. It’s not like you were all bitten by radioactive spiders or anything, so there’s no real reason to suspect that your average reaction times as a class should be higher or lower than the average for the reaction time website.

So if we go with a nondirectional hypothesis, our null hypothesis states that statistics students have the same average reaction times as the general population: \(\mu = 284\). And the alternative hypothesis would be that statistics students have different reaction times to the general population.

Step 2: Locate the Critical Region

Step two is to specify our alpha level and identify the associated critical regions of the distribution. And again, for the \(t\) test, unlike the \(z\) test, this now depends on the sample size, which determines the degrees of freedom for the distribution. For this kind of single-sample \(t\) test, \(df = n - 1\). So since we had a sample of 23 scores, our degrees of freedom is 22. We’ll use our traditional \(\alpha = 0.05\) here. So the critical \(t\) values are \(\pm 2.07\). So if we find a \(t\) statistic for our sample that exceeds \(\pm 2.07\), we will reject the null hypothesis.

Step 3: Calculate the T Statistic

Next we calculate the \(t\) statistic for our sample. Remember, \(t = \frac{M - \mu}{s_M}\), the sample mean minus the population mean, divided by \(s_M\), the estimated standard error of the mean. Again, the numerator is the same as it was for our \(z\) test. The only difference is that the denominator is the estimated standard error. Since we don’t know the population standard deviation, we therefore have to estimate it using the sample standard deviation before we can find the standard error of the sampling distribution.

So this \(t\) statistic quantifies the difference between the observed sample mean and the hypothesized population mean, divided by the estimated standard error. In other words, it’s a ratio of how much difference we saw between the means to the standard error, which tells us how much of a difference we would expect due to chance alone.

So our population mean is still 284 milliseconds. And having calculated the class’s sample average, it comes out at 322.59 milliseconds, and the standard deviation comes to 45.31 milliseconds.

Step 4: Make a Decision

t Statistic Quiz

For the reaction-time example, M = 322.59, mu = 284, and the estimated standard error is about 9.46. What is t?

If the critical values are +/-2.07 and t = 4.08, what should you do?

Reject the null hypothesis

Fail to reject the null hypothesis

Change to a one-tailed test

Conclude the sample mean equals the population mean

Having calculated our statistic, step four is to make our decision. We reject the null hypothesis if the test statistic is sufficiently unlikely, assuming the null hypothesis to be true. In other words, if our calculated test statistic is in the critical regions we determined in step two, we know our sample is unlikely enough to make us reject the null hypothesis.

And you should have calculated that the \(t\) statistic of 4.08 is indeed within the critical region. So our class’s average reaction time appears to be significantly different from that website’s reported population average of 284 milliseconds.

Step 4a: Effect size

We need to remind ourselves about the idea of effect size that we introduced with the \(z\) test. Remember, effect size quantifies the absolute magnitude of the difference between groups, independent of sample size or statistical significance. We can use a version of Cohen’s \(d\) for the single sample \(t\) test, just as we did with the \(z\) test. Like the \(t\) statistic itself as compared with the \(z\) statistic, the only difference here is that we use sample standard deviation (\(s\)) in place of the unknown population standard deviation (\(\sigma\)).

Interpreting the Results

To go back to this diagram, we’re concluding that the two populations are different. We could begin speculating about why that might be, and it depends what exactly we think the treatment is and how it caused the difference. If we wanted to get headlines, maybe we could say something like “statistics makes your brain work slower.” Or if we were being a little more restrained, we might say that maybe all of the great statistics knowledge you were learning during that lecture was making your brain work pretty hard already, which made your reaction times a bit slower.

Or maybe a lot of people on that website take the test a bunch of times, rather than just five, like most of you did. And maybe their reaction times got a little faster with practice, so maybe our class was just unpracticed at this task rather than slower in general. Or maybe we just have a pretty small sample here and happened by chance to get an extreme sample mean from the null distribution. Maybe we’re making a Type I error. This is a good lesson on why just making up hypotheses and testing them with small samples can produce statistically significant but effectively meaningless results. But it was good practice at calculating a \(t\) test using data you generated yourselves. And that’s all we were looking for.

Research Designs for the Single Sample \(t\) Test

First, let’s take a moment to think about what kind of research designs necessitate the single sample \(t\) test that we’ve been talking about. Remember, the situation is one in which we know the population mean but not the population standard deviation.

One example that came to my mind, and one that features in the problem sets, is to do with the population distribution of sleep. A common sense idea that you’ve probably heard of is that people should get 8 hours of sleep per night. Now, this is scientifically questionable — the research is not at all clear on that — but based on that common sense theory, we might expect that the population distribution will have an average of 8 hours of sleep. That doesn’t tell us what the population standard deviation is; we’d have to estimate that based on a sample. Say we want to know if a sample of Barnard students get significantly more or less sleep than this expected population. This would be a good situation for a single sample \(t\) test.

Another situation that comes to mind is one in which we have a good, practical or theoretical reason to think we know the population mean. If we measure something in a way that should produce a predictable average score, we might justifiably assume that that will be the population average. Say we’re measuring people’s current level of happiness. We measure it using a five-point Likert scale. We have people circle the number from 1 to 5 that best reflects their current happiness level, and we label the points “a lot below average,” “a little below average,” “average,” “a little above average,” and “a lot above average.” It seems reasonable to assume that on average people would score average for this scale. By chance, some people would be a bit happier right now and some people a little bit less happy. There’ll be individual differences among the scores, sure, but that should balance out to the center of the scale. But again, we don’t know how variable the scores would be, so again, we’d have to estimate the variability from a sample.

So we could administer some kind of treatment — maybe giving people some cash or a gift, or having them do some pro-social act — and then see if the treatment affected their happiness. We’d use the single sample \(t\) test to see how likely it seemed that the treated population had a mean of three, as we’d expect the untreated population to have.

Now it’s your turn: can you think of a research design of your own that would call for a single-sample \(t\) test?

Think of a Design

Think of a research design that would call for a single-sample t-test.

Assumptions of the \(t\) Test

Something we didn’t mention as we worked through our example last time is the assumptions that the \(t\) test rests upon. There are three assumptions to bear in mind.

First, it assumes independent random sampling has been used — in other words, that values in the sample are independent of one another.

Second, it assumes that the population is normally distributed, though with a large enough sample size, this assumption can be violated. If you think back to the central limit theorem, you can probably see the connection. The logic of hypothesis testing follows directly from the insights of the central limit theorem — specifically, its ability to predict the characteristics of distributions of sample means. So if we violate the assumptions of the central limit theorem to do with independent random sampling and normality, then the hypothesis test will be invalidated too.

Lastly, the \(t\) test assumes homogeneity of variance. This just means we’re assuming that the variability in the original and the treated populations is the same. This assumption is not unique to the \(t\) test; it’s a general feature of this inferential procedure. If you think about the \(z\) test, we knew the original population standard deviation and we just assumed that it would be unchanged in the null distribution. Now, with the \(t\) test, we don’t know the original population standard deviation. We’re inferring the treated population’s standard deviation based on our sample, and we’re assuming that is just the same as the original population.

Another way to think about this is: we’re investigating the possibility that some treatment changed the population mean, but we assume that the treatment does not affect the variability of the distribution. We think the treatment might nudge scores higher or lower on average, but without making them any more or less spread out. As is often the case, this is a convenient simplifying assumption. Whether or not it’s a reasonable assumption depends on the specifics of our research question and design. In some situations we can actually test the extent to which this assumption is met or violated. We’ll see an example of that in the next lecture.

Reporting Results

The final step of conducting a hypothesis test, which we’ve skipped over until now, is to formally report the results. There’s a fairly standard way of doing this. You can think of it as a list of things that you just have to report, and you will usually do it in the same order every time. There’s just not too much room here for creative license.

Here’s an example based on our class reaction time data:

Given the average reaction time for the population of \(\mu = 284\) ms, according to humanbenchmark.com, a two-tailed single-sample \(t\)-test suggests that statistics students have significantly different reaction times (\(M = 322.59\); \(SD = 45.31\)) than the general population; \(t(22) = 4.08\), \(p < .05\), \(d = 0.85\).

There’s a lot of information packed into this fairly short, dry, technical sentence. You should report the kind of hypothesis test you’re doing, and if relevant, the expected population mean based on the null hypothesis. You should state verbally whether the test was significant or non-significant, and you should indicate the descriptive statistics — the sample mean and standard deviation — either in the text or as a summary table or graph. And then you report the numbers: the test statistic and its degrees of freedom in parentheses, the \(p\) value, and the effect size.

Learning Checks

An effect that exists is more likely to be detected if sample size n is large.

True

False

What are the differences between the t distribution and the z distribution?

The results of a hypothesis test are reported as t(21) = 2.38, p < .05. What does the 21 represent?

Degrees of freedom

Sample mean

Sample size exactly

The p-value

For that same reported result, t(21) = 2.38, p < .05: how many people were in the sample?

What value of t would you expect to see if the null hypothesis is true?

Which combination of factors is most likely to produce a significant value for the t statistic?

Large mean difference and small sample variability

Small mean difference and large sample variability

Small mean difference and small sample variability

Large mean difference and large sample variability

An effect that exists is less likely to be detected if the sample standard deviation s is large.

True

False