Normal curve with two-tailed critical regions shaded.
We’ll start this chapter by revisiting the idea of radioactive spider bites affecting reaction times. This time, though, we’ll move beyond one unusually quick kid and consider a whole sample of Spider-People. That will present a problem: how to decide whether our sample mean is unusual enough to count as evidence for a change in the population mean. We’ll solve it using the tools we’ve picked up over the last few chapters: \(z\)-scores, probability, sampling distributions, and the Central Limit Theorem. By putting them together, we’ll complete our first hypothesis test.
Spider-Man revisited
Back in Chapter 5 we considered the possibility that a radioactive spider bite might have improved the reaction times of one individual. We supposed that following the bite, our test subject, Peter Parker, had a reaction time of 189 milliseconds on a test for which the general population had a mean of 284 milliseconds and a standard deviation of 50 milliseconds. We calculated his \(z\)-score to be \(z = \frac{189 - 284}{50} = -1.9\). Assuming reaction times form a normally distributed population, we then used the unit normal table in Chapter 6 to find the proportion of scores as fast or faster than his, which turned out to be less than three percent. We decided his score seemed a bit unusual, and we wondered whether it might be evidence that the spider bite had changed his reaction time.
That thought process already contained the basic logic of a hypothesis test: we suspected the bite would cause a change, but we asked what were the chances of seeing a reaction time this fast or faster if the spider bite had done nothing. An entirely unexceptional score would give us no reason to doubt that assumption. A very unusual score would give us reason to question it.
But real research doesn’t usually rely on examining just one individual. We usually want a sample. We’d need a bunch of kids who we treated to radioactive spider bites to see if it would alter their reaction times. Obviously in the real world this experiment probably would not make it past the ethics board, but the Spiderverse™️ provides just such a scenario.
Figure 8.1: A sample of my son’s toy Spider-People, \(n = 3\)
So say we find three kids, all bitten by radioactive spiders, and we measure their reaction times on the same test. Now the sample mean is 189 milliseconds. How does that change our analysis? Is it just as surprising as a single score of 189ms? More surprising? Less? Surprising enough to count as evidence that the population mean changed?
One Spider-Man or three?
Before we calculate anything, what do you make of this result? Is a
sample of three people with a mean of 189 milliseconds stronger evidence
that the spider bites changed reaction times than one person scoring 189
milliseconds? Why or why not?
Explanation
There isn’t a single response to mark right or wrong here. If you
think the sample is more convincing, try to pin down why: what happens
to a mean when you average several scores, rather than taking just one?
Keep your reasoning in mind as we work through the problem. The next
section gives us the tools to compare the individual score and the
sample mean more precisely.
Null Hypothesis Significance Testing
Moving from one Spider-Man to a sample of Spider-People changes the calculations, but the general logic is the same. For Peter’s individual score, we used the population distribution to ask what reaction times we would commonly see if nothing had changed. For our sample mean, we’ll need the sampling distribution we learned about in Chapter 7. This describes the distribution of all possible sample means for that population and sample size. Once we know its characteristics we can ask what proportion of means are as extreme as or more extreme than the one we observed, again assuming that the spider bites didn’t change reaction times. We only test one treated sample, but we care about the hypothetical population it represents—what would happen if everyone received the treatment. The sample mean is our clue to that population mean, as Figure 8.2 illustrates.
Figure 8.2: The inferential process for a single-sample hypothesis test, from a treated sample to a conclusion about the population.
Reference population; sample receives the treatment; inference from the treated sample to the treated population; does it have the same μ as the reference population?
The formal procedure is called Null Hypothesis Significance Testing, or NHST. The spider bite is the treatment: the condition or intervention applied to our sample. A change in reaction times associated with the treatment would be a treatment effect (though a significance test alone cannot establish what caused the change).
Under NHST, as the name implies, we begin with a null hypothesis. This is a kind of skeptical working model: suppose the spider bites did nothing, so the population of bitten kids still has the same mean reaction time as the population of unbitten kids, 284 milliseconds. This becomes the null hypothesis, denoted \(H_0\). The alternative hypothesis, \(H_1\), says that the population mean changed.
For this first test, we’ll allow for a change in either direction. The null says the population mean remains 284; the alternative says it differs.
If the null is true, our three Spider-People should behave like a random sample from a population with a mean of 284. But we know that sample means vary due to sampling error, even when the population mean hasn’t changed; the sample’s mean probably won’t be exactly 284ms. So if our treated sample has a different mean, we need to judge whether the difference is more than we would typically expect from sampling error alone. We do that by imagining the sampling distribution we would get if the treatment had no effect—the null distribution—which tells us which sample means would be common and which would be unusual. Then we see where our observed mean of 189 falls.
That raises another question: how unlikely is unlikely enough? Before analyzing the data, we choose a decision criterion, called alpha and written \(\alpha\). For this first test, we’ll use \(\alpha = .05\): results in the most extreme five percent of the null distribution will count as sufficiently unusual to reject the null. (We’ll return to why we might choose that criterion and how the hypotheses determine where we place it.) We call the region defined by alpha the critical region. Its boundary is the critical value (or values, if the region is split between the two tails following a non-directional hypothesis).
There are different kinds of tests we can submit our data to, depending on the nature of the data and the information we have (or assumptions we can make) about the untreated population. In any case, we calculate a test statistic from the sample and compare it to the null distribution. If it falls in the critical region, we reject the null hypothesis; if not, we fail to reject it.
Every hypothesis test follows these four broad steps: state the hypotheses, set the decision criteria, collect and analyze the data, and make a decision. We’ll run through those steps with the Spider-People, then look back at the choices and reasoning within each one.
The \(z\)-test
The specific variety of test we’ll use here is called the \(z\)-test. The \(z\)-test is appropriate when we know the population standard deviation, and the sampling distribution of the mean is normal or approximately normal. As the name suggests, the test statistic is a kind of \(z\)-score:
\[z = \frac{M - \mu_0}{\sigma_M} \tag{8.1}\]
The numerator is the difference between the sample mean and the null hypothesis mean. The denominator is the standard error of the mean, which describes the variability of the null distribution. The resulting \(z\)-score tells us how many standard errors our sample mean is from the null hypothesis mean. (Compare this to the original \(z\)-scores we learned about in Chapter 5, which measured how many standard deviations an individual score was from the population mean. Same idea, applied to a different distribution.)
For this example, we’ll treat the three observations as independent and assume that, under the null, their population is normal with mean 284 and standard deviation 50. Those assumptions make the \(z\)-test appropriate here. (But note that I just made up the \(\sigma = 50\) and I told you I was taking liberties with the shape of the population. In real research, we rarely know the population standard deviation, so the \(z\)-test is not used very often. But it is a good place to start.)
Applying the full NHST \(z\)-test procedure to our Spider-Sample, the first step is to state our formal hypotheses. We’ll use a non-directional alternative hypothesis: \(H_0: \mu = 284\) and \(H_1: \mu \ne 284\).
The equations from Chapter 7 tell us that the resulting null sampling distribution has the same mean supplied by our null hypothesis (\(\mu_M = \mu_0 = 284\)) and a standard error of \(\sigma_M = \sigma / \sqrt{n} = 50 / \sqrt{3} \approx 28.87\) ms. The Central Limit Theorem tells us about its shape: if the population is normal, the sampling distribution is normal at any sample size; otherwise, it becomes approximately normal as the sample size increases. Because our sample contains only three observations, the test relies on our simplifying assumption that reaction times are normally distributed; with a large enough sample size, we could still run the \(z\)-test even if the population were skewed.
For the decision criterion, we’ll use \(\alpha = .05\). Our non-directional alternative hypothesis states that a change in either direction would count as evidence against the null. Therefore, we split that five percent between the two tails of the distribution, making this a two-tailed test. Safe in the knowledge that the sampling distribution we’re dealing with is normal, we can find the \(z\)-score that cuts off 2.5% in each tail in the unit normal table.
Table 8.1: An excerpt of the unit normal table from \(z = 1.90\) to \(z = 2.00\), highlighting the .025 upper-tail area at \(z = 1.96\) for a two-tailed test with \(\alpha = .05\).
Excerpt of the unit normal table for z from 1.90 to 2.00, with the area to the right of z = 1.96, .025, highlighted.
Since the standardized normal distribution described by the table is symmetrical, the critical values for a two-tailed test are \(z = \pm 1.96\). That means if our sample mean is more than 1.96 standard errors above or below the null mean, we will reject the null hypothesis. (Another way to think about this is that we could convert the \(\pm 1.96\)\(z\)-scores back into the original units of milliseconds, using the rearranged equation \(\mu_0 \pm z \times \sigma_M = 284 \pm 1.96 \times 28.87\), giving us critical regions below 227.4ms and above 340.6ms. If our sample mean is beyond those limits, we will reject the null hypothesis, because those are the times which put the mean’s corresponding \(z\) beyond \(\pm 1.96\).)
Next, we take our data and calculate the test statistic. First, just for comparison, recall that Peter’s individual score gave us \(z = -1.9\). That value would not be within the critical regions of the null distribution; it is less extreme than the cutoff of \(\pm 1.96\) that we obtained from Table 8.1. So now we know that Peter’s score by itself was a bit unusual, but not quite extreme enough to reject the null hypothesis, according to our stated \(\alpha = .05\).
For the sample of three, however, putting the values into the equation gives us a \(z\)-score of \(-3.29\):
The numerator is the same difference we calculated for Peter alone. It’s the denominator that changes: we divide by the standard error of the mean rather than the standard deviation of individual scores.
The resulting \(z = -3.29\) for the sample of \(n = 3\) is quite a bit more extreme than \(-1.9\) for the single score, because the distribution of sample means has lower variability than the distribution of individual scores; a sample mean as far from the population mean as this is less common than an individual score that far away. Now the \(z\)-score for the sample is beyond the critical values we stated above, \(\pm 1.96\). That puts the sample mean in the lower critical region, so we would reject the null hypothesis. We call this result statistically significant at \(\alpha = .05\).
Figure 8.3 shows what the sampling distribution looks like and where our sample mean falls. I’ve included axes for both the original units (milliseconds) and the \(z\)-scores representing the sampling distribution, and shaded the critical regions red. Remember, the \(z\)-scores are just a change of label. You can see our sample mean pretty far down in the extreme tail of the distribution.
Figure 8.3: The null sampling distribution for means of \(n = 3\) reaction times, with critical regions beyond \(z = \pm 1.96\) and the observed mean of 189 ms at \(z = -3.29\).
Sampling distribution of mean reaction times for samples of three, centered on 284 ms with standard error 28.87 ms; aligned raw-score and z-score axes show the same positions, the two critical regions beyond z = ±1.96 are shaded red, and a vertical line marks the sample mean of 189 ms at z = −3.29.
In other words, if spider bites leave the population mean at 284 milliseconds, getting a sample of three people with an average of 189 milliseconds is very unlikely. The observed sample is difficult to reconcile with our assumption that the bites did nothing. That doesn’t prove what caused the difference, but it gives us evidence against the null hypothesis.
We’re Going to Need More Spiders
0 of 2
1
Suppose we had tested nine bitten kids instead of three, and their
mean reaction time was still 189 milliseconds. Using the same population
values, \(\mu_0 = 284\) and \(\sigma = 50\), what is the \(z\)-score for this sample mean?
The standard error is now \(\sigma_M = 50 /
\sqrt{9}\).
Answer
-5.7
Explanation
\(\sigma_M = 50 / \sqrt{9} =
16.67\), so \(z = -95 / 16.67 =
-5.70\). That’s even farther into the lower tail than our sample
of three, at \(z = -3.29\).
Check your work
What standard error did you calculate?
Answer
16.67
Explanation
\(\sigma_M = 50 / \sqrt{9} = 50 / 3 =
16.67\).
2
One kid at 189ms gave \(z = -1.9\),
three kids averaging 189ms gave \(z =
-3.29\), and nine kids averaging 189ms gave the answer above. Why
does the same mean become more extreme as the sample gets bigger?
Answer
The standard error gets smaller, so the same difference spans more
standard errors.
Explanation
The numerator, \(189 - 284\), is the
same every time. What changes is the denominator. Means of larger
samples vary less from sample to sample, so the null distribution gets
narrower, and a mean of 189 lands farther out in its tail.
Hypothesis tests, step by step
Having worked through that \(z\)-test, we can scrutinize the logic and choices we made in more detail. When we learn other variations of hypothesis tests, using different statistics, the details of the calculations will change, but the general procedure will remain the same.
Step 1: State the hypotheses
We articulated two hypotheses: the null hypothesis, which we assume to be true for the purpose of the test, and the alternative hypothesis, which we will entertain if the data are inconsistent with the null. The null hypothesis usually states that there is no effect or no difference. The alternative hypothesis is usually a statement that there is an effect or a difference.
These two propositions are logical complements; one or the other must be true. But they aren’t exactly symmetric. The name of our procedure, Null Hypothesis Significance Testing, tells you that the null plays a more important role. Ronald A. Fisher, a British statistician and one of the central figures in the development of modern statistics and significance testing, stated the case bluntly: “Every experiment may be said to exist only in order to give the facts a chance of disproving the null hypothesis.” (Fisher, 1935, p. 16)
“Disproving” is overstating things a bit. A statistical result can provide quantitative evidence, but cannot establish with logical certainty that the null hypothesis is false. But Fisher’s line captures the centrality of the null. In our two-tailed example, the null supplies a specific population mean: the population mean reaction time remains exactly 284 milliseconds after a spider bite. Along with our other assumptions, that value defines a specific sampling distribution. It therefore gives us something we can check directly: if the population mean really is 284, how often would we get a sample mean this far from it or farther?
The alternative hypothesis is less specific. If the null says that the population mean is 284, the alternative just says that it is not 284. It usually doesn’t supply a particular value around which we can construct an alternative sampling distribution. So we aren’t really testing the alternative hypothesis directly; we just entertain it if the data provide sufficient evidence against the null.
A result that would be very unusual under the null gives us reason to reject it, though it doesn’t prove any specific alternative. Rejecting the null because our sample mean was 189ms doesn’t mean we’re accepting 189 as the true mean of the hypothetical population of Spider-People. We’re just saying it doesn’t seem like \(\mu_{\text{bitten}} = 284\). (We could state a specific prediction, such as \(\mu_{\text{bitten}} = 250\), giving us another sampling distribution to work with. That will be part of assessing statistical power in the next chapter. But our alternative here, \(\mu_{\text{bitten}} \neq 284\), does not make one precise numerical prediction.)
Hypotheses don’t have to concern a treatment. Say we’re interested in whether there is an association between the number of years of education someone has completed and their lifetime earnings.
General research question: Is there an association between education and income? 🎓🤑
\(H_0\): There is no association between education and income.
\(H_1\): There is an association between education and income.
Stating the alternative also means deciding whether it is directional. A directional hypothesis predicts an increase or a decrease; a non-directional hypothesis allows for a difference either way. With the spider bites, we were following the comic book logic that reaction times would be improved, so we could justify a directional hypothesis:
Here the null includes both no change and a change in the other direction. We calculate the cutoff at its boundary, \(\mu = 284\), using the same sampling distribution as before. Higher population means would make a sample mean low enough to reject the null even less likely.
But we can’t always be so sure about the direction. Perhaps in the real world radioactivity is not good for your reactions. If either change would be worth knowing about, we can use a non-directional hypothesis. Picking a direction means that even an extreme result in the other direction will not lead us to reject the null in this test. In practice, social scientists usually use non-directional hypotheses, for reasons we’ll come to in step two, where we’ll use this decision to place the critical region—or regions.
We won’t say much more about formulating hypotheses in this book. We’ll focus more on the statistical analysis used to test them. But if you really think about it, formulating our hypotheses should be by far the most important and effortful step of the entire research process. Formulating new, interesting, sensible, plausible, meaningful hypotheses isn’t easy. Even to begin to come up with reasonable and potentially insightful hypotheses requires familiarity with the existing research and theoretical perspectives on the phenomena you’re interested in, a coherent definition of the construct(s) in question, and a good sense of what makes for a good operational definition.
State a Pair of Hypotheses
Come up with a psychological research question of your own. State a
null and an alternative hypothesis for it, and decide whether the
alternative should be directional.
Example answer
Research question: Does sleep deprivation affect memory? \(H_0\): Sleep deprivation has no effect on
the mean number of words people recall. \(H_1\): Sleep deprivation changes the mean
number of words people recall. This alternative is non-directional; I
might predict that sleep deprivation makes recall worse, but an
improvement would also be worth knowing about.
Explanation
Check that your null and alternative cover all possibilities without
overlapping, so that one or the other must be true. If you made your
alternative directional, ask yourself whether you’d really be willing to
treat a result in the other direction, however extreme, as no evidence
against the null.
Step 2: Set the decision criteria
For step two, we’re asking, “if the null hypothesis is true, what sample statistics are we most likely or unlikely to see?” We already decided in step one which directions would count as evidence against the null. Now we choose how much of the null distribution will count as sufficiently unusual and put the critical region in the appropriate place. That proportion is \(\alpha\), which is also called the significance level.
A useful way to think about \(\alpha\) is that it reflects our tolerance for rejecting the null hypothesis by mistake. The sampling distribution we obtain from the null hypothesis does contain those extreme outcomes, after all; \(\alpha\) quantifies how often they would occur if we were to repeat the same experiment many times, assuming that the null is true. In other words, \(\alpha\) is the long-run probability of rejecting a true null hypothesis. Since we usually run our study just once, we can’t know whether it is one of those rare (but possible) extreme samples. Our rejection of the null is a gamble, and \(\alpha\) sets how often we are willing to lose it when the null is true.
By convention, psychology often uses an alpha level of .05. Why five percent? Ronald Fisher called it a convenient boundary in an influential 1925 statistics book, building on an older rule of thumb that results about two standard deviations from expectation were noteworthy (Fisher, 1925, pp. 47–48). Exactly two standard deviations would put \(.0228\) in each tail of the distribution, \(.0456\) total split between the two tails. So \(\alpha = .05\), or 5%, was a nice round number close to that, and you know how people like round numbers. It also caught on partly out of practicality. Textbooks often included tables for a few values of alpha rather than the whole continuous range of proportions. The convention stuck, though other thresholds can be more appropriate when the consequences of a false alarm differ.
One- and two-tailed tests
A directional hypothesis leads to a one-tailed test, with one critical region in the predicted direction. A non-directional hypothesis leads to a two-tailed test, with critical regions at both extremes. So if \(\alpha = .05\), a two-tailed test divides it between the tails, placing two and a half percent at each end of the distribution with 95% in the middle. Compare the regions in Figure 8.4.
Figure 8.4: Critical regions for \(\alpha = .05\). A one-tailed test puts all five percent in the predicted tail; a two-tailed test splits it between the two.
Standard normal curve with the lowest 5% shaded as a single critical region below z = −1.64.
Standard normal curve with two critical regions shaded, 2.5% below z = −1.96 and 2.5% above z = +1.96.
The associated critical value(s) can be found in the unit normal table, looking for the column that gives the proportion of the curve beyond a given \(z\)-score. For a one-tailed test with \(\alpha = .05\), we look for an area of about .05 in one tail:
Table 8.2: An excerpt of the unit normal table from \(z = 1.62\) to \(z = 1.67\), highlighting the .0505 upper-tail area at \(z = 1.64\) for a one-tailed test with \(\alpha = .05\).
Unit normal table excerpt: the area to the right is .0526 at z = 1.62, .0516 at 1.63, .0505 at 1.64, .0495 at 1.65, .0485 at 1.66, and .0475 at 1.67. The .0505 entry at z = 1.64 is highlighted.
In Table 8.2, at \(z = 1.64\), the area to the right is \(.0505\), or about five percent. The exact five-percent cutoff lies between \(1.64\) and \(1.65\) (about \(1.645\)), but our table gives \(z\)-scores to two decimal places. We’ll use \(1.64\) as the table-based approximation in this chapter and its quizzes. For an upper-tailed test, the cutoff is about \(+1.64\); by symmetry, for a lower-tailed test it is about \(-1.64\). For a two-tailed test, we look for .025 in each tail, giving the cutoffs of about \(\pm 1.96\) (as we found in Table 8.1).
Two-tailed tests are the default in the social sciences for the most part, even when researchers do have a hunch about the direction an effect might go in. Splitting alpha between both tails means that a result in the predicted direction has to be more extreme to reject the null: beyond \(-1.96\) rather than \(-1.64\) for a one-tailed test. So the two-tailed test is the more cautious choice, and it leaves room for an unexpected result in the other direction (like the possibility that spider bites are bad for your reaction speed, actually).
It’s also useful to stick with two-tailed tests as a default because it deters researchers from switching to a directional test after seeing the data. When we specify alpha and determine the critical regions, we’re following through on the hypotheses we stated in step one, and we’re still doing it before looking at the data. As you’ll see in Activity 8.1, it’s possible to have a result that would be significant in a one-tailed test but not in a two-tailed test. Changing the decision criteria after looking at the data would not make for a meaningful hypothesis test, any more than shooting arrows at the side of a barn and drawing a target around them afterwards would make for good target practice. Because a one-tailed test makes the null easier to reject in the predicted direction, it’s fair to be skeptical of one-tailed tests unless the direction was clearly committed to in advance.
Step 3: Collect and analyze the data
So having stated our hypotheses and decided on the decision criteria, we collect the data and calculate the appropriate sample statistics.
Again, this order of events is crucial to the logic of the hypothesis test: we’re testing how well the data fit a model specified in advance, rather than forming a model around the data we happened to find. This is sometimes called confirmatory hypothesis testing. Exploring data and noting interesting patterns can be valuable too, and those patterns can suggest new hypotheses. But that’s a different use of the data. A hypothesis suggested by the results should be treated as exploratory. To test it confirmatorily, we’d need new data.
The exact test statistic we calculate depends on the nature of the data. We’ll encounter several other tests later, including the \(t\)-test, analysis of variance (ANOVA), and correlation and regression. They all rely on the same fundamental hypothesis-testing ideas we’re outlining here. Recall that the reason we used a \(z\)-test for the Spider-People is that we assumed we knew the population standard deviation. If we didn’t want to make that assumption, we’d need a slightly different test, as we’ll see in Chapter 11.
As long as we’re talking about collecting data, allow me to remind you that the version of the Central Limit Theorem we’re using assumes independent observations drawn from the same population distribution. Random sampling is one way to support those assumptions, though in practice social scientists often collect data in less tidy ways, as we mentioned in the previous chapter.
Step 4: Make a decision
If the test statistic falls in the critical region, we reject the null hypothesis. Otherwise, we fail to reject it. That’s how we reached our decision for the Spider-People: \(z = -3.29\) was beyond the lower critical value of \(-1.96\).
There’s an equivalent way to reach the same decision, using the \(p\)-value, which we calculate from the observed data. It is the probability, assuming the null model and its other assumptions are correct, of obtaining a test statistic as extreme as or more extreme than the one we observed. For the Spider-People, it’s the probability of getting a sample mean at least as far from 284 as ours, if the null hypothesis is true.
Our Spider-Sample mean of 189ms produced a \(z\)-score of \(-3.29\). We can find the \(p\)-value associated with that score by looking it up in the unit normal table.
Table 8.3: An excerpt of the unit normal table from \(z = 3.25\) to \(z = 3.33\), highlighting the .0005 upper-tail area at \(z = 3.29\) used to find the Spider-People sample’s \(p\)-value.
Excerpt of the unit normal table for z from 3.25 to 3.33, with the area to the right of z = 3.29, .0005, highlighted.
Table 8.3 gives an area of \(.0005\) to the right of \(z = 3.29\). By symmetry, \(P(z \le -3.29) = .0005\) as well. Because our hypothesis is non-directional, an equally extreme result in the other tail also counts, giving a two-tailed \(p\)-value of approximately \(.0010\).
If \(p < \alpha\), this is another way of saying that the test statistic is in the critical region, and we reject the null hypothesis: our result is statistically significant. Here \(p \approx .001\), well below \(\alpha = .05\), so we reach the same decision as the critical-value comparison.
If \(p \ge \alpha\), we fail to reject the null hypothesis, and we call the result nonsignificant. (Not insignificant! In everyday language, “significant” suggests something large or important. Here it only indicates whether the result cleared the bar we set with \(\alpha\).) When you see \(p < .05\) in a published journal article, it usually means \(\alpha\) was chosen to be \(.05\), and the \(p\)-value for the data was less than that.
Failing to reject the null isn’t the same as accepting it or showing that nothing happened. It means this study did not provide sufficient evidence against it.
Activity 8.1 shows the decision-making process in action. We can compare how different choices would lead to different decisions with the same data, starting with our Spider-People.
Activity 8.1: Explore alpha and critical regions
Figure 8.5: How the location of the critical region, the observed \(z\)-score, and \(\alpha\) affect the decision to reject the null hypothesis.
Explore alpha and critical regions. Use the accompanying steps for an explanation.
Choose Before Looking
0 of 2
1
Before collecting any data, a researcher predicts that a treatment
will reduce scores, and specifies a lower-tailed test with
\(\alpha = .05\). The study produces
\(z = +2.40\). What should they
conclude?
Answer
Fail to reject the null hypothesis, because the result is not in the
lower-tailed critical region
Explanation
With all of \(\alpha\) in the lower
tail, the only critical region lies below about \(-1.64\), so \(z =
+2.40\) is not in it; its lower-tailed \(p\) is about .99. Switching to a two-tailed
or upper-tailed test after seeing the result would mean choosing the
decision rule to fit the data. The unexpected increase might be worth
following up, but as an exploratory finding to test with new data.
2
A researcher planned to use \(\alpha =
.01\) and obtained \(p = .032\).
The pattern looks interesting, so after seeing the result they switch to
\(\alpha = .05\) and call it a
confirmed finding. What would be more appropriate?
Answer
Report that the result did not meet the planned criterion
Explanation
Under the planned rule, \(.032 >
.01\), so the result is not statistically significant. The
pattern might still be worth exploring, but as an exploratory finding to
test with new data. Changing the rule after seeing the data draws the
target around the result rather than testing the result against a target
fixed in advance.
Practice: CBT for OCD
We’ll end by having you perform your own hypothesis test, revisiting the cognitive behavioral therapy example from Chapter 5. Recall I told you there about a study using the Dimensional Obsessive-Compulsive Scale to measure symptoms before and after treatment (Abramowitz et al., 2010). The quiz below uses the same numbers we used in that chapter, but this time assuming they came from a sample rather than an individual. This will give you practice with all the steps of the procedure: articulating the null and alternative hypotheses, setting your decision criteria, calculating the test statistic, and making your decision.
For this practice, treat the untreated population as having \(\mu = 30.25\) and \(\sigma = 14.89\), and suppose a sample of \(n = 40\) people who completed treatment has a mean of \(M = 15.49\).
A \(z\)-test for CBT
0 of 4
1
Step 1. We want to know whether the treated population’s mean differs
from 30.25 in either direction. Which pair of hypotheses fits?
The null says that the treated population mean remains at the
reference value. The non-directional alternative allows it to differ in
either direction. Hypotheses are about population parameters, not the
sample mean we happen to observe.
2
Step 2. With a non-directional alternative and \(\alpha = .05\), what are the critical
values?
Answer
\(z = \pm 1.96\)
Explanation
A two-tailed test splits \(\alpha\)
between the tails, so we need the \(z\)-score that leaves .025 in each: \(\pm 1.96\). A single cutoff of \(-1.64\) would be a one-tailed test. Using
\(\pm 1.64\) would put about five
percent in each tail, for an \(\alpha\) of .10, and \(\pm 2.58\) corresponds to \(\alpha = .01\).
3
Step 3. For the purposes of this practice problem, treat \(\mu_0 = 30.25\) and \(\sigma = 14.89\) as known
reference-population parameters. The treated sample has \(M = 15.49\) with \(n = 40\). Calculate the \(z\)-score for the sample mean.
Hint 1
First find the standard error: \(\sigma_M =
\sigma / \sqrt{n}\).
Step 4. Compare your \(z\)-score
with the critical values. What do you decide?
Answer
Reject the null hypothesis
Explanation
\(z = -6.27\) is far beyond the
lower critical value of \(-1.96\), so
the result is statistically significant and we reject the null. That’s
evidence against the null, not proof of the alternative. And a
significance test alone can’t tell us that the therapy caused the
change.
Now that you’ve done that, you should know that I mishandled the data again. As before, the numbers are from the real study, but 30.25 and 14.89 were pre-treatment sample statistics, not known population parameters. More importantly, the means of 30.25 and 15.49 came from the same 40 people measured before and after treatment. That makes this a related-samples design, so this \(z\)-test is not the analysis we would use for the real study. We’ll return to the actual design, and the test it calls for, when we introduce the related-samples \(t\)-test in Chapter 13.
Summary
NHST gives us a disciplined way to ask whether a sample result is difficult to explain under a null model. But a result being significant or nonsignificant isn’t the end of the story. Determining that an effect is statistically significant doesn’t mean that it is necessarily large or of practical importance. And a nonsignificant result needn’t mean that there is no effect. To understand those possibilities, we need to look more closely at effect size, inferential errors, and statistical power, our topics for the next chapter.
Learning Checks
0 of 7
1
In our two-tailed Spider-People example, why do we assess the null
hypothesis rather than testing the alternative directly?
Answer
The null states a specific value, which defines a specific sampling
distribution.
Explanation
A null such as \(\mu = 284\) gives
us one precise null distribution to compare our result against. An
alternative such as \(\mu \neq 284\)
allows for any other value, so it doesn’t give us one particular
distribution to work with.
2
Which of these do we need in order to use a \(z\)-test for a sample mean?
Answer
The population standard deviation, \(\sigma\)
Explanation
The \(z\)-test uses \(\sigma\) to calculate the standard error,
\(\sigma_M = \sigma / \sqrt{n}\).
That’s why it’s rarely used in practice. (It also needs the sampling
distribution of the mean to be normal, which a small sample only gives
us if the population itself is normal.)
3
Before collecting data, a researcher predicts that a treatment will
increase scores, and plans a one-tailed test with \(\alpha = .05\). What is the approximate
critical value, using this chapter’s table?
Answer
\(z = +1.64\)
Explanation
All five percent goes in the predicted tail, and an increase means
the upper tail. The table gives an upper-tail area of .0505 at \(z = +1.64\), our approximation for the
five-percent cutoff. The values \(\pm
1.96\) leave .025 in each tail, which is what a two-tailed test
needs.
4
A study reports \(p = .03\). What
does that mean?
Answer
If the null were true, a result at least this extreme would happen
about 3% of the time.
Explanation
A \(p\)-value is calculated by
assuming the null is true. So it can’t also tell us the
probability that the null is true. It tells us how unusual our result
would be in a world where the null hypothesis holds.
5
A researcher sets \(\alpha = .05\).
What does that five percent describe?
Answer
How often, in the long run, we would reject the null when it is
actually true
Explanation
If the null is true, five percent of possible results still land in
the critical region, so a researcher who uses \(\alpha = .05\) will reject a true null
about five percent of the time. For any one study, though, we can’t know
whether it was one of those unlucky samples.
6
A result is statistically significant at \(\alpha = .05\). What does that tell us?
Answer
The result fell in the critical region, so \(p < .05\).
Explanation
“Significant” only means that the result cleared the bar we set with
\(\alpha\). It doesn’t mean the effect
is big or important, it doesn’t prove the null false, and a significance
test alone can’t establish what caused the difference.
7
A study produces \(p = .12\), with
\(\alpha = .05\). What is the
appropriate conclusion?
Answer
Fail to reject the null; this study didn’t provide sufficient
evidence against it.
Explanation
Since \(p > \alpha\), we fail to
reject the null. But that doesn’t show the null is true: a real effect
might have been missed. The \(p\)-value
isn’t the probability that the null is true. And “nonsignificant” isn’t
a verdict about how big or important an effect is.
Abramowitz, J. S., Deacon, B. J., Olatunji, B. O., Wheaton, M. G., Berman, N. C., Losardo, D., Timpano, K. R., McGrath, P. B., Riemann, B. C., Adams, T., Bjorgvinsson, T., Storch, E. A., & Hale, L. R. (2010). Assessment of obsessive-compulsive symptom dimensions: Development and evaluation of the dimensional obsessive-compulsive scale. Psychological Assessment, 22(1), 180–198. https://doi.org/10.1037/a0018260
Fisher, R. A. (1925). Statistical methods for research workers. Oliver; Boyd.
Fisher, R. A. (1935). The design of experiments. Oliver; Boyd.
