9  Statistical Power

A null distribution with red critical tails, followed by a shifted alternative distribution with power shaded violet beyond the same critical values.

In this chapter we’ll develop our understanding of hypothesis testing by exploring some related concepts: effect size, inferential errors, and statistical power. These are nuanced ideas, and statistical power in particular can be difficult to get your head around both conceptually and computationally. But they are key to fully understanding the strengths and limitations of hypothesis testing, so we’ll take our time with them here before moving on to other ideas and tests.

NHST logic recap

First, let’s recap the ideas we covered in the previous chapter as a way of refining our thinking and preparing for the new aspects we’ll examine. We’ll do so in a different way than before, not by looking at a single dataset or one boring old curve. Instead we’ll use something more dynamic, more delightful, more fuzzy and yellow.

ActivityNHST in action: a swarm of bees

Animated swarm of 15 bees illustrating sampling error, a hidden population mean, the null model, and false alarms. The true hive then shifts while the null and its critical regions stay fixed, illustrating correct detections and misses.

It’s a swarm of bees! Well, a simulated swarm: little yellow circles following a flocking algorithm. Aren’t they charming? Watch them buzz this way and that for a moment. They try not to bump into each other. And they always know where their hive is—the brown diamond in the middle. They venture away, but always return, bringing back little bits of pollen or whatever it is they’re up to. Bees!

What does this have to do with null hypothesis significance testing, you may ask? Well, in this analogy, the hive at the center is the population mean, \(\mu\). It is the real and unchanging center. The bees buzzing around it are a sample, constantly changing from one moment to the next as if we’re drawing repeated samples from the population.¹ The sample at any particular instant gives a clue about those population characteristics. NHST offers one way of interpreting the clues.

We can freeze time. At this instant, the dashed line marks the average horizontal position of all the bees together: their sample mean, \(M\). (We’ll only care about horizontal position; the up-and-down motion is just for show.) It’s close to the hive, but not exactly on it. That’s sampling error, the discrepancy between the sample statistic and the population parameter.

Now, what if we didn’t know for sure where the hive was? I’ve hidden it, but it still exists and hasn’t changed. We can only see the sample mean, \(M\). If we were to use that as our estimate of the population mean, we know we wouldn’t be exactly right, but we don’t know the sample’s exact error.

When we’re really doing research, we usually don’t know the true population mean. So under NHST, we specify a null hypothesis: a model in which the population mean has a particular value, \(\mu_0\). I’ve replaced the hive with a dashed outline representing our null hypothesis about the hive’s position. It so happens that we’re exactly right; the null predicts that the hive is where it really is.

So now we can describe the sampling distribution of the mean derived from that null hypothesis (its guess about the population center, as well as knowledge of the population variability, sample size, and normality, as we discussed in Chapter 7). We set alpha (here .05) and define the critical regions. When a sample mean is in those regions, we would reject the null hypothesis.

Let’s let the swarm get back to work. The mean jitters around the hive, occasionally straying quite far away but generally staying close to the hidden hive and its equivalent dashed null outline. Occasionally it is far enough away that we would reject the null for that sample at that moment. Remember, we know that the null here is true—it is right where the true population center is—so those rejections are mistakes, false alarms.

Let’s start keeping track of the sample means, building a histogram of their positions over time. Watch the bees a while and see how the distribution forms. Or if you want to speed things along, click the fast-forward button to add a minute of simulated samples. We’ll also track how much of that distribution falls into the critical regions of the null distribution, i.e. the proportion of samples that would lead us to reject the null. Over time, this proportion should be roughly five percent, because that’s the definition of \(\alpha = .05\).

Now, what if the null is wrong? I’m going to show the hive again, and also move it over a bit. So now the null hive—the dashed outline—makes an incorrect prediction about where the true population center is. This changes our thought process a bit. Rather than risking falsely rejecting a correct null, we need to think about how likely it is we will correctly reject this wrong null. And what are the chances we will fail to reject it, making another kind of mistake?

The last step gives us an outline for the rest of this chapter. The distance between the null and the actual hive—the difference between the null and true population means—is an effect size. We’ll first look at how to describe that difference relative to population variability, then examine the decisions and mistakes we can make, and finally arrive at the idea of statistical power, the probability that we’ll detect a real effect of a given size.

Effect Size

A “statistically significant” result, like the one we found for the sample of three Spider-People in the previous chapter, tells us that the observed test statistic is sufficiently improbable under the null model to meet our chosen criterion. It’s a binary decision, reject or fail to reject the null; \(p < \alpha\) or \(p \ge \alpha\). It doesn’t tell us how big the difference is. In everyday language, “significant” often means important, big, meaningful, but not in hypothesis testing. In this context, it only means we’ve met a statistical decision rule, nothing more or less.

The size of the difference is part of the \(z\) equation from Chapter 8:

\[ z = \frac{M - \mu_0}{\sigma_M} \]

Here, \(M\) is the sample mean, and \(\mu_0\) is the mean specified by the null hypothesis, so the difference between them is right there in the numerator. But that gets divided by the standard error, \(\sigma_M\). You’ll remember from the previous chapter that standard error is calculated by dividing the population standard deviation by a function of sample size, \(\sigma_M = \frac{\sigma}{\sqrt{n}}\). For a given difference and population variability, increasing \(n\) reduces the standard error and makes its \(z\) more extreme. Even a tiny difference can therefore be statistically significant with a large enough sample. Conversely, a substantial difference can be nonsignificant with a small sample.

So the significance of the test is only part of the story. Measures of effect size describe the magnitude of a difference or relationship. For our one-sample test, we could report the raw difference, \(M - \mu_0\), in the original units. The bigger the difference, the larger the effect. But it’s often nice to have a standardized measure that allows us to compare effects across different studies and contexts.

Cohen’s \(d\)

One such standardized measure is Cohen’s \(d\). (Cohen didn’t say so himself, but let’s all agree that the \(d\) stands for “difference.”) For our \(z\)-test, where the population standard deviation is known, we calculate Cohen’s \(d\) as:

\[ d = \frac{M - \mu_0}{\sigma} \tag{9.1}\]

Notice how close that is to the sample \(z\)-score equation, \(z = \frac{M - \mu_0}{\sigma_M}\). The numerator is exactly the same, but the denominator is the plain population standard deviation, \(\sigma\), rather than the standard error. A value of \(d = 0.5\) means the sample mean is half a population standard deviation above the null mean; \(d = -0.5\) means it is half a standard deviation below. The sign tells us the direction, while the absolute value tells us the magnitude.

Unlike \(z\), this calculation does not increase automatically with sample size: the same \(M\), \(\mu_0\), and \(\sigma\) give the same \(d\) regardless of \(n\). So in this sense, the effect size is independent of sample size. (But \(d\) calculated from a sample is still an estimate, and larger samples generally give more precise estimates of the population effect.)

Effect size and statistical significance

Let’s demonstrate the difference with a hypothetical study. Suppose we’re educational psychologists trying to develop an intervention to boost students’ SAT performance. And suppose we think we’ve hit upon an effective treatment: giving the test taker a banana just before taking the test. Maybe the potassium helps with cognitive function or something like that, who knows. Let’s say the relevant population without the treatment has a mean of \(\mu = 500\) and a known standard deviation of \(\sigma = 100\), making a \(z\)-test appropriate. (And that we’re sampling independently and the sampling distribution is normal.)

We choose a two-tailed test with \(\alpha = .05\) before collecting data, giving critical values of \(z \approx \pm 1.96\). After the banana treatment, we observe a sample mean of \(M = 501\), one point above the null mean, \(\mu_0 = 500\).

First, suppose the sample contains 50,000 students.

\[ z = \frac{M - \mu_0}{\sigma_M} = \frac{501 - 500}{100 / \sqrt{50{,}000}} \approx 2.24 \]

\[ d = \frac{M - \mu_0}{\sigma} = \frac{501 - 500}{100} = 0.01 \]

The result is statistically significant: \(z\) exceeds 1.96. But the effect size is tiny, just one hundredth of a standard deviation.

Now suppose the same sample mean, \(M = 501\), came from a sample of just 1,000 students.

\[ z = \frac{M - \mu_0}{\sigma_M} = \frac{501 - 500}{100 / \sqrt{1{,}000}} \approx 0.32 \]

\[ d = \frac{M - \mu_0}{\sigma} = \frac{501 - 500}{100} = 0.01 \]

In this case, the result is nonsignificant. The smaller sample produces a larger standard error and therefore a less extreme \(z\). But both samples gave the same estimated effect: a difference of one point, or one hundredth of a standard deviation.

Assuming there really is a benefit to eating a banana before this test, one study was able to detect it and reject the null while the other failed to detect it (a kind of inferential error we’ll describe more precisely a little later in the chapter). The sample size helped with statistical significance. But would supplying every student with a banana be a good use of resources? Significance alone can’t answer that. A cheap intervention with a small benefit might be worthwhile; a costly one with the same benefit might not be. Effect size can help to inform that judgment.

Calculate and interpret Cohen’s \(d\)

1

A sample of 25 participants has a mean score of 44. The null mean is 50, and the known population standard deviation is 10. Calculate Cohen’s \(d\).

2

What do the sign and magnitude of your Cohen’s \(d\) tell us about the sample mean compared with the null mean?

Quiz 9.1

Interpreting Effect Size

This raises the question, what should we consider a meaningful effect size? Cohen outlined some rough rules of thumb: \(d = 0.2\) could be considered a small effect, \(d = 0.5\) medium, and \(d = 0.8\) large. However, he strongly cautioned against over-reliance on these guidelines. “The terms ‘small,’ ‘medium,’ and ‘large’ are relative, not only to each other, but to the area of behavioral science or even more particularly to the specific content and research method being employed in any given investigation,” Cohen wrote. “In the face of this relativity, there is a certain risk inherent in offering conventional operational definitions for these terms for use in power analysis in as diverse a field of inquiry as behavioral science” (Cohen, 1988). Cohen apparently regretted having taken that risk, because the rough guidelines came to be applied without much thought (Funder & Ozer, 2019).

Interpreting effect size requires consideration of context. Psychologists Funder & Ozer (2019) stress that even very small effects can matter when they accumulate over time or affect many people. They also caution that unusually large estimates in psychological research should be scrutinized; not that very large effects cannot be real, but they are uncommon within psychological research.

Effect sizes in context

What do differences of these sizes look like in practice? Simmons (2014) reported the following effect-size estimates from an MTurk survey.² (These compare two groups rather than a sample with a fixed null mean, but \(d\) still expresses the difference between means in standard deviation units.) Activity 9.1 visualizes these differences using smoothed curves based on the raw data.

Table 9.1: Effect-size estimates for five group comparisons in an MTurk survey, reported by Simmons (2014).

Groups compared	Outcome	Magnitude of \(d\)
Men / women	Height	1.83
Liberal / conservative	Importance of social equality	0.69
Like eggs / dislike eggs	Frequency of eating egg salad	0.58
Smoker / nonsmoker	Estimated risk of smoking-related death	0.33
Prefer science / prefer art	Number of planets named correctly	0.05

Activity 9.1: Visualizing Cohen’s \(d\)

Smoothed distributions of each pair of groups' survey responses, with each group's mean marked. A bracket labels the difference between means in standard deviation units, and a z scale runs beneath the original scale. The final step shows two idealized normal curves with a slider.

Men’s and women’s heights differed by \(d = 1.83\) in Simmons’ data: well beyond Cohen’s large-effect benchmark of \(0.8\). But notice that the distributions still overlap; some women are taller than some men, even though the averages are quite far apart.

Liberals and conservatives differed in the importance they placed on social equality by \(d = 0.69\), measured on a 7-point scale. That’s between Cohen’s medium and large benchmarks.

People who liked and disliked eggs differed in how often they ate egg salad by \(d = 0.58\). A difference that sounds obvious isn’t necessarily enormous relative to individual variability.

Smokers and nonsmokers differed in their estimates of smoking-related mortality risk by \(d = 0.33\). Notice that the curves don’t have to be normal to calculate \(d\); it is just a standardized difference between means.

People who preferred science and those who preferred art differed in the number of planets they could name by just \(d = 0.05\); in fact, in this case the difference was not statistically significant. This sample doesn’t give us convincing evidence of a difference in planet naming; nor does that nonsignificant result establish that the true difference is zero.

Try your own values, using these normal curves. Cohen’s benchmarks are \(d = 0.2\), \(0.5\), and \(0.8\). Notice how the distance between means and overlap between distributions changes.

Errors in Hypothesis Testing

Null hypothesis significance testing is an inferential process, by which I mean we’re using limited information from a sample to make a general conclusion about a population. At the end of it, there are only two decisions we can make: either we reject or fail to reject the null hypothesis. But there are two potential states of reality: either the null hypothesis is true or it is false. So there are four possible situations in total, shown in Table 9.2. Two are correct decisions; the other two are errors.

Table 9.2: Decisions and their probabilities when the null is true or a specified alternative is true.

		Actual situation
		\(H_0\) true	\(H_0\) false
Researcher’s Decision	Reject \(H_0\)	Type I error \(\alpha\)	Correct \(1-\beta\)
	Fail to reject \(H_0\)	Correct \(1-\alpha\)	Type II error \(\beta\)

Attaching some terms to these possibilities will help us think more precisely about them, and eventually quantify the likelihood of different outcomes.

Type I Error (False Positive)

Starting with the top-left cell: the null hypothesis, \(H_0\), is true in reality, but our test leads us to reject it. This is a Type I error, or false positive: thinking that there is an effect when actually there isn’t.

The probability of making a Type I error is \(\alpha\), provided the null hypothesis and the assumptions of our test are correct. With \(\alpha = .05\), about five percent of repeated samples would land in the critical regions, producing false positives.

Maybe this sounds bad. Why tolerate a 5% chance of a false positive? Well, given the inferential nature of hypothesis testing, we have to tolerate some risk. Too little tolerance and we would never be able to produce satisfying evidence of real effects; too much tolerance and we would be prone to false positives. There’s no law of nature that dictates the appropriate level of risk. As we mentioned in the previous chapter, \(\alpha = .05\) became a common convention among social scientists. But the appropriate balance between false alarms and missed effects depends on their consequences.

In the bottom-left cell, the null is still true, but this time we fail to reject it. We have made the correct decision: our data did not give us enough evidence to reject a true null. That doesn’t mean a nonsignificant result establishes that there is no effect; the same decision could occur when the null is false.

The probability of making the correct decision when the null is true is \(1 - \alpha\). Given a true null, either we reject it or we don’t. Those two options cover all the bases, so their probabilities must sum to 1.

Type II Error (False Negative)

Moving over to the right-hand column of Table 9.2, \(H_0\) is false in reality. So now the possibilities are correctly rejecting or mistakenly failing to reject that false null. The latter is a Type II error, or false negative: failing to detect an effect that really exists.

The probability of making a Type II error is referred to as \(\beta\). Given the names, you might be tempted to think that \(\beta\) is just the flip side of \(\alpha\), but it’s not. Each column describes a different assumed state of reality. The left-hand probabilities apply when the null is true; the right-hand probabilities apply when a particular alternative is true. Within either column, the two outcomes exhaust the possibilities, so their probabilities sum to 1. But \(\alpha\) and \(\beta\) are not complements of each other. (Though, as we’ll see, making \(\alpha\) smaller makes \(\beta\) larger, other things being equal.)

So if the probability of making a Type II error is \(\beta\), then the probability of correctly rejecting the null for that alternative is \(1 - \beta\). That quantity has a name of its own: it is called the statistical power of the test.

Statistical Power

The power of a test is the probability that it will reject the null hypothesis if a specified effect really exists. We’re not saying that we know for sure that the null is false; we’re just choosing a particular effect to consider, and asking how likely it is that we will correctly reject the null under those circumstances. Unlike \(\alpha\), power isn’t something we choose directly. It depends on sample size, variability, effect size, \(\alpha\), and whether the test is one- or two-tailed. If we know (or can make assumptions about) those quantities, we can calculate power under that model.

When we say a test has something like .80 power, we mean it would detect the specified effect about 80 percent of the time if we could repeat the study many times with new, independent samples. We don’t mean there’s an 80 percent probability that the hypothesis, or a particular significant finding, is correct. The probability refers to the test’s ability to detect a real effect, the frequentist idea of long-run proportions of outcomes over repeated samples.

To illustrate, let’s check in with our bees again.

Activity 9.2: Watch power in action: the swarm returns

Animated bee swarm illustrating statistical power: when the hive moves away from the null position, the proportion of sample means that land in the critical regions is the power of the test, and it rises with sample size and effect size and falls with variability.

When we first introduced the bees, they were buzzing around a hive that was exactly where the null hypothesis predicted it to be. The sample means were piling up around the null, and occasionally one would land in a critical region, leading us to reject the null. But since the null was true, those rejections were mistakes: false positives, Type I errors. Over time, the proportion of sample means that landed in the critical regions should settle near \(\alpha = .05\), as expected.

Then we moved the hive. This illustrates another possible reality, in which the null hypothesis is false. The two possible outcomes are that we will either fail to reject the null (a Type II error, with probability \(\beta\)), or that we will correctly reject the null (\(1 - \beta\)).

How can we figure out what those proportions are? Well, we already defined the critical regions for the null hypothesis. They don’t change just because the null is false. They still tell us which sample means would lead us to reject that null, and which would not.

So now we’ll just collect sample means for a while and see where they pile up. The swarm is still jittering around the true hive, and the sample means are piling up in its vicinity. Some of those sample means will land in the critical regions of the null distribution, and some will not. The ones that land in the critical regions are correct rejections of the null hypothesis. The ones that land between the critical boundaries are Type II errors. The proportion of those false negatives gives us \(\beta\).

So to quantify the statistical power of our bees here, all we have to do is turn the tracker back on and check the “proportion significant” value. That’s the proportion of the sample means in the critical regions, which would cause us to reject the null hypothesis and declare a significant result. \(\beta\), the proportion of sample means that land between the critical regions, is the probability of a Type II error.

Soon we’ll walk through calculating power when we don’t have a swarm of bees to watch, but let’s think about what factors might affect a test’s power. What if the hive’s true position was closer to the null’s prediction, or farther away? What if we increased the number of bees here? Or reduced how far they can wander from the hive? Right now, power is around .80. Let’s see how that value changes.

First we’ll move the hive closer to the null, reducing effect size. If you let the bees buzz around a while (or click the fast-forward button), you’ll see that their yellow sampling distribution moves over with the hive, and more of it lands in the nonsignificant region of the null curve. Correspondingly, power—the proportion significant—is reduced.

What if we reduce the population variability? The bees stick closer to their hive. Their yellow distribution becomes narrower, centered on their hive. And the null distribution, responding to the reduced population standard deviation in its standard error, shrinks. This brings the critical regions in closer to the null hive, with the result that more of the sample means land in the critical regions. Power increases.

Let’s add more bees, increasing the sample size. The standard error shrinks again, this time responding to the larger \(n\) in its denominator. The sample means pile up more tightly around the true hive. More of them land in the critical regions, and power increases again.

We’ll explore these factors more systematically in the next section, but for now let’s reset to our original 15 bees producing a test with a power of around .80. That means that if we were to repeat this experiment many times, we would correctly reject the null hypothesis about 80% of the time. The bees here provide an empirical demonstration of that figure, but moving on we’ll show how to calculate it based on the test parameters.

Calculating Power

To calculate statistical power from scratch, we’ll need some step-by-step instructions. I’ll also show you how it looks visualized with some boring old curves, because unfortunately we can’t always do statistics using bees.

The key idea is that we need to think about two distributions: the null hypothesis distribution and the alternative hypothesis distribution. Our bees built their own alternative sampling distribution as they buzzed around their hive, but in practice, we just use the same principles to create the alternative distribution that we use for the null sampling distribution. The alternative distribution then represents the distribution of sample means we would expect to see if some specific alternative hypothesis were true. This requires having some idea of the effect size, usually based on prior research or theory, and, for our \(z\)-test, knowing the population standard deviation.

For a two-tailed test, here are the steps we need to follow to calculate statistical power:

1. Find the null distribution’s critical values, given \(\alpha\)
2. Find the null distribution raw sample means corresponding to those critical values
3. Find the \(z\)-scores for those sample means under the alternative hypothesis distribution
4. Find the area of the alternative distribution that falls between those \(z\)-scores. This gives \(\beta\), the probability of a Type II error
5. Subtract \(\beta\) from 1 to get the statistical power of the test

Let’s work through the steps with some real numbers, illustrating what happens at each step along the way. The parameters driving the bee swarm simulation:

\(\mu_0 = 50\); \(\sigma = 38.73\); \(n = 15\); \(\sigma_M \approx 10\); and the alternative population mean, \(\mu_1 = 78\).

(What are the units of measurement? Bee-scores, of course.)³

Step 1. For a two-tailed test with \(\alpha = .05\), the unit normal table gives critical \(z\)-scores of \(\pm 1.96\). So far this is just the regular decision-criterion step of the NHST procedure.

Table 9.3: An excerpt of the unit normal table highlighting the .0250 upper-tail area at \(z = 1.96\), used to set the critical values for a two-tailed test with \(\alpha = .05\).

Excerpt of the unit normal table for z from 1.92 to 2.00, with the area to the right of z = 1.96, .0250, highlighted.

Step 2. Convert those critical \(z\)-scores into raw sample means, using the rearranged \(z\) equation with the null distribution’s characteristics:

\[ M = \mu_0 + z \cdot \sigma_M \]

We obtained two \(z\)-scores in step one, corresponding to the two-tailed critical regions. (Statistical power can work with a one-tailed test as well; we would only have a single critical \(z\)-score in that case.) So we can plug those into the equation:

\[ M_{\text{lower}} = 50 + (-1.96) \cdot 10 = 30.40 \]

\[ M_{\text{upper}} = 50 + (1.96) \cdot 10 = 69.60 \]

Step 3. Now we turn to the alternative distribution, centered at \(\mu_1 = 78\). The standard error is the same, because the sample size and population standard deviation are the same. We need \(z\)-scores for the same sample means, but measured from this distribution’s center. So we use \(\mu_1\) in the numerator of the \(z\) equation:

\[ z_{\text{lower}} = \frac{30.40 - 78}{10} \approx -4.76 \]

\[ z_{\text{upper}} = \frac{69.60 - 78}{10} \approx -0.84 \]

Step 4. Find the proportion of the alternative distribution between those \(z\)-scores. Using the unit normal table, we subtract the area to the left of the lower boundary from the area to the left of the upper boundary. By symmetry, the area to the left of \(z = -0.84\) equals the area to the right of \(z = 0.84\), which the table gives as .2005. The area to the left of \(z = -4.76\) rounds to .0000: \(.2005 - .0000 = .2005\). This is \(\beta\): the proportion that will cause us to fail to reject the false null.

Table 9.4: An excerpt of the unit normal table highlighting the .2005 upper-tail area at \(z = 0.84\). By symmetry, this is also the area below \(z = -0.84\), used to calculate \(\beta\).

Excerpt of the unit normal table for z from 0.80 to 0.88, with the area to the right of z = 0.84, .2005, highlighted.

Step 5. Statistical power is \(1 - \beta = 1 - .2005 = .7995\), or about .80. Given our stated sample size, population standard deviation, and means, we would correctly reject the null about 80% of the time over repeated studies.

Activity 9.3 repeats this step-by-step process visually.

Activity 9.3: Explore statistical power

Explore statistical power. Use the accompanying steps for an explanation.

Start with a null population mean of 50, a population SD of 38.73, and samples of 15. The standard error is \(38.73 / \sqrt{15} \approx 10\). This curve shows sample means under the null, labeled in \(z\) units. For a two-tailed test with \(\alpha = .05\), the critical \(z\) values are about \(\pm 1.96\).

On the raw-score ruler, those same boundaries are \(50 - 1.96(10) = 30.40\) and \(50 + 1.96(10) = 69.60\). Sample means outside these boundaries lead us to reject the null.

Now the alternative sampling distribution is centered on 78, still with a standard error of 10. The raw boundaries stay fixed, but their corresponding \(z\)-scores change: the lower boundary is \((30.40 - 78) / 10 = -4.76\) standard errors from the alternative mean; the upper one is \((69.60 - 78) / 10 = -0.84\).

Now we look up those \(z\)-scores in the unit normal table to find the area between them. The gray area under the alternative curve lies between the critical boundaries. These samples fail to reject a false null: \(\beta \approx .200\).

The purple area outside the boundaries is power: \(1 - \beta \approx .800\). So this design detects this particular effect about 80 percent of the time over repeated studies. (Both rejection tails count, though the far-left contribution here is tiny.)

Calculate power for a new study

For a two-tailed \(z\) test with \(\alpha = .05\), suppose \(\mu_0 = 100\), \(\sigma = 20\), and \(n = 25\). What is the power to detect a true population mean of \(\mu_1 = 108\)? Assume normal sampling distributions with the same population SD under both hypotheses. Enter power as a proportion from 0 to 1. You can use the unit normal table.

Quiz 9.2

Factors Affecting Power

Now let’s explore how changing each of the parameters affects power. See if you can predict and explain each change before moving on to the next step.

Activity 9.4: Factors affecting statistical power

Null and alternative sampling distributions with power shaded, and sliders for alpha, sample size, population standard deviation and the difference between means.

What will happen to power if we change \(\alpha\)? Try to form a prediction. Then move the slider and watch what happens to power. Then change the test from two-tailed to one-tailed. Can you explain why these changes affect power?

Reducing \(\alpha\) makes for a more stringent test: the critical regions are smaller, so less of the alternative distribution causes us to reject the null. At the same \(\alpha\), a one-tailed test has greater power for effects in its prespecified direction, because its rejection region is larger in that tail. An effect in the opposite direction does not get that advantage. As in Chapter 8, the direction must be chosen before looking at the data.

How will power change if we increase or decrease the number of participants? Move the sample size slider and watch what happens.

Increasing sample size increases power for this fixed population difference because it reduces the standard error, meaning that the sampling distributions will overlap less. (Of course, a large sample and correspondingly good statistical power don’t repair biased sampling or poor measurement.)

How will power change if we increase or decrease the population standard deviation? Move the population variability slider and watch what happens.

Here we hold the raw difference between population means fixed. Increasing variability increases the standard error, which increases overlap and reduces power. Decreasing variability has the opposite effect. (Notice that with the raw difference fixed, a smaller \(\sigma\) also means a larger \(d\): the same difference is bigger relative to the variability.)

How will power change if we increase or decrease the difference between the means? Move the effect size slider and watch what happens.

Increasing the difference between the means (and so the effect size) increases power because the distributions will be farther apart and overlap less.

Putting these together: what really matters is how many standard errors apart the two means are, \((\mu_1 - \mu_0)/\sigma_M\). That equals \(d\sqrt{n}\), so power depends on the standardized effect size and the sample size (along with \(\alpha\) and the number of tails).

Practical Applications of Power

Do you feel like sometimes all this frequentist talk about long-run proportions of outcomes is a bit abstract? None of it would be very useful if we actually had to repeat our study over and over again to see how often we reject the null or whatever. But the value of thinking about things this way is that it helps us understand the implications of the one study we do conduct, even before we collect the data.

When planning research, we can ask how much power a proposed study would have to detect a particular effect. But we can also turn the question around: how many participants would we need to have a good chance of detecting an effect of that size? This is a useful question to ask before we collect our data.

Of course, we don’t usually know the true effect size in advance. (If we did, we’d hardly need to do the study.) Rather, we can speculate about a plausible effect on the basis of earlier research or theory. Or we could ask what is the smallest difference we would care about, and design the study to have a good chance of detecting that difference.

A common convention is to aim for a power of .80. It still means missing an effect of the specified size about one time in five. If the consequences of a miss are serious, we might want a higher target. Like \(\alpha = .05\), this isn’t a law of nature. It’s an attempt to balance the risk of a Type II error (failing to reject a false null hypothesis) with the practical constraints of conducting research.

For our two-tailed, one-sample \(z\) test, a useful approximation for the required sample size is:

\[ n \approx \frac{\left(z_{1-\alpha/2} + z_{1-\beta}\right)^2}{d^2} \tag{9.2}\]

Here, \(d\) is the hypothesized population effect, \((\mu_1 - \mu_0)/\sigma\), rather than an effect estimated from data we have yet to collect. The notation \(z_q\) means the standard-normal score with proportion \(q\) below it. So \(z_{1-\alpha/2}\) is our positive critical score, and \(z_{1-\beta}\) is the score with our target power below it. For 80% power, that second score is about \(0.84\).

For example, suppose we want to detect an effect of \(d = 0.5\). For a two-tailed test with \(\alpha = .05\) and a target power of .80:

\[ n \approx \frac{(1.96 + 0.84)^2}{0.5^2} = 31.36 \]

We round up, because we cannot have 0.36 of a person take our study, giving us 32 participants. For a one-tailed test in the predicted direction, use \(z_{1-\alpha}\) instead of \(z_{1-\alpha/2}\). The two-tailed formula is an approximation because it neglects rejection in the opposite tail, whose contribution is tiny in this example. Activity 9.5 calculates power using both tails and finds the smallest whole-number sample size that reaches the target.

To give a sense of this in context, recall the differences we pictured in Activity 9.1. Simmons (2014) also reported the following sample-size requirements for 80 percent power, based on the effect estimates from their data:

Table 9.5: Sample sizes per group required for 80% power to detect the effect sizes reported by Simmons (2014).

Outcome	Magnitude of \(d\)	\(n\) per group for 80% power
Height	1.83	6
Importance of social equality	0.69	33
Frequency of eating egg salad	0.58	47
Estimated risk of smoking-related death	0.33	149
Number of planets named correctly	0.05	6,176

If the true effects matched these estimates, detecting the height difference would need only around twelve people, six men and six women. But detecting the tiny difference in planet naming would need more than 12,000 people. (These are the author’s reported sample sizes for comparisons of two groups; \(n\) is per group, rather than for the one-sample \(z\) test we used above. Different populations, measures, and designs can produce different requirements.) Think about the sample sizes in the research you read, the subtlety of the effects, and whether the studies had sufficient power to detect them.

Notice that having a fairly small sample size does not necessarily mean a study will have low power. A large effect can be reliably detected with few participants. But trying to detect more subtle effects with small samples can lead to underpowered studies and misleading results. When a study is underpowered, the samples that do reach the significance threshold tend to be the ones whose means happened to land unusually far from the null. So significant results from underpowered studies tend to overestimate the true effect. That is one reason to be wary of surprisingly large effects from small samples.

Activity 9.5: Planning a study with power

Compare a proposed study's power with the sample size needed for 80 percent power. First vary the design to calculate power, then vary the hypothesized population difference while the required sample size is calculated.

Suppose we’re planning a study with 25 participants, a population SD of 10, and a two-tailed test with \(\alpha = .05\). If the true population difference is five points, power is about .71. Use the sliders to explore how different combinations of sample size, alpha, and hypothesized difference affect power. The value at the bottom is the power of the design you’ve chosen.

Now let’s turn the question around. Keep the population SD at 10 and the two-tailed \(\alpha\) at .05, but require .80 power. For a hypothesized difference of five points, we need 32 participants, as we calculated above. This time, sample size is the answer rather than something you choose with a slider.

Move the difference slider and watch the required sample size change. A smaller difference requires a larger sample to detect it with the same power. The curves narrow as the required sample grows, so that about 80 percent of the alternative distribution still falls in the rejection regions. The displayed \(n\) rounds up to a whole number of participants; the curves show the exact threshold before rounding.

We now have several different questions to keep straight. Effect size describes how big a difference is. \(\alpha\) describes the chance of rejecting a true null. \(\beta\) describes the chance of missing a specified real effect, and power, \(1-\beta\), describes the chance of detecting it. Those probabilities concern repeated use of a test under stated assumptions.

We can evaluate a design’s power for independently justified hypothetical effects either before or after data collection. But plugging the effect observed in our own sample back into a power calculation doesn’t tell us whether that result is trustworthy. This is sometimes called observed power, and it adds no new information to the test’s \(p\)-value (Hoenig & Heisey, 2001). After collecting data, the estimated effect and its uncertainty are more useful for interpretation. Our own study still gives us just one imperfect estimate of the population difference. How precise is that estimate? That’s the question we’ll turn to in the next chapter.

Learning Checks

1

One sample has 30 participants and another has 100. Both have a mean score of 55, and both are compared with a null mean of 50 using a known population SD of 10. How do their estimates of Cohen’s \(d\) compare?

Response options

The larger sample has a larger magnitude of \(d\).

Both samples give the same \(d\).

The larger sample has a smaller magnitude of \(d\).

2

A researcher uses \(\alpha = .05\) to test whether a population mean differs from a specified null mean. If the null is true and the test’s assumptions hold, what does that five percent describe?

Response options

The probability that the null hypothesis is true.

The probability that a particular significant result is a mistake.

The proportion of individual scores outside the critical boundaries.

The long-run proportion of samples leading to rejection of the true null.

3

A study design has .80 power to detect a true population difference of five points. If we repeated this design with new, independent samples from a population with that difference, what would we expect?

Response options

About 80% of studies would correctly reject the null.

About 80% of studies would miss the effect.

About 80% of studies would estimate the difference as exactly five points.

Every study would reject the null, since the effect is real.

4

Researchers planned a study with .80 power to detect \(d = .5\). Their result is nonsignificant. What does this establish about the population effect?

Response options

The population effect is exactly zero.

There is an 80% probability that the population effect is zero.

The study did not provide sufficient evidence to reject the null.

The population effect must be smaller than \(d = .5\).

5

For the same nonzero population difference, sample size, and population SD, a researcher lowers \(\alpha\) from .05 to .01. The test remains two-tailed. How does power change?

Response options

Power increases.

Power decreases.

Power stays the same.

6

Researchers plan a two-tailed, one-sample \(z\) test. Keeping \(\alpha\) and target power fixed, they decide to plan for \(d = .25\) rather than \(d = .5\). Using the chapter’s approximate sample-size equation, how does the required sample size change before rounding?

Response options

It stays the same.

It doubles.

It halves.

It quadruples.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.

Funder, D. C., & Ozer, D. J. (2019). Evaluating effect size in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2(2), 156–168. https://doi.org/10.1177/2515245919847202

Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1), 19–24. https://doi.org/10.1198/000313001300339897

Simmons, J. P. (2014, April 4). [18] MTurk vs. The lab: Either way we need big samples. Data Colada. https://datacolada.org/18

---

1. Okay, listen, this is calibrated to offer an illustration of the sampling distribution, but it is not a demonstration of independent random sampling. The bees interact, and each bee’s position is correlated from one frame to the next; a bee exploring to the left of the hive won’t suddenly teleport off to the right. But the approximation is good enough for our purposes here. And come on, they’re delightful little bees!

2. The open-ended responses used here were trimmed below the 5th and above the 95th percentiles, following the author’s analysis. The estimates describe this sample data; they should not be taken as fixed population characteristics.

3. That was a joke. The units are arbitrary, but the calculations do reproduce the characteristics of the little bee-simulation engine.