Power Analysis and Sample Size Determination in Economics Research

Many empirical economics studies fail before data collection begins because the sample is too small to detect the effect the study claims to test. Power analysis is the research-design calculation that links the effect size, significance level, statistical power, and required sample size before evidence is collected.

The method matters because non-significant results are often misread. A study that fails to reject the null may have found no effect, or it may have been too underpowered to detect an effect of practical importance. Power analysis separates those two cases before the research design is locked in.

In economics, this issue appears in randomized trials, labour-market studies, education interventions, survey experiments, field experiments, and policy evaluations. A sample that is too small can turn a real economic effect into statistical silence. A sample that is larger than needed can waste research funds, respondent time, and administrative capacity.

The Sample Size Question Economists Avoid

Power analysis begins with a simple research-design question: what sample size is needed to detect an economically meaningful effect with an acceptable probability? The answer depends on the size of the effect, the noise in the outcome variable, the significance level, the desired power, and the design structure.

J-PAL describes power calculations as either determining the sample size needed to detect a minimum detectable effect or determining the effect size that can be detected for a given sample size and other parameters. J-PAL power calculations. That definition is useful because it treats power as a design constraint, not a post-hoc diagnostic.

The central object is the minimum detectable effect. This is the smallest effect size that the study has a good chance of detecting under the planned design. A study may care about a wage increase of 2 percent, a school-attendance gain of 5 percentage points, or a tax-compliance increase of 10 percent. The sample size should be built around that meaningful effect, not around convenience.

Power analysis, therefore, belongs near the beginning of a research project. It should be considered when the hypothesis is formed, before fieldwork, before administrative data extraction, and before the pre-analysis plan is frozen. The article on hypothesis testing in economics explains the inferential logic behind null hypotheses, p-values, Type I error, Type II error, and statistical power. This article focuses on the planning side: how large the study needs to be.

From Effect Size to Required N

Effect size is the scale of the difference the study wants to detect. In a two-group comparison, it is often expressed in standardized units, usually Cohen’s \(d\): the difference between treatment and control means divided by the outcome standard deviation.

Here, \(\mu_T\) is the treatment-group mean, \(\mu_C\) is the control-group mean, and \(\sigma\) is the standard deviation of the outcome. If \(d = 0.20\), the treatment effect is one-fifth of a standard deviation. If \(d = 0.80\), the effect is four times larger in standardized terms.

Cohen’s conventional benchmarks of 0.20, 0.50, and 0.80 are widely used as small, medium, and large standardized effects, although those labels should not replace economic judgment. A 0.20 standard deviation gain in test scores may matter in education policy. A 0.20 standard deviation change in a noisy survey outcome may not justify the same interpretation. Brydges on Cohen effect-size conventions

The required sample size rises sharply when the detectable effect becomes smaller. For a two-sample comparison with equal treatment and control groups, a common approximation for sample size per arm is:

In this expression, \(n\) is the required sample size per arm, \(\alpha\) is the significance level, \(\beta\) is the Type II error probability, \(1-\beta\) is statistical power, \(d\) is the standardized effect size, and \(z\) values are critical values from the standard normal distribution. The inverse square term \(1/d^2\) is the key insight. Cutting the target effect size in half roughly quadruples the required sample size.

Alpha Beta and Statistical Power

Power analysis connects two error probabilities. The significance level \(\alpha\) is the probability of rejecting a true null hypothesis. The Type II error probability \(\beta\) is the probability of failing to reject a false null hypothesis. Statistical power is the complement of \(\beta\).

A conventional design target is 80 percent power, meaning the study has an 80 percent probability of detecting the specified effect if that effect is truly present under the assumed model. A higher power, such as 90 or 95 percent, reduces the chance of missing a real effect but increases the required sample size.

The American Statistical Association’s statement on p-values warns against treating statistical significance as a mechanical threshold for scientific truth. It states that a p-value does not measure the size of an effect or the importance of a result. ASA statement on p-values. Power analysis fits that warning because it shifts attention from significance alone to design capacity.

An underpowered study can produce two kinds of damage. It can miss a real effect, creating a false sense that the policy or mechanism has no impact. It can also produce unstable, significant estimates when chance variation pushes the estimate far from the truth. Both failures are common when the sample size is decided after budgets, field logistics, or available data, rather than before the research question is settled.

Sample Size in Economic Studies

The same power formula produces very different sample-size requirements depending on the effect size and desired power. The table below uses a stylized two-arm comparison, equal treatment and control groups, a two-sided test, and \(\alpha = 0.05\). It reports the approximate sample size per arm, rounded upward.

Table 1. Power Calculation Reference: Required Sample Size by Effect Size

Effect Size \(d\)	Power 0.80	Power 0.90	Power 0.95	Research Design Note
0.10 very small	1,570 per arm	2,102 per arm	2,600 per arm	Often feasible only with large administrative data or national surveys.
0.20 small	393 per arm	526 per arm	650 per arm	Common target in field experiments and education evaluations.
0.30 modest	175 per arm	234 per arm	289 per arm	Feasible in many programme evaluations if attrition is controlled.
0.50 medium	63 per arm	85 per arm	104 per arm	Common in tightly controlled lab or survey-experiment settings.
0.80 large	25 per arm	33 per arm	41 per arm	Rare for many economic-policy outcomes, but possible for strong behavioural treatments.

The table shows the core power-analysis trade-off. Detecting a large effect can require only a modest sample. Detecting a small effect requires a much larger study. This is why the effect size must be chosen using economic reasoning. A design powered only to detect large effects may miss the smaller effects that matter for policy.

The table is also a simplified benchmark. Clustered assignment, repeated measures, non-compliance, attrition, unequal treatment shares, multiple outcomes, and heterogeneous treatment effects can all raise the required sample. In field settings, the design must often inflate the baseline calculation before data collection begins.

The Power Curve in Practice

A power curve shows how the probability of detecting an effect rises as sample size increases. The curve is steeper for larger effects and flatter for smaller effects. This visual is useful because it shows why small studies can be adequate for large treatment effects but weak for modest effects.

The curve shows why planning matters. At small sample sizes, only large effects are likely to be detected. Medium effects require more observations. Small effects can remain below conventional power thresholds even when the study has several hundred observations per arm.

For economics, this lesson is especially important because policy effects are often modest. A job-training programme, cash-transfer intervention, financial-literacy module, or information treatment may produce effects that matter economically but remain difficult to detect statistically without a large enough design.

Pre-Registration Makes Power Credible

Power analysis is strongest when it is written before results are known. A pre-analysis plan can document the primary outcome, target effect size, power assumptions, treatment arms, clustering level, and sample-size decision before data collection or analysis. This reduces the risk that researchers revise the power justification after seeing the data.

The Open Science Framework defines preregistration as posting a time-stamped, read-only study plan before data collection or analysis. OSF registrations and preregistrations The American Economic Association maintains the AEA RCT Registry for randomized controlled trials in the social sciences and allows investigators to share protocols, survey instruments, and related study information. AEA RCT Registry policy

The MASEconomics article on pre-registration economics develops this workflow in detail. Power analysis belongs inside that workflow because sample-size decisions are not neutral. They define what the study can detect, what it cannot detect, and how any non-significant result should be interpreted.

Pre-registration does not make a weak power calculation strong. It makes the assumptions visible. A registered plan can still use unrealistic effect sizes, underestimate attrition, or ignore clustering. The benefit is that these assumptions become inspectable rather than hidden.

Clustered Designs Need Larger Samples

Many economic studies do not randomly assign individuals one by one. They assign villages, schools, firms, classrooms, districts, or markets. This creates clustering. Outcomes within the same cluster tend to be correlated because individuals share teachers, managers, local institutions, prices, or shocks.

Clustering reduces the amount of independent information in the sample. One hundred individuals from one village do not provide the same information as one hundred individuals drawn independently across many villages. The design effect summarizes this inflation in the required sample size.

Here, \(DE\) is the design effect, \(m\) is the average cluster size, and \(\rho\) is the intra-cluster correlation. The effective sample size falls as \(\rho\) rises. Even a small intra-cluster correlation can matter when clusters are large.

J-PAL’s power-calculation materials emphasize that clustered and more complex designs require attention to design parameters beyond simple individual-level sample size. J-PAL mechanics of power calculations In applied economics, this issue is common in education, health, agriculture, and development studies, where implementation occurs at the school, clinic, village, or district level.

Clustered power analysis, therefore, shifts attention from the number of individuals to the number of independent units. Adding more individuals to a few clusters may help less than adding more clusters. A field experiment with 20 villages may remain underpowered even if each village has many surveyed households.

Attrition Changes the Final Design

Attrition occurs when observations planned for analysis are lost before outcome measurement. Households move, firms exit the sample, students leave school, survey respondents refuse follow-up, and administrative records fail to match across datasets. If attrition is large, the realized sample may be smaller than the planned sample.

Power analysis should account for expected attrition before the study begins. If the target final sample is 500 observations per arm and expected attrition is 20 percent, the initial recruitment target must be larger than 500 per arm. Otherwise, the final study will be underpowered even if the original calculation was correct.

Here, \(a\) is the expected attrition rate. If the desired final sample is 500 and attrition is expected to be 20 percent, the initial sample should be \(500/(1-0.20)=625\). This calculation is simple, but it is often ignored when field constraints dominate planning.

Attrition also threatens validity when it is non-random. If treated households are more likely to remain in the study than control households, the final comparison may no longer reflect the original design. Power and validity, therefore, interact. A larger initial sample protects statistical precision, but careful tracking and attrition analysis protect interpretation.

Multiple Outcomes Reduce Power

Many economic studies test several outcomes: income, employment, consumption, savings, education, health, expectations, or firm investment. Each additional test raises the risk of false discoveries if results are interpreted one by one without adjustment. This is where power analysis connects to multiple testing and p-hacking.

A study that tests many outcomes may need to define a primary outcome or outcome index before analysis. Otherwise, significant results may appear simply because many tests were tried. The planned MASEconomics article on P-Hacking and the Garden of Forking Paths should later be linked from this point because it will explain how hidden flexibility can create false-positive evidence.

Power calculations should therefore focus on the primary outcome. Secondary outcomes may still be reported, but they should not drive the main design unless the study is powered for them. If the primary outcome is rare, noisy, or difficult to measure, the required sample may be much larger than the sample needed for secondary outcomes.

In pre-analysis planning, the researcher should state the primary hypothesis, primary outcome, unit of analysis, and minimum detectable effect. This makes it easier to distinguish confirmatory analysis from exploratory findings.

When Power Calculations Mislead

Power calculations can be misleading when the inputs are chosen poorly. The most common failure is choosing an effect size that is too large because it makes the required sample size affordable. If a study is powered only to detect a dramatic effect, it may be unable to detect the smaller effect that would still matter economically.

Another failure is ignoring variance. Power depends not only on effect size and sample size but also on outcome variability. Noisy outcomes require larger samples. Better measurement can sometimes improve power more cheaply than adding respondents. Administrative data, repeated observations, baseline controls, and outcome indexes can improve precision when used appropriately.

Power calculations also become fragile when based on unrealistic assumptions about compliance, attrition, clustering, or treatment uptake. A programme may be assigned to a treatment group, but not everyone assigned receives it. If take-up is low, the intention-to-treat effect may be smaller than the treatment-on-treated effect. The study must be powered for the estimand it plans to report.

Finally, power calculations are not a substitute for theory. A statistically detectable effect is not automatically meaningful. A meaningful effect is not automatically feasible to detect. Good design begins by asking which effect would change the interpretation of the economic question.

Power Analysis Across Economic Fields

In development economics, power analysis helps decide how many households, villages, schools, or firms are needed for a randomized evaluation. In labour economics, it helps assess whether a study can detect wage or employment changes of a meaningful size. In education economics, it helps determine whether test-score effects can be distinguished from sampling noise. In behavioural economics, it helps decide whether lab or survey experiments are large enough to test decision mechanisms.

In macroeconomics and finance, power often appears differently because researchers cannot easily choose the number of recessions, policy shocks, crises, or central-bank decisions available for study. The same principle remains: if the data are too limited or noisy to detect the claimed effect, inference should be cautious. Small samples in time-series settings can make null results especially hard to interpret.

Econometric tools can help, but cannot remove the design problem. Multiple regression models can improve precision when controls explain outcome variation. Bootstrap methods in econometrics can support inference when analytic standard errors are difficult. These tools strengthen analysis only when the underlying research design has enough information to answer the question.

The practical rule is clear: power analysis should be discussed before the study treats non-significance as evidence. Without that step, a null result may mean too little sample, not no economic effect.

MASEconomics Explains

Conclusion

Power analysis is the design calculation that determines whether an economics study has enough information to detect the effect it claims to test. It links \(\alpha\), \(\beta\), effect size, variance, sample size, and research design before the data are interpreted.

The method is most useful when applied before data collection or analysis. It prevents weak null results from being overread, forces the researcher to define an economically meaningful effect, and connects sample size decisions to the credibility of the final evidence.

Frequently Asked Questions

Thanks for reading! If you found this helpful, share it with friends and spread the knowledge. Happy learning with MASEconomics