If you have ever run a statistical test and reported only a p-value, you told half the story. Researchers, data analysts, and scientists across fields keep making the same mistake: they declare a result “statistically significant” without telling anyone how big the effect actually is. That missing piece, the effect size, is what separates a finding that matters from one that is mathematically real but practically useless.
In this guide, I will explain why you should report effect sizes alongside p-values, walk through the calculation step by step, and show you the common pitfalls that catch even experienced analysts. Whether you work in academic research, clinical trials, or modern A/B testing, this distinction changes how you interpret results.
The core idea is simple. A p-value answers one question: “Is there likely something happening here?” An effect size answers a different, often more important question: “How much of something is happening?” Without both answers together, your readers, stakeholders, and future researchers cannot judge whether your finding deserves attention. As the statistician Jacob Cohen put it, “The primary product of a research inquiry is one or more measures of effect size, not P values.”
Table of Contents
What Is Effect Size?
Effect size is a quantitative measure of the magnitude of a phenomenon. In plain terms, it tells you how large or meaningful a difference is between groups, or how strong a relationship is between variables. Unlike a p-value, which only signals whether an effect exists, an effect size tells you how big that effect actually is.
Think of it this way. If a new teaching method raises test scores, the p-value tells you the improvement probably did not happen by chance. The effect size tells you whether that improvement is large enough to justify changing curriculum, or so small that nobody would notice in practice. Both pieces matter, but the effect size speaks directly to real-world impact.
There are several common effect size indices, and the right choice depends on your study design:
- Cohen’s d: The standardized difference between two group means, divided by the pooled standard deviation. Best for comparing two groups.
- Pearson’s r: A correlation coefficient ranging from -1 to 1 that measures the strength and direction of a relationship between two continuous variables.
- Glass’s Delta: Similar to Cohen’s d but uses only the control group’s standard deviation. Useful when the treatment group variance is affected by the intervention itself.
- Eta squared (η²): The proportion of variance in the dependent variable explained by an independent variable. Common in ANOVA designs.
- Odds ratio: Used in logistic regression and clinical studies to express how odds change between conditions.
Each of these indices standardizes results so you can compare findings across studies using different scales. A Cohen’s d of 0.5 means the same thing whether your measurement is in points, pounds, or milliseconds. This standardization is what makes effect sizes indispensable for meta-analysis, where researchers combine results from dozens of studies into a single estimate.
The distinction between absolute effect size and standardized effect size also matters. An absolute effect size reports the raw difference, like “patients improved by 3 points on a 50-point scale.” A standardized effect size converts that to a common metric. Both are useful, and good reporting includes both so readers can judge clinical or practical importance alongside comparability.
Why You Should Report Effect Sizes Alongside P-Values
The p-value has a fundamental limitation that most researchers underestimate: it depends heavily on sample size. With a small sample, even a large, important effect might not reach significance. With a massive sample, even a trivially small effect will cross the magic threshold of p less than 0.05. This means a significant p-value tells you nothing reliable about importance.
Consider this scenario that comes up constantly on statistics forums. A researcher runs a study with 2,000 participants and finds a difference between two groups with p equal to 0.01. They declare victory. But the actual difference is a 2 percent change on a metric where 10 percent would be the minimum worth acting on. The result is statistically significant and practically meaningless at the same time. Without an effect size, the reader has no way to detect this trap.
Reddit users on r/AskStatistics and r/statistics describe this frustration constantly. One common complaint: reviewing “tables full of p-values and significance stars” with no indication of magnitude. Practitioners note that statistically significant results frequently show no real impact after implementation. A graduate student on stats.stackexchange summed it up well: “In practice nobody should directly care about a p-value. They should care about the effect size being claimed, often not even reported.”
Here are the specific reasons reporting effect sizes alongside p-values is essential:
P-values conflate effect magnitude with sample size. A p-value of 0.04 from 50 participants and a p-value of 0.04 from 50,000 participants can represent wildly different effects. The effect size strips away the sample size influence, giving you a cleaner read on magnitude.
Effect sizes enable comparison across studies. If one study reports a Cohen’s d of 0.3 and another reports 0.6, you can immediately see which finding is larger. Comparing p-values across studies with different sample sizes tells you nothing meaningful.
Non-significant results still have a story. A common question on forums is whether you should report effect sizes when the p-value is not significant. The answer is yes. A non-significant result with a large effect size and small sample suggests your study was underpowered, not that the effect does not exist. A non-significant result with a tiny effect size and a huge sample tells you the effect is genuinely negligible. These are very different conclusions, and you can only tell them apart with the effect size.
Decision-makers need magnitude, not just direction. In business, medicine, and policy, the question is rarely “does this work at all?” It is “does this work enough to justify the cost, risk, or effort?” Only an effect size can answer that.
The American Statistical Association released a statement in 2016 reinforcing this point, warning against using p-values as the sole measure of evidence. Their guidance explicitly called for effect sizes, confidence intervals, and practical context to accompany any significance test. The field has known about this problem for decades, but old reporting habits die hard.
How to Calculate Effect Size
The most widely used effect size for comparing two groups is Cohen’s d. The formula is straightforward: subtract the mean of group 2 from the mean of group 1, then divide by the pooled standard deviation. In notation, d equals the mean difference divided by the pooled standard deviation.
To calculate the pooled standard deviation, you combine the variance from both groups weighted by their sample sizes. The formula is the square root of the sum of each group’s sum of squares divided by the total degrees of freedom. Most statistical software, including R, Python’s scipy, SPSS, and JASP, computes this automatically. But understanding the math helps you spot errors and explain results.
Here is a worked example. Suppose you test a new tutoring program against a control group on exam scores:
- Treatment group mean: 78, standard deviation: 10, n = 40
- Control group mean: 72, standard deviation: 11, n = 40
- Mean difference: 78 minus 72 equals 6 points
- Pooled standard deviation: approximately 10.5
- Cohen’s d: 6 divided by 10.5 equals approximately 0.57
A Cohen’s d of 0.57 falls in the medium range. That tells you the tutoring program produced a meaningful improvement, not just a detectable one. The p-value confirms the difference is unlikely to be chance. The effect size confirms it is worth caring about.
For correlation studies, you use Pearson’s r instead. If you measure study time and exam scores across 100 students and find r equals 0.45, that is a moderate positive correlation. Squaring r gives the proportion of variance explained, in this case about 20 percent. That means study time accounts for roughly a fifth of the variation in exam scores, which is substantial for a single predictor.
For ANOVA designs with three or more groups, eta squared or its partial variant is the standard choice. It expresses how much of the total variance your independent variable explains. An eta squared of 0.01 is small, 0.06 is medium, and 0.14 is large by conventional benchmarks.
Always report confidence intervals for your effect sizes, not just point estimates. A Cohen’s d of 0.5 with a 95 percent confidence interval of 0.08 to 0.92 tells a very different story than the same d with an interval of 0.42 to 0.58. Wide intervals signal uncertainty that the point estimate alone hides.
Effect Size Interpretation
Interpreting effect sizes requires context, but Cohen’s benchmarks give you a starting framework. These guidelines, published by Jacob Cohen in 1988, remain the most cited reference for judging magnitude:
- Small effect: Cohen’s d = 0.2, Pearson’s r = 0.1, eta squared = 0.01. The effect is real but likely too small to notice in individual cases.
- Medium effect: Cohen’s d = 0.5, Pearson’s r = 0.3, eta squared = 0.06. The effect is visible enough to matter in many practical settings.
- Large effect: Cohen’s d = 0.8, Pearson’s r = 0.5, eta squared = 0.14. The effect is obvious and likely meaningful at the individual level.
These benchmarks come from surveying the published literature in the social sciences. Cohen himself described them as rough rules of thumb, not laws of nature. A “small” effect in particle physics might be a massive discovery. A “large” effect in a well-established marketing channel might be unremarkable. Always interpret effect sizes within the norms of your specific field.
A helpful way to visualize Cohen’s d is through distribution overlap. A d of 0.2 means the two distributions overlap by about 92 percent. A d of 0.5 reduces overlap to about 67 percent. A d of 0.8 drops overlap to about 53 percent. At d equals 1.2, less than 40 percent of the distributions overlap, meaning most individuals in one group score above the average of the other.
For a concrete and famous example, consider the Physicians’ Health Study on aspirin and heart attacks. The study found that daily aspirin reduced heart attack risk with a p-value so small it was essentially zero. But the effect size, measured as a relative risk reduction, was around 44 percent in relative terms. That sounds large until you look at absolute risk: the absolute difference was less than 1 percentage point. For every 1,000 people taking aspirin, roughly 5 fewer would have a heart attack. The p-value said the effect was real. The effect size, viewed in absolute terms, fueled a decade of debate about whether mass aspirin use was worth the bleeding risk.
This example shows why reporting both relative and absolute effect sizes matters. Relative measures can exaggerate importance when baseline risk is low. Absolute measures ground the finding in terms decision-makers can act on.
Statistical Power and Sample Size
Statistical power is the probability that your study will detect an effect if one truly exists. It is the complement of Type II error, the mistake of missing a real effect. Power depends on three things: the effect size you are trying to detect, your sample size, and the significance level you set.
This is where effect size and sample size connect directly. To calculate how many participants you need, you must first specify the minimum effect size you consider worth detecting. That minimum detectable effect is a practical decision, not a statistical one. It depends on what size of impact would change your decision or behavior.
If you set your minimum interesting effect at Cohen’s d of 0.2, you need a much larger sample than if you set it at 0.8. For 80 percent power at alpha 0.05, detecting a small effect (d = 0.2) requires roughly 788 participants total. Detecting a medium effect (d = 0.5) requires about 128 participants. Detecting a large effect (d = 0.8) requires only about 52 participants. These numbers come from standard power analysis formulas available in tools like G*Power, R’s pwr package, and Python’s statsmodels.
Underpowered studies are a silent epidemic in research. A study with 30 participants trying to detect a small effect has power below 30 percent. That means a 70 percent chance of missing a real effect, and then concluding the effect does not exist. Reporting the effect size alongside the non-significant p-value at least lets readers see that the study could not have detected the effect even if it were there.
Type I versus Type II error tradeoffs become clearer with effect sizes in view. Type I error is finding an effect that does not exist (a false positive). Type II error is missing an effect that does exist (a false negative). Convention sets Type I error at 5 percent but ignores Type II error entirely in many studies. The result is a literature biased toward false negatives for small but real effects, especially in underpowered fields like neuroscience and early-stage clinical research.
When planning a study, work backwards from the effect size you care about. Decide the smallest meaningful effect, choose your desired power (80 percent is standard, 90 percent is better), set your alpha, and compute the required sample size. This approach prevents the common trap of running a study with whatever participants you can find and then trying to interpret whatever p-value comes out.
Practical Significance vs Statistical Significance
Statistical significance asks: “Could this result be due to chance?” Practical significance asks: “Does this result matter?” These are different questions, and a yes to the first does not guarantee a yes to the second.
In A/B testing, this distinction plays out daily. A tech company runs an experiment with 500,000 users and finds that a new checkout design increases conversion with p equal to 0.001. The effect size is a 0.1 percentage point lift. Statistically significant? Yes. Practically significant? Only if that 0.1 percent translates to revenue that exceeds the cost of implementing and maintaining the new design. Engineers and product managers who understand effect sizes make better deployment decisions because they can have that conversation directly.
Here is a practical framework for reporting that bridges both types of significance. Always state the effect size with its confidence interval. State the p-value alongside it for completeness. Then add one sentence on practical interpretation: what does this magnitude mean for the people or systems affected? This three-part reporting structure gives readers everything they need to evaluate your finding.
For example: “The treatment group scored 6 points higher than the control group (Cohen’s d = 0.57, 95 percent CI 0.12 to 1.02, p = 0.013). This medium effect size suggests the intervention produces a noticeable improvement that most educators would consider meaningful.” Compare that to the common but inadequate alternative: “The difference was significant (p less than 0.05).”
When results are non-significant, the effect size still tells you something important. A large effect size with a wide confidence interval that crosses zero suggests the study was underpowered, and a larger study might find significance. A tiny effect size with a narrow confidence interval near zero suggests the effect is genuinely small, and more data would not change that. These different situations call for different next steps, and only the effect size distinguishes them.
Meta-analysis depends entirely on effect size reporting. When researchers combine studies, they pull effect sizes, not p-values. If your study reports only p-values, it cannot contribute to a meta-analysis. Given that meta-analyses often drive evidence-based policy and clinical guidelines, this is no small omission. Every study that fails to report effect sizes weakens the entire evidence base for its field.
FAQs
What is effect size and why is it important?
Effect size measures the magnitude of a difference or relationship in a study. It is important because p-values only tell you whether an effect exists, while effect sizes tell you how large that effect is. Without effect sizes, readers cannot judge whether a finding has real-world importance.
Why is the P value not enough?
The p-value depends on sample size, so a large study can produce a significant p-value for a trivially small effect, while a small study can miss a large real effect. The p-value answers whether an effect likely exists, not whether it matters, which is why effect sizes must accompany it.
How do you calculate effect size?
For two groups, Cohen’s d is calculated by dividing the difference between group means by the pooled standard deviation. For correlations, Pearson’s r measures the strength of the relationship. Most statistical software computes these automatically, but understanding the formula helps with interpretation.
What is statistical power and why do I need it?
Statistical power is the probability of detecting a real effect, calculated as 1 minus the Type II error rate. You need adequate power to avoid missing genuine effects. Power depends on effect size, sample size, and significance level, which is why specifying your minimum detectable effect before running a study is essential.
How do you calculate sample size using effect size?
You decide the smallest effect size worth detecting, choose your desired power (typically 80 percent or higher), set your alpha level, then use a power analysis tool like G*Power to compute the required sample size. Larger effects require fewer participants, while smaller effects demand much larger samples.
Conclusion
Reporting effect sizes alongside p-values is not optional good practice. It is the difference between communicating clearly and leaving your audience to guess. The p-value tells readers that something happened. The effect size tells them whether it matters.
When you understand why you should report effect sizes alongside p-values, every part of your research workflow improves. You design better studies by planning around the effects you care about. You interpret results more honestly by separating detection from importance. You contribute to meta-analyses that the next generation of researchers will rely on. And you avoid the most common trap in quantitative work: declaring victory over a finding that no one should act on.
Start with a simple change in your reporting habit. Every time you write a p-value, write the effect size and its confidence interval right next to it. Add one sentence on what that magnitude means in practice. That small shift will make your work clearer, more useful, and more credible to everyone who reads it.