If you have ever stared at a Mantel-Haenszel DIF output and wondered whether an item is biased or fair, you are in the right place. Mantel-Haenszel DIF statistics are the most widely used tool in psychometric testing for flagging test questions that perform differently across groups of examinees who share the same underlying ability. Learning how to interpret Mantel-Haenszel DIF statistics correctly helps test developers, psychometricians, and assessment specialists protect fairness and validity on standardized exams, licensure tests, and psychological assessments.
The Mantel-Haenszel procedure, introduced for DIF detection by Holland and Thayer in 1986, compares a reference group (typically the majority group) with a focal group (typically the group of interest) across score levels. It produces two key outputs: a chi-square statistic that tests whether DIF is statistically significant, and a common odds ratio that quantifies the magnitude and direction of that DIF. Interpretation requires looking at both numbers together, plus applying the ETS classification system that categorizes items into A, B, or C based on combined significance and magnitude criteria.
In this guide, I walk through each step of interpretation with concrete examples, threshold values, and common pitfalls. Whether you are running DIF analysis in R using the difR package, in Winsteps, or in Stata, the interpretation logic stays the same. By the end, you will know exactly what to do when you see an MH chi-square of 6.42 with an odds ratio of 1.65.
Table of Contents
What Is Differential Item Functioning (DIF)?
Differential item functioning, or DIF, occurs when examinees from different groups have a different probability of answering a test item correctly after being matched on the underlying ability the test measures. The key phrase is “after being matched on ability.” DIF is not about raw group differences in performance. It is about whether an item behaves differently for groups that should, in theory, perform the same.
Imagine two test-takers, one from a reference group and one from a focal group, both scoring 25 out of 40 on a math test. If the focal group member is systematically less likely to answer a particular item correctly than the reference group member at that same total score, the item may be showing DIF. That gap, repeated across thousands of matched pairs, is what Mantel-Haenszel DIF statistics detect.
DIF does not automatically prove bias. An item flagged for DIF requires content review by subject-matter experts to determine whether the performance difference stems from a construct-irrelevant factor (such as cultural wording) or a legitimate secondary skill. The statistics flag the item; human judgment explains it.
How to Interpret Mantel-Haenszel DIF Statistics: Core Concept
The Mantel-Haenszel approach is a stratified analysis method. It takes the full sample, slices it into strata based on total test score (the matching variable), and within each stratum compares the odds of a correct response between reference and focal groups. Those stratum-level comparisons are then pooled into a single common odds ratio and an associated chi-square test.
Interpreting the output requires understanding three linked pieces. First, the MH chi-square tells you whether the DIF is statistically significant. Second, the MH common odds ratio (often labeled alpha-MH) tells you how large that DIF is and which group it disadvantages. Third, the ETS classification combines both into a practical A, B, or C label that drives real decisions about whether to revise or retire the item.
This stratified design is why Mantel-Haenszel is sometimes called the Cochran-Mantel-Haenszel procedure in DIF contexts. The method controls for ability level by conditioning on the total score, effectively asking the fairness question within homogeneous ability bands rather than across the full sample.
The Mantel-Haenszel Formula Explained
The common Mantel-Haenszel odds ratio is calculated by summing across K strata (score levels) using the formula: alpha-MH equals the sum of (A-sub-k times D-sub-k divided by N-sub-k) over the sum of (B-sub-k times C-sub-k divided by N-sub-k), where A and D are the concordant cells and B and C are the discordant cells of the two-by-two table at each stratum.
For a dichotomously scored item, each stratum produces a two-by-two table. The rows are group membership (reference versus focal), and the columns are item response (correct versus incorrect). The Mantel-Haenszel statistic pools information across all those tables into one common odds ratio that estimates the consistent direction and magnitude of group difference.
A value of alpha-MH equal to 1.0 means no DIF. Values above 1.0 indicate the item favors the reference group. Values below 1.0 indicate the item favors the focal group. Because the odds ratio is not symmetric around 1.0, researchers often transform it using the natural log so that positive and negative DIF magnitudes are directly comparable.
Interpreting the MH Chi-Square Statistic
The Mantel-Haenszel chi-square tests the null hypothesis that the common odds ratio equals 1.0, meaning no DIF exists. It follows a chi-square distribution with one degree of freedom. The standard significance threshold is a p-value less than 0.05, which corresponds to a chi-square value greater than approximately 3.84.
A significant chi-square alone is not enough to flag an item for action. With large samples, even trivially small DIF can reach statistical significance. This is why the chi-square must always be read alongside the odds ratio and the ETS magnitude classification. Significance answers “is there something?” while the odds ratio answers “how much?”
Some software packages report the MH statistic as the Mantel-Haenszel mean score test when items are polytomously scored. In that case the test evaluates whether the mean item score differs between groups within each stratum, but the interpretation logic remains the same: significance tells you DIF is present, and an effect size tells you whether it matters.
Understanding the Mantel-Haenszel Odds Ratio (Alpha-MH)
The alpha-MH is the effect size that matters most for practical decisions. Educational Testing Service transformed this odds ratio into a more interpretable metric called MH D-DIF, computed as negative 2.35 times the natural log of alpha-MH. This transformation places the index on a scale where higher values indicate more DIF against the focal group.
An alpha-MH of 1.0 means no DIF. An alpha-MH of 1.5 means the reference group has 1.5 times the odds of answering correctly compared to a matched focal group member. An alpha-MH of 0.67 means the focal group has the advantage. The further the value moves from 1.0 in either direction, the larger the DIF.
For reporting, many testing programs convert alpha-MH into the MH D-DIF delta scale because it is symmetric and easier to communicate to non-technical stakeholders. An MH D-DIF of 0 means no DIF, positive values indicate DIF against the focal group, and negative values indicate DIF against the reference group.
ETS Classification System: Categories A, B, and C
Educational Testing Service developed a classification scheme that combines statistical significance with effect size magnitude. This system is the industry standard for DIF interpretation and is used by testing programs worldwide. It sorts every analyzed item into one of three categories: A, B, or C.
Category A items show negligible DIF. They either fail to reach statistical significance or have an MH D-DIF absolute value below 1.0. These items are considered fair and require no action. The vast majority of well-developed items should land in Category A.
Category B items show moderate DIF. They reach statistical significance with an MH D-DIF absolute value between 1.0 and 1.5, or they are not statistically significant but have an MH D-DIF at or above 1.0. Category B items are typically flagged for content review and may be revised but are not automatically removed.
Category C items show large DIF. They are both statistically significant and have an MH D-DIF absolute value of 1.5 or greater. Category C items are considered serious and usually trigger content review with a strong presumption toward revision or removal. Most testing programs treat any Category C item as a red flag requiring immediate attention.
The dual requirement of significance and magnitude is what makes the ETS system so practical. It prevents the over-flagging that pure significance testing would cause with large samples, and it prevents the under-flagging that pure effect size thresholds would cause with small samples.
Uniform vs Non-Uniform DIF
Mantel-Haenszel DIF statistics are designed to detect uniform DIF, which means the advantage or disadvantage is consistent across all ability levels. If the focal group is disadvantaged at low, medium, and high ability levels to roughly the same degree, the MH procedure captures that pattern well.
Non-uniform DIF is different. It occurs when the direction or magnitude of group difference changes across ability levels. For example, an item might disadvantage the focal group at low ability levels but advantage them at high ability levels. The standard Mantel-Haenszel procedure can miss non-uniform DIF because the common odds ratio assumes a single consistent direction.
To detect non-uniform DIF, the Mantel-Haenszel framework can be extended using an additional chi-square test that examines the interaction between group and ability level. Some software packages, including difR in R, provide both the uniform DIF test and the non-uniform DIF test as part of the same analysis. If the non-uniform test is significant but the uniform test is not, the item is showing DIF that changes character across the ability continuum.
For practical interpretation, if you see a significant MH chi-square with an odds ratio close to 1.0, suspect non-uniform DIF. The pooling across strata may be averaging out opposing effects, leaving a misleadingly small common odds ratio.
Step-by-Step Interpretation Guide
Follow these steps every time you read a Mantel-Haenszel DIF output to avoid missing critical information or over-interpreting noise.
Step 1: Check the sample sizes. Confirm the focal and reference groups each have enough examinees per stratum. A common rule of thumb is at least 200 examinees in the focal group for stable MH estimates. Sparse strata produce unreliable odds ratios and inflated Type I error rates.
Step 2: Read the MH chi-square and its p-value. A chi-square above 3.84 with a p-value below 0.05 tells you DIF is statistically present. If the chi-square is below 3.84 and the item is not flagged on magnitude alone, the item is likely Category A.
Step 3: Examine the alpha-MH odds ratio. A value at 1.0 means no DIF. Values above 1.0 favor the reference group; values below 1.0 favor the focal group. The further from 1.0, the larger the effect.
Step 4: Convert to MH D-DIF if your software reports it. This delta index puts both directions of DIF on a symmetric scale and is what the ETS classification cutoffs use directly.
Step 5: Apply the ETS classification. Use the A, B, C criteria to label the item. Category A means no action. Category B means content review. Category C means serious review with revision or removal likely.
Step 6: Send flagged items for content review. Statistics identify which items to look at. Subject-matter experts determine why the DIF exists and whether it reflects construct-irrelevant variance that warrants action.
Step 7: Document your decision. Whether you keep, revise, or retire the item, record the DIF statistics, the classification, and the content review rationale. This documentation supports test validity arguments and audit trails.
Common Mistakes When Interpreting Mantel-Haenszel DIF Statistics
The most common mistake I see practitioners make is treating statistical significance as the entire story. With large samples, an MH chi-square can be significant for an odds ratio of 1.05, which is practically meaningless. Always pair significance with the effect size before deciding an item is problematic.
A second frequent error is interpreting the direction of DIF incorrectly. An alpha-MH above 1.0 favors the reference group, not the focal group. Practitioners sometimes reverse this in their heads, especially when the focal group is the one they are trying to protect. Write the direction down explicitly to avoid this confusion.
A third mistake is ignoring non-uniform DIF. If you only run the standard MH uniform DIF test, you may miss items where the group difference reverses across ability levels. Always check whether your software reports a non-uniform DIF test, and run it alongside the uniform test.
A fourth error is over-relying on MH for small samples. The Mantel-Haenszel procedure assumes reasonably dense strata. If your focal group is small, the odds ratio estimates become unstable and the chi-square loses power. Consider logistic regression DIF methods or IRT-based DIF as alternatives when sample sizes are limited.
Finally, never confuse DIF with impact. Impact refers to raw group differences in mean performance, which can reflect genuine ability differences. DIF controls for ability by matching on total score. The two concepts answer different questions and should not be used interchangeably.
Sample Size Considerations
Sample size directly affects both the power and the stability of Mantel-Haenszel DIF statistics. The focal group is usually the smaller group, and it drives the precision of the analysis.
A widely cited guideline suggests a minimum of 200 examinees in the focal group for acceptable MH estimation. Below that threshold, odds ratio estimates become volatile, and the risk of false negatives increases because the chi-square lacks power to detect real DIF.
At the other extreme, very large samples can produce significant chi-square values for trivially small odds ratios. This is exactly why the ETS classification system weights effect size alongside significance. With 50,000 examinees per group, an alpha-MH of 1.03 may produce a p-value below 0.05, but it would still fall into Category A because the magnitude is negligible.
For stratum-level density, aim for at least 5 examinees per group per score level, and ideally more. If many strata have sparse cells, consider collapsing adjacent score levels or switching to a different DIF method that handles sparse data better.
Software Implementation Notes
Different software packages present Mantel-Haenszel DIF statistics in slightly different formats, which can confuse practitioners moving between tools.
The difR package in R, widely used in academic settings, reports the MH chi-square, the alpha-MH common odds ratio, and the MH D-DIF delta transformation. It also provides both uniform and non-uniform DIF tests. The output format is consistent and well-documented in the package vignette.
Winsteps, popular in Rasch measurement contexts, reports MH DIF statistics under its DIF analysis output. It uses the same underlying Mantel-Haenszel computation but presents results with slightly different labeling. Winsteps emphasizes the ETS classification directly in the output.
Stata users typically implement MH DIF through the mhodds command or through community-contributed packages. The output includes the chi-square and odds ratio, though the ETS classification may need to be computed manually from the reported values.
Regardless of the tool, the interpretation logic is identical. Identify the chi-square, identify the odds ratio, apply the ETS criteria, and route flagged items for content review.
What is differential item functioning (DIF)?
Differential item functioning occurs when examinees from different groups have a different probability of answering a test item correctly after being matched on the underlying ability the test measures. DIF flags items that may be biased against a specific group, but content review is needed to confirm whether the difference reflects construct-irrelevant variance.
What is the Mantel-Haenszel test statistic?
The Mantel-Haenszel test statistic is a chi-square-based measure that tests whether a test item functions differently for two groups matched on total score. It pools two-by-two comparisons across score strata into a common odds ratio and a significance test, allowing psychometricians to detect and quantify item bias.
What does DIF indicate in a psychological test?
In a psychological test, DIF indicates that an item behaves differently for examinees from different groups who share the same level of the construct being measured. This suggests the item may be measuring something besides the intended construct, which threatens validity and fairness for the affected group.
What is differential item functioning in IRT?
In item response theory, DIF refers to group differences in the item parameters (such as difficulty or discrimination) of an IRT model. IRT-based DIF analysis compares item characteristic curves across groups, while the Mantel-Haenszel approach uses observed total scores as the matching variable instead of estimated latent ability.
What is the difference between the Mantel-Haenszel test and the Cochran-Mantel-Haenszel test?
In DIF analysis the two terms are often used interchangeably. Strictly speaking, the Cochran-Mantel-Haenszel test is the general stratified analysis framework, while the Mantel-Haenszel statistic specifically refers to the version adapted for DIF detection by Holland and Thayer. Both condition on a matching variable and produce a common odds ratio.
What is the Mantel-Haenszel mean score test?
The Mantel-Haenszel mean score test extends the MH framework to polytomously scored items by testing whether the mean item score differs between groups within each ability stratum. It produces a chi-square statistic that flags items where the average response differs across matched groups.
What sample size is needed for reliable DIF detection?
A widely used guideline recommends at least 200 examinees in the focal group for stable Mantel-Haenszel DIF estimates. Stratum-level density should ideally include at least 5 examinees per group per score level. Larger samples increase power but also raise the risk of flagging trivially small DIF, which is why effect size thresholds matter.
Conclusion
Knowing how to interpret Mantel-Haenszel DIF statistics means reading three numbers in sequence: the chi-square for significance, the alpha-MH odds ratio for magnitude and direction, and the ETS classification for practical action. Category A items are fair, Category B items warrant review, and Category C items demand serious attention.
Always pair the statistics with content review, watch for non-uniform DIF, and respect sample size limits. When you apply these steps consistently, you protect both the fairness of your assessments and the validity evidence that supports them. The Mantel-Haenszel procedure remains the workhorse of DIF detection because it is transparent, well-validated, and directly interpretable through the ETS framework that testing programs have trusted for decades.