Tukey's HSD (Honestly Significant Difference): A post-hoc test that compares all possible pairs of group means while controlling the family-wise error rate.
- Verify ANOVA was significant
- Identify group means and sample sizes
- Obtain MSE from ANOVA output
- Find critical q-value from studentized range table
- Calculate HSD = q * √(MSE/n)
- Calculate all pairwise mean differences
- Compare differences with HSD
- Identify significant pairs
F = 5.8, p = 0.003 < 0.05, so ANOVA is significant
Proceed with post-hoc tests
k = 4 groups, n = 10 per group
MSE = 24.6
α = 0.05
For α = 0.05, k = 4, df_error = 40 - 4 = 36
q₀.₀₅,₄,₃₆ ≈ 3.81
HSD = q * √(MSE/n) = 3.81 * √(24.6/10)
HSD = 3.81 * √2.46 = 3.81 * 1.57 = 5.98
|A - B| = |78.5 - 82.3| = 3.8
|A - C| = |78.5 - 85.1| = 6.6
|A - D| = |78.5 - 76.2| = 2.3
|B - C| = |82.3 - 85.1| = 2.8
|B - D| = |82.3 - 76.2| = 6.1
|C - D| = |85.1 - 76.2| = 8.9
3.8 < 5.98 → Not significant (A vs B)
6.6 > 5.98 → Significant (A vs C)
2.3 < 5.98 → Not significant (A vs D)
2.8 < 5.98 → Not significant (B vs C)
6.1 > 5.98 → Significant (B vs D)
8.9 > 5.98 → Highly significant (C vs D)
Significant differences: Method A vs Method C, Method B vs Method D, Method C vs Method D
Tukey's HSD identifies significant differences between: Method A and Method C, Method B and Method D, and Method C and Method D. The critical difference is 5.98 points.
• Family-wise error rate: Controls overall Type I error for multiple comparisons
• Pairwise comparison: Tests all possible group pairs
• Equal sample sizes: Uses pooled standard error
• Method C (85.1) is significantly better than Methods A, B, and D
• Method D (76.2) is significantly worse than Methods A, B, and C
• No significant difference between Methods A and B
Bonferroni correction: A method that adjusts the significance level for multiple comparisons by dividing α by the number of comparisons.
For k = 4 groups: Number of pairs = k(k-1)/2 = 4(3)/2 = 6
Adjusted α = α_original / number of comparisons
Adjusted α = 0.05 / 6 = 0.0083
For 2-tailed test with adjusted α/2 = 0.0042, df = 36
t_critical ≈ 2.92
SE_diff = √(MSE/n + MSE/n) = √(24.6/10 + 24.6/10) = √4.92 = 2.22
Critical difference = t_critical × SE_diff = 2.92 × 2.22 = 6.48
Same differences as before:
A-B: 3.8 < 6.48 → Not significant
A-C: 6.6 > 6.48 → Significant
A-D: 2.3 < 6.48 → Not significant
B-C: 2.8 < 6.48 → Not significant
B-D: 6.1 < 6.48 → Not significant
C-D: 8.9 > 6.48 → Significant
Tukey: A-C, B-D, C-D significant
Bonferroni: A-C, C-D significant
Bonferroni is more conservative (fewer significant results)
The Bonferroni correction identifies fewer significant differences than Tukey's HSD: only A vs C and C vs D are significant. Bonferroni is more conservative with lower Type I error rate.
• Conservative approach: Reduces Type I error at cost of power
• Adjustment factor: α_new = α_original / m
• Multiple comparisons: Controls family-wise error rate
• Tukey: More powerful, assumes equal sample sizes
• Bonferroni: More conservative, applicable to any test
• Choice depends on priorities: power vs. error control
Uncorrected pairwise tests: Conducting multiple t-tests without adjusting for multiple comparisons, which inflates Type I error rate.
SE_diff = √(MSE/n + MSE/n) = √(24.6/10 + 24.6/10) = 2.22
t(A-B) = (78.5 - 82.3) / 2.22 = -3.8 / 2.22 = -1.71
t(A-C) = (78.5 - 85.1) / 2.22 = -6.6 / 2.22 = -2.97
t(A-D) = (78.5 - 76.2) / 2.22 = 2.3 / 2.22 = 1.04
t(B-C) = (82.3 - 85.1) / 2.22 = -2.8 / 2.22 = -1.26
t(B-D) = (82.3 - 76.2) / 2.22 = 6.1 / 2.22 = 2.75
t(C-D) = (85.1 - 76.2) / 2.22 = 8.9 / 2.22 = 4.01
For α = 0.05 (uncorrected), df = 36
t_critical = ±2.03
|t(A-B)| = 1.71 < 2.03 → Not significant
|t(A-C)| = 2.97 > 2.03 → Significant
|t(A-D)| = 1.04 < 2.03 → Not significant
|t(B-C)| = 1.26 < 2.03 → Not significant
|t(B-D)| = 2.75 > 2.03 → Significant
|t(C-D)| = 4.01 > 2.03 → Significant
Uncorrected: A-C, B-D, C-D significant (3 pairs)
Tukey: A-C, B-D, C-D significant (3 pairs)
Bonferroni: A-C, C-D significant (2 pairs)
With 6 uncorrected tests at α = 0.05:
Probability of at least one Type I error ≈ 1 - (0.95)⁶ = 1 - 0.735 = 0.265
So 26.5% chance of false discovery with uncorrected tests
Uncorrected tests have much higher Type I error rate
Corrected methods control family-wise error rate
Uncorrected pairwise tests yield 3 significant pairs but have inflated Type I error rate (≈26.5% vs. 5% for corrected methods). Always use correction for multiple comparisons.
• Error rate inflation: Multiple tests increase Type I error probability
• Correction necessity: Always adjust for multiple comparisons
• Method comparison: Balance between power and error control
• Without correction: High chance of false discoveries
• With correction: More conservative but controlled error rate
• Trade-off: Power vs. Type I error control
Post-hoc test: A statistical test performed after an ANOVA to determine which specific group means differ significantly.
Multiple comparisons: The problem of conducting multiple statistical tests simultaneously, increasing the chance of Type I errors.
Family-wise error rate: The probability of making at least one Type I error across all comparisons in a family of tests.
Type I error: Incorrectly rejecting a true null hypothesis (false positive).
Type II error: Failing to reject a false null hypothesis (false negative).
Statistical power: The probability of correctly rejecting a false null hypothesis.
Effect size: A measure of the magnitude of a phenomenon, independent of sample size.
- Confirm ANOVA significance: Verify that overall ANOVA is significant before post-hoc tests
- Select post-hoc method: Choose appropriate test based on sample sizes and objectives
- Identify parameters: Extract MSE, sample sizes, and degrees of freedom
- Calculate critical values: Obtain appropriate critical values from tables
- Compute test statistics: Calculate all pairwise comparisons
- Apply decision rules: Compare with critical values
- Adjust for multiple testing: Use appropriate correction method
- Report results: Clearly state which pairs differ significantly
- Interpret practically: Connect statistical findings to research question
• Error rate control: Correct for multiple comparisons to maintain α level
• Power considerations: More conservative methods reduce power
• Sample size effects: Larger samples provide more power for detection
• Method selection: Choose based on sample sizes and research goals
• Interpretation: Focus on practically significant differences
• Tukey: Best for all pairwise comparisons with equal n
• Bonferroni: Conservative, works for any comparisons
• Scheffé: Most conservative, for complex contrasts
• Fisher LSD: Unprotected, high error rate
Tukey-Kramer method: Extension of Tukey's HSD for unequal sample sizes, using harmonic mean of sample sizes.
k = 3 groups
n₁ = 8, n₂ = 12, n₃ = 6
Means: 25.3, 28.7, 22.1
MSE = 18.4
For α = 0.05, k = 3, df_error = (8+12+6) - 3 = 23
q₀.₀₅,₃,₂₃ ≈ 3.53
HSD_{ij} = q * √(MSE * (1/n_i + 1/n_j)/2)
For each pair, calculate the critical difference separately
Pair 1-2: HSD = 3.53 * √(18.4 * (1/8 + 1/12)) = 3.53 * √(18.4 * 0.208) = 3.53 * √3.83 = 3.53 * 1.96 = 6.92
Pair 1-3: HSD = 3.53 * √(18.4 * (1/8 + 1/6)) = 3.53 * √(18.4 * 0.292) = 3.53 * √5.37 = 3.53 * 2.32 = 8.19
Pair 2-3: HSD = 3.53 * √(18.4 * (1/12 + 1/6)) = 3.53 * √(18.4 * 0.250) = 3.53 * √4.60 = 3.53 * 2.14 = 7.56
|1-2| = |25.3 - 28.7| = 3.4
|1-3| = |25.3 - 22.1| = 3.2
|2-3| = |28.7 - 22.1| = 6.6
3.4 < 6.92 → Not significant (Group 1 vs Group 2)
3.2 < 8.19 → Not significant (Group 1 vs Group 3)
6.6 < 7.56 → Not significant (Group 2 vs Group 3)
None of the pairwise comparisons are significant at α = 0.05 using Tukey-Kramer correction
Using the Tukey-Kramer method for unequal sample sizes, none of the pairwise comparisons are statistically significant at α = 0.05. The differences between group means are smaller than the critical differences required for significance.
• Unequal n adjustment: Use harmonic mean of sample sizes
• Individual HSD: Calculate for each pair separately
• Conservative test: Tukey-Kramer controls family-wise error rate
Method comparison: Evaluating different post-hoc procedures to understand their relative conservativeness and power.
Group means: A=50, B=55, C=52, D=58
Pairwise differences:
A-B: |50-55| = 5, A-C: |50-52| = 2, A-D: |50-58| = 8
B-C: |55-52| = 3, B-D: |55-58| = 3, C-D: |52-58| = 6
SE_diff = √(MSE/n + MSE/n) = √(20/10 + 20/10) = √4 = 2
Critical t-value: t₀.₀₂₅,₃₆ ≈ 2.03
LSD = 2.03 × 2 = 4.06
Significant if |difference| > 4.06
A-B: 5 > 4.06 ✓, A-D: 8 > 4.06 ✓, C-D: 6 > 4.06 ✓
Others not significant
q₀.₀₅,₄,₃₆ ≈ 3.81
HSD = 3.81 × √(20/10) = 3.81 × √2 = 3.81 × 1.41 = 5.37
Significant if |difference| > 5.37
A-D: 8 > 5.37 ✓, C-D: 6 > 5.37 ✓
Others not significant
Number of comparisons: m = 4(3)/2 = 6
Adjusted α = 0.05/6 = 0.0083
t₀.₀₀₄₂,₃₆ ≈ 2.92
Critical difference = 2.92 × 2 = 5.84
Only A-D: 8 > 5.84 ✓
Fisher's LSD: 3 significant pairs (A-B, A-D, C-D)
Tukey's HSD: 2 significant pairs (A-D, C-D)
Bonferroni: 1 significant pair (A-D)
Bonferroni is most conservative (lowest Type I error)
Tukey's HSD is moderate
Fisher's LSD is least conservative (highest Type I error)
Different methods yield different results: Fisher's LSD (3 significant pairs), Tukey's HSD (2 significant pairs), Bonferroni (1 significant pair). Bonferroni is most conservative, Fisher's LSD least conservative.
• Conservativeness order: Bonferroni > Tukey > Fisher
• Power trade-off: More conservative = less power
• Error control: Choose method based on priorities
• Fisher's LSD: Highest power, highest error rate
• Tukey's HSD: Good balance of power and error control
• Bonferroni: Lowest error rate, lowest power
• Choose based on research priorities
Fisher's Least Significant Difference (LSD): An unprotected post-hoc test that uses the same critical value as the overall ANOVA, without correcting for multiple comparisons.
Scheffé method: A very conservative post-hoc test that controls for all possible contrasts, not just pairwise comparisons.
Dunnett's test: A post-hoc test designed specifically for comparing multiple treatments to a single control group.
Games-Howell test: A post-hoc test that does not assume equal variances, suitable for heterogeneous variances.
Contrasts: Specific comparisons between group means, often planned before seeing the data.
Planned comparisons: Comparisons specified before conducting the experiment, which don't require multiple comparison corrections.
- Examine sample sizes: Equal vs. unequal sample sizes
- Check variance homogeneity: Equal vs. unequal variances
- Identify comparison type: All pairs vs. specific comparisons vs. vs. control
- Consider error rate priorities: Balance between Type I and Type II errors
- Account for power requirements: Need for sensitivity to detect differences
- Select appropriate method: Match method to data characteristics
- Implement and verify: Calculate and check results
- Interpret practically: Connect to research objectives
• Method selection: Choose based on data characteristics and research goals
• Error rate balance: Consider trade-off between Type I and Type II errors
• Power efficiency: More conservative methods sacrifice power for error control
• Assumption checking: Verify equal variance assumption when using standard methods
• Practical significance: Consider effect sizes and real-world importance
• Equal n, equal variances, all pairs: Tukey's HSD
• Unequal n, equal variances, all pairs: Tukey-Kramer
• Any n, unequal variances: Games-Howell
• vs. control only: Dunnett's test
• Planned comparisons: Fisher's LSD