Solved Exercises on Introduction to Post-Hoc Tests (Multiple Comparisons)

Master post-hoc tests: Tukey's HSD, Bonferroni correction, pairwise comparisons, and multiple testing adjustments through these 5 detailed exercises.

Solution: Exercises 1 to 3
1 Tukey's HSD Test
Exercise 1
An ANOVA found significant differences among four teaching methods (F = 5.8, p = 0.003). The group means are: Method A: 78.5, Method B: 82.3, Method C: 85.1, Method D: 76.2. Sample size for each group is 10, and MSE = 24.6. Use Tukey's HSD to identify which pairs differ significantly at α = 0.05.
Definition:

Tukey's HSD (Honestly Significant Difference): A post-hoc test that compares all possible pairs of group means while controlling the family-wise error rate.

Tukey's HSD Methodology:
  1. Verify ANOVA was significant
  2. Identify group means and sample sizes
  3. Obtain MSE from ANOVA output
  4. Find critical q-value from studentized range table
  5. Calculate HSD = q * √(MSE/n)
  6. Calculate all pairwise mean differences
  7. Compare differences with HSD
  8. Identify significant pairs
Step 1: Verify ANOVA significance

F = 5.8, p = 0.003 < 0.05, so ANOVA is significant

Proceed with post-hoc tests

Step 2: Identify parameters

k = 4 groups, n = 10 per group

MSE = 24.6

α = 0.05

Step 3: Find critical q-value

For α = 0.05, k = 4, df_error = 40 - 4 = 36

q₀.₀₅,₄,₃₆ ≈ 3.81

Step 4: Calculate HSD

HSD = q * √(MSE/n) = 3.81 * √(24.6/10)

HSD = 3.81 * √2.46 = 3.81 * 1.57 = 5.98

Step 5: Calculate all pairwise differences

|A - B| = |78.5 - 82.3| = 3.8

|A - C| = |78.5 - 85.1| = 6.6

|A - D| = |78.5 - 76.2| = 2.3

|B - C| = |82.3 - 85.1| = 2.8

|B - D| = |82.3 - 76.2| = 6.1

|C - D| = |85.1 - 76.2| = 8.9

Step 6: Compare differences with HSD

3.8 < 5.98 → Not significant (A vs B)

6.6 > 5.98 → Significant (A vs C)

2.3 < 5.98 → Not significant (A vs D)

2.8 < 5.98 → Not significant (B vs C)

6.1 > 5.98 → Significant (B vs D)

8.9 > 5.98 → Highly significant (C vs D)

Step 7: Draw conclusions

Significant differences: Method A vs Method C, Method B vs Method D, Method C vs Method D

HSD = 5.98, significant pairs: A-C, B-D, C-D
Final answer:

Tukey's HSD identifies significant differences between: Method A and Method C, Method B and Method D, and Method C and Method D. The critical difference is 5.98 points.

Applied rules:

• Family-wise error rate: Controls overall Type I error for multiple comparisons

• Pairwise comparison: Tests all possible group pairs

• Equal sample sizes: Uses pooled standard error

Post-hoc interpretation:

• Method C (85.1) is significantly better than Methods A, B, and D

• Method D (76.2) is significantly worse than Methods A, B, and C

• No significant difference between Methods A and B

2 Bonferroni Correction
Exercise 2
Using the same data as Exercise 1, apply the Bonferroni correction for multiple comparisons. Compare the results with Tukey's HSD.
Definition:

Bonferroni correction: A method that adjusts the significance level for multiple comparisons by dividing α by the number of comparisons.

Step 1: Calculate number of comparisons

For k = 4 groups: Number of pairs = k(k-1)/2 = 4(3)/2 = 6

Step 2: Adjust significance level

Adjusted α = α_original / number of comparisons

Adjusted α = 0.05 / 6 = 0.0083

Step 3: Calculate t-critical for adjusted α

For 2-tailed test with adjusted α/2 = 0.0042, df = 36

t_critical ≈ 2.92

Step 4: Calculate standard error for difference

SE_diff = √(MSE/n + MSE/n) = √(24.6/10 + 24.6/10) = √4.92 = 2.22

Step 5: Calculate critical difference

Critical difference = t_critical × SE_diff = 2.92 × 2.22 = 6.48

Step 6: Compare with pairwise differences

Same differences as before:

A-B: 3.8 < 6.48 → Not significant

A-C: 6.6 > 6.48 → Significant

A-D: 2.3 < 6.48 → Not significant

B-C: 2.8 < 6.48 → Not significant

B-D: 6.1 < 6.48 → Not significant

C-D: 8.9 > 6.48 → Significant

Step 7: Compare with Tukey's results

Tukey: A-C, B-D, C-D significant

Bonferroni: A-C, C-D significant

Bonferroni is more conservative (fewer significant results)

Bonferroni: critical diff = 6.48, significant: A-C, C-D
Final answer:

The Bonferroni correction identifies fewer significant differences than Tukey's HSD: only A vs C and C vs D are significant. Bonferroni is more conservative with lower Type I error rate.

Applied rules:

• Conservative approach: Reduces Type I error at cost of power

• Adjustment factor: α_new = α_original / m

• Multiple comparisons: Controls family-wise error rate

Comparison of methods:

• Tukey: More powerful, assumes equal sample sizes

• Bonferroni: More conservative, applicable to any test

• Choice depends on priorities: power vs. error control

3 Pairwise T-Tests
Exercise 3
Conduct pairwise t-tests for the same data without any correction. Compare results with corrected methods.
Definition:

Uncorrected pairwise tests: Conducting multiple t-tests without adjusting for multiple comparisons, which inflates Type I error rate.

Step 1: Calculate standard error for difference

SE_diff = √(MSE/n + MSE/n) = √(24.6/10 + 24.6/10) = 2.22

Step 2: Calculate t-statistics for each pair

t(A-B) = (78.5 - 82.3) / 2.22 = -3.8 / 2.22 = -1.71

t(A-C) = (78.5 - 85.1) / 2.22 = -6.6 / 2.22 = -2.97

t(A-D) = (78.5 - 76.2) / 2.22 = 2.3 / 2.22 = 1.04

t(B-C) = (82.3 - 85.1) / 2.22 = -2.8 / 2.22 = -1.26

t(B-D) = (82.3 - 76.2) / 2.22 = 6.1 / 2.22 = 2.75

t(C-D) = (85.1 - 76.2) / 2.22 = 8.9 / 2.22 = 4.01

Step 3: Find critical t-value

For α = 0.05 (uncorrected), df = 36

t_critical = ±2.03

Step 4: Make decisions for each comparison

|t(A-B)| = 1.71 < 2.03 → Not significant

|t(A-C)| = 2.97 > 2.03 → Significant

|t(A-D)| = 1.04 < 2.03 → Not significant

|t(B-C)| = 1.26 < 2.03 → Not significant

|t(B-D)| = 2.75 > 2.03 → Significant

|t(C-D)| = 4.01 > 2.03 → Significant

Step 5: Compare all three methods

Uncorrected: A-C, B-D, C-D significant (3 pairs)

Tukey: A-C, B-D, C-D significant (3 pairs)

Bonferroni: A-C, C-D significant (2 pairs)

Step 6: Analyze Type I error inflation

With 6 uncorrected tests at α = 0.05:

Probability of at least one Type I error ≈ 1 - (0.95)⁶ = 1 - 0.735 = 0.265

So 26.5% chance of false discovery with uncorrected tests

Step 7: Draw conclusion about error rates

Uncorrected tests have much higher Type I error rate

Corrected methods control family-wise error rate

Uncorrected: 3 significant pairs, Type I error rate ≈ 26.5%
Final answer:

Uncorrected pairwise tests yield 3 significant pairs but have inflated Type I error rate (≈26.5% vs. 5% for corrected methods). Always use correction for multiple comparisons.

Applied rules:

• Error rate inflation: Multiple tests increase Type I error probability

• Correction necessity: Always adjust for multiple comparisons

• Method comparison: Balance between power and error control

Multiple testing consequences:

• Without correction: High chance of false discoveries

• With correction: More conservative but controlled error rate

• Trade-off: Power vs. Type I error control

Post-Hoc Test Concepts and Methods
\( \text{Tukey HSD} = q_{\alpha,k,N-k} \cdot \sqrt{\frac{MSE}{n}}, \quad \text{Bonferroni } \alpha_{adj} = \frac{\alpha}{m} \)
Post-Hoc Test Formulas
Number of Comparisons
\( m = \frac{k(k-1)}{2} \)
For k groups
Family-wise Error Rate
\( \alpha_{FW} = 1-(1-\alpha)^m \)
Uncorrected inflation
Tukey Critical Value
\( q_{\alpha,k,v} \)
Studentized range distribution
Pairwise Difference
\( |\bar{x}_i - \bar{x}_j| \)
Mean comparison
Key definitions:

Post-hoc test: A statistical test performed after an ANOVA to determine which specific group means differ significantly.

Multiple comparisons: The problem of conducting multiple statistical tests simultaneously, increasing the chance of Type I errors.

Family-wise error rate: The probability of making at least one Type I error across all comparisons in a family of tests.

Type I error: Incorrectly rejecting a true null hypothesis (false positive).

Type II error: Failing to reject a false null hypothesis (false negative).

Statistical power: The probability of correctly rejecting a false null hypothesis.

Effect size: A measure of the magnitude of a phenomenon, independent of sample size.

Complete Post-Hoc Analysis Methodology:
  1. Confirm ANOVA significance: Verify that overall ANOVA is significant before post-hoc tests
  2. Select post-hoc method: Choose appropriate test based on sample sizes and objectives
  3. Identify parameters: Extract MSE, sample sizes, and degrees of freedom
  4. Calculate critical values: Obtain appropriate critical values from tables
  5. Compute test statistics: Calculate all pairwise comparisons
  6. Apply decision rules: Compare with critical values
  7. Adjust for multiple testing: Use appropriate correction method
  8. Report results: Clearly state which pairs differ significantly
  9. Interpret practically: Connect statistical findings to research question
Tip 1: Never perform post-hoc tests if ANOVA is not significant (protected test).
Tip 2: Tukey's HSD is best for equal sample sizes and all pairwise comparisons.
Tip 3: Bonferroni is more conservative but works for any number of comparisons.
Tip 4: For unequal sample sizes, use Tukey-Kramer modification.

Tip 5: Consider effect sizes along with significance tests.

Common applications: Clinical trials, agricultural experiments, educational research, industrial quality control, market research.
Advanced considerations: Planned comparisons, Dunnett's test for control group, Games-Howell for unequal variances.
Key principles and guidelines:

• Error rate control: Correct for multiple comparisons to maintain α level

• Power considerations: More conservative methods reduce power

• Sample size effects: Larger samples provide more power for detection

• Method selection: Choose based on sample sizes and research goals

• Interpretation: Focus on practically significant differences

Multiple comparison methods comparison:

• Tukey: Best for all pairwise comparisons with equal n

• Bonferroni: Conservative, works for any comparisons

• Scheffé: Most conservative, for complex contrasts

• Fisher LSD: Unprotected, high error rate

Solution: Exercises 4 to 5
4 Unequal Sample Sizes
Exercise 4
Compare three groups with unequal sample sizes: Group 1 (n₁=8): mean = 25.3; Group 2 (n₂=12): mean = 28.7; Group 3 (n₃=6): mean = 22.1. MSE = 18.4. Use Tukey-Kramer method at α = 0.05.
Definition:

Tukey-Kramer method: Extension of Tukey's HSD for unequal sample sizes, using harmonic mean of sample sizes.

Step 1: Identify parameters

k = 3 groups

n₁ = 8, n₂ = 12, n₃ = 6

Means: 25.3, 28.7, 22.1

MSE = 18.4

Step 2: Find critical q-value

For α = 0.05, k = 3, df_error = (8+12+6) - 3 = 23

q₀.₀₅,₃,₂₃ ≈ 3.53

Step 3: Apply Tukey-Kramer formula

HSD_{ij} = q * √(MSE * (1/n_i + 1/n_j)/2)

For each pair, calculate the critical difference separately

Step 4: Calculate HSD for each pair

Pair 1-2: HSD = 3.53 * √(18.4 * (1/8 + 1/12)) = 3.53 * √(18.4 * 0.208) = 3.53 * √3.83 = 3.53 * 1.96 = 6.92

Pair 1-3: HSD = 3.53 * √(18.4 * (1/8 + 1/6)) = 3.53 * √(18.4 * 0.292) = 3.53 * √5.37 = 3.53 * 2.32 = 8.19

Pair 2-3: HSD = 3.53 * √(18.4 * (1/12 + 1/6)) = 3.53 * √(18.4 * 0.250) = 3.53 * √4.60 = 3.53 * 2.14 = 7.56

Step 5: Calculate actual differences

|1-2| = |25.3 - 28.7| = 3.4

|1-3| = |25.3 - 22.1| = 3.2

|2-3| = |28.7 - 22.1| = 6.6

Step 6: Compare differences with HSD

3.4 < 6.92 → Not significant (Group 1 vs Group 2)

3.2 < 8.19 → Not significant (Group 1 vs Group 3)

6.6 < 7.56 → Not significant (Group 2 vs Group 3)

Step 7: Draw conclusion

None of the pairwise comparisons are significant at α = 0.05 using Tukey-Kramer correction

No significant differences: All |diff| < HSD
Final answer:

Using the Tukey-Kramer method for unequal sample sizes, none of the pairwise comparisons are statistically significant at α = 0.05. The differences between group means are smaller than the critical differences required for significance.

Applied rules:

• Unequal n adjustment: Use harmonic mean of sample sizes

• Individual HSD: Calculate for each pair separately

• Conservative test: Tukey-Kramer controls family-wise error rate

5 Multiple Methods Comparison
Exercise 5
Compare the results of Fisher's LSD, Tukey's HSD, and Bonferroni for the following data: 4 groups, each with n=10, MSE=20, means: 50, 55, 52, 58. ANOVA F = 8.2, p = 0.001.
Definition:

Method comparison: Evaluating different post-hoc procedures to understand their relative conservativeness and power.

Step 1: Calculate pairwise differences

Group means: A=50, B=55, C=52, D=58

Pairwise differences:

A-B: |50-55| = 5, A-C: |50-52| = 2, A-D: |50-58| = 8

B-C: |55-52| = 3, B-D: |55-58| = 3, C-D: |52-58| = 6

Step 2: Calculate standard error

SE_diff = √(MSE/n + MSE/n) = √(20/10 + 20/10) = √4 = 2

Step 3: Fisher's LSD (unprotected)

Critical t-value: t₀.₀₂₅,₃₆ ≈ 2.03

LSD = 2.03 × 2 = 4.06

Significant if |difference| > 4.06

A-B: 5 > 4.06 ✓, A-D: 8 > 4.06 ✓, C-D: 6 > 4.06 ✓

Others not significant

Step 4: Tukey's HSD

q₀.₀₅,₄,₃₆ ≈ 3.81

HSD = 3.81 × √(20/10) = 3.81 × √2 = 3.81 × 1.41 = 5.37

Significant if |difference| > 5.37

A-D: 8 > 5.37 ✓, C-D: 6 > 5.37 ✓

Others not significant

Step 5: Bonferroni correction

Number of comparisons: m = 4(3)/2 = 6

Adjusted α = 0.05/6 = 0.0083

t₀.₀₀₄₂,₃₆ ≈ 2.92

Critical difference = 2.92 × 2 = 5.84

Only A-D: 8 > 5.84 ✓

Step 6: Compare results

Fisher's LSD: 3 significant pairs (A-B, A-D, C-D)

Tukey's HSD: 2 significant pairs (A-D, C-D)

Bonferroni: 1 significant pair (A-D)

Step 7: Analyze conservativeness

Bonferroni is most conservative (lowest Type I error)

Tukey's HSD is moderate

Fisher's LSD is least conservative (highest Type I error)

Fisher: 3 sig, Tukey: 2 sig, Bonferroni: 1 sig
Final answer:

Different methods yield different results: Fisher's LSD (3 significant pairs), Tukey's HSD (2 significant pairs), Bonferroni (1 significant pair). Bonferroni is most conservative, Fisher's LSD least conservative.

Applied rules:

• Conservativeness order: Bonferroni > Tukey > Fisher

• Power trade-off: More conservative = less power

• Error control: Choose method based on priorities

Method comparison summary:

• Fisher's LSD: Highest power, highest error rate

• Tukey's HSD: Good balance of power and error control

• Bonferroni: Lowest error rate, lowest power

• Choose based on research priorities

Advanced Post-Hoc Concepts and Applications
\( \text{Fisher's LSD} = t_{\alpha/2,df_{error}} \cdot \sqrt{MSE \left(\frac{1}{n_i} + \frac{1}{n_j}\right)}, \quad \text{Scheffé} = \sqrt{(k-1) \cdot F_{\alpha,k-1,N-k}} \cdot \sqrt{MSE \left(\frac{1}{n_i} + \frac{1}{n_j}\right)} \)
Advanced Post-Hoc Formulas
Key definitions:

Fisher's Least Significant Difference (LSD): An unprotected post-hoc test that uses the same critical value as the overall ANOVA, without correcting for multiple comparisons.

Scheffé method: A very conservative post-hoc test that controls for all possible contrasts, not just pairwise comparisons.

Dunnett's test: A post-hoc test designed specifically for comparing multiple treatments to a single control group.

Games-Howell test: A post-hoc test that does not assume equal variances, suitable for heterogeneous variances.

Contrasts: Specific comparisons between group means, often planned before seeing the data.

Planned comparisons: Comparisons specified before conducting the experiment, which don't require multiple comparison corrections.

Advanced Post-Hoc Selection Methodology:
  1. Examine sample sizes: Equal vs. unequal sample sizes
  2. Check variance homogeneity: Equal vs. unequal variances
  3. Identify comparison type: All pairs vs. specific comparisons vs. vs. control
  4. Consider error rate priorities: Balance between Type I and Type II errors
  5. Account for power requirements: Need for sensitivity to detect differences
  6. Select appropriate method: Match method to data characteristics
  7. Implement and verify: Calculate and check results
  8. Interpret practically: Connect to research objectives
Tip 1: For planned comparisons, you don't need to correct for multiple testing.
Tip 2: Use Games-Howell if Levene's test shows unequal variances.
Tip 3: Dunnett's test is ideal when comparing treatments to a control group.
Tip 4: Consider effect sizes alongside significance tests for practical importance.

Tip 5: Report confidence intervals for mean differences, not just significance.

Common pitfalls: Forgetting to check ANOVA significance, using uncorrected tests, choosing inappropriate method for data characteristics.
Advanced applications: Factorial ANOVA post-hocs, repeated measures post-hocs, multivariate post-hocs.
Key principles and considerations:

• Method selection: Choose based on data characteristics and research goals

• Error rate balance: Consider trade-off between Type I and Type II errors

• Power efficiency: More conservative methods sacrifice power for error control

• Assumption checking: Verify equal variance assumption when using standard methods

• Practical significance: Consider effect sizes and real-world importance

Method selection guide:

• Equal n, equal variances, all pairs: Tukey's HSD

• Unequal n, equal variances, all pairs: Tukey-Kramer

• Any n, unequal variances: Games-Howell

• vs. control only: Dunnett's test

• Planned comparisons: Fisher's LSD

Questions & Answers

Question: Why do we need post-hoc tests? Why can't we just do multiple t-tests?

Answer: Multiple t-tests would inflate the Type I error rate:

  • Single test: α = 0.05 means 5% chance of false discovery
  • Multiple tests: With 6 comparisons, probability of at least one false discovery ≈ 26.5%
  • Family-wise error rate: Increases dramatically with more tests

Post-hoc tests control this inflation by adjusting the significance level or critical value:

  • Tukey's HSD: Uses studentized range distribution
  • Bonferroni: Divides α by number of comparisons
  • Scheffé: Very conservative approach for all possible contrasts

The ANOVA omnibus test acts as a gatekeeper - if it's not significant, post-hoc tests shouldn't be performed (except for planned comparisons).

Post-hoc tests balance the need to identify specific differences while maintaining the overall error rate.

Without correction, you'd find "significant" results that are actually due to chance, leading to false discoveries.

Question: What's the difference between Tukey's HSD and Bonferroni correction?

Answer: Tukey's HSD and Bonferroni differ in approach and conservativeness:

  • Tukey's HSD: Specifically designed for all pairwise comparisons, uses studentized range distribution
  • Bonferroni: General-purpose method that divides α by number of tests

Key differences:

  • Power: Tukey's HSD is more powerful for all pairwise comparisons
  • Conservativeness: Bonferroni is more conservative (lower Type I error, lower power)
  • Applicability: Bonferroni works for any set of tests; Tukey for all pairs
  • Sample sizes: Tukey assumes equal sample sizes (Kramer modification for unequal)

Tukey's HSD is optimal when you want to compare all pairs of means. Bonferroni is better when you're making a subset of comparisons or when sample sizes vary significantly.

The choice depends on your specific research questions and priorities regarding Type I vs. Type II error rates.

Both methods control the family-wise error rate but through different mechanisms.

Question: When should I use Dunnett's test versus Tukey's HSD?

Answer: Choose based on your comparison structure:

  • Tukey's HSD: When you want to compare ALL possible pairs of means
  • Dunnett's test: When you want to compare multiple treatments to ONE control group

Dunnett's test is specifically designed for treatment vs. control comparisons:

  • More powerful for treatment-control comparisons
  • Adjusts for multiple comparisons only among treatment-control pairs
  • Doesn't compare treatments to each other

For example:

  • Drug study: Several experimental drugs vs. placebo → Dunnett
  • Teaching methods: All methods compared to each other → Tukey

Dunnett's test is more focused and therefore more powerful when you only need treatment vs. control comparisons.

If you need all pairwise comparisons, Tukey's HSD is more appropriate.

The choice should align with your research design and specific hypotheses.