P-values and Statistical Significance: Comprehensive Guide

Master p-values and statistical significance: calculation, interpretation, decision-making, and practical applications through these 5 detailed exercises.

P-values and Significance Fundamentals
p-value = P(data | H₀ true)
P-value Definition
Key Definitions:

P-value: Probability of obtaining results at least as extreme as observed data, assuming null hypothesis is true

Statistical Significance: Result is unlikely to occur by chance alone, based on predetermined threshold

Alpha Level (α): Threshold for determining significance (typically 0.05, 0.01, or 0.10)

Decision Rule: If p-value ≤ α, reject H₀; if p-value > α, fail to reject H₀

P-value Interpretation:

Small p-value (≤ α): Strong evidence against H₀, suggests significant result

Large p-value (> α): Weak evidence against H₀, suggests non-significant result

Smaller p-value: Stronger evidence against null hypothesis

Tip 1: P-value is NOT the probability that H₀ is true
Tip 2: Smaller p-value = stronger evidence against H₀
Tip 3: Always compare p-value to α, never interpret in isolation
Common α Levels: 0.05 (standard), 0.01 (conservative), 0.10 (liberal)
Significance Categories: Very significant (p < 0.001), Significant (p < 0.05), Not significant (p ≥ 0.05)
Solution: Exercises 1 to 3
1 Basic P-value Calculation
Exercise 1
A z-test yields a test statistic of z = 2.1. Find the p-value for a two-tailed test and determine if it's statistically significant at α = 0.05.
Problem Setup:

Given: z = 2.1, two-tailed test, α = 0.05

P-value Calculation Process:

For two-tailed test: p-value = 2 × P(Z ≥ |observed value|)

Step 1: Find area in one tail

P(Z ≥ 2.1) = 1 - P(Z ≤ 2.1) = 1 - 0.9821 = 0.0179

(Using standard normal table)

Step 2: Calculate two-tailed p-value

p-value = 2 × 0.0179 = 0.0358

Step 3: Compare to significance level

Since p-value (0.0358) < α (0.05), reject H₀

Step 4: Interpret the result

The result is statistically significant at the 0.05 level

Final Answer:

P-value = 0.0358

Decision: Reject H₀ (statistically significant)

Applied Rules:

• Two-tailed Formula: Multiply one-tail probability by 2

• Decision Rule: Reject H₀ if p-value ≤ α

• Significance Interpretation: p-value < 0.05 indicates significance

2 Right-tailed Test
Exercise 2
A t-test with df = 24 gives t = 1.85. Find the p-value for a right-tailed test and determine significance at α = 0.05.
Problem Setup:

Given: t = 1.85, df = 24, right-tailed test, α = 0.05

Step 1: Identify the test type

This is a right-tailed test, so p-value = P(T ≥ 1.85)

Step 2: Find the p-value using t-distribution

With df = 24, P(T ≥ 1.85) ≈ 0.038

(Using t-table or calculator)

Step 3: Compare to significance level

Since p-value (0.038) < α (0.05), reject H₀

Step 4: Interpret the result

There is sufficient evidence to support the alternative hypothesis

Final Answer:

P-value ≈ 0.038

Decision: Reject H₀ (statistically significant)

Applied Rules:

• Right-tailed Formula: p-value = P(T ≥ observed value)

• Degrees of Freedom: Required for t-distribution

• Decision Rule: Compare p-value to α threshold

3 Left-tailed Test
Exercise 3
A z-test gives z = -1.64. Find the p-value for a left-tailed test and determine significance at α = 0.01.
Problem Setup:

Given: z = -1.64, left-tailed test, α = 0.01

Step 1: Identify the test type

This is a left-tailed test, so p-value = P(Z ≤ -1.64)

Step 2: Find the p-value using standard normal

P(Z ≤ -1.64) = 0.0505

(Using standard normal table)

Step 3: Compare to significance level

Since p-value (0.0505) > α (0.01), fail to reject H₀

Step 4: Interpret the result

There is insufficient evidence to support the alternative hypothesis at α = 0.01

Final Answer:

P-value = 0.0505

Decision: Fail to reject H₀ (not statistically significant at α = 0.01)

Applied Rules:

• Left-tailed Formula: p-value = P(Z ≤ observed value)

• Negative Z-value: Still follows same probability calculation

• Strict Alpha: α = 0.01 requires stronger evidence

Solution: Exercises 4 to 5
4 Comparing Significance Levels
Exercise 4
A study reports a p-value of 0.035. Determine the decision at α = 0.01, α = 0.05, and α = 0.10. Explain the implications of different significance levels.
Problem Setup:

Given: p-value = 0.035, multiple α levels to evaluate

Step 1: Evaluate at α = 0.01

Since 0.035 > 0.01, fail to reject H₀

Conclusion: Not significant at 1% level

Step 2: Evaluate at α = 0.05

Since 0.035 ≤ 0.05, reject H₀

Conclusion: Significant at 5% level

Step 3: Evaluate at α = 0.10

Since 0.035 ≤ 0.10, reject H₀

Conclusion: Significant at 10% level

Step 4: Analyze implications

α = 0.01: Most conservative, requires strongest evidence

α = 0.05: Standard threshold, balance between Type I and II errors

α = 0.10: More liberal, willing to accept higher risk of Type I error

Final Answer:

α = 0.01: Fail to reject H₀ (not significant)

α = 0.05: Reject H₀ (significant)

α = 0.10: Reject H₀ (significant)

Applied Rules:

• Alpha Sensitivity: Same p-value leads to different decisions

• Conservative vs Liberal: Lower α requires stronger evidence

• Context Matters: Choose α based on consequences of errors

5 Practical Significance vs Statistical Significance
Exercise 5
A large study (n = 5000) finds a mean difference of 0.001 units with p-value = 0.02. Another small study (n = 25) finds a mean difference of 1.5 units with p-value = 0.08. Discuss the relationship between statistical and practical significance.
Problem Setup:

Study 1: n = 5000, difference = 0.001, p = 0.02

Study 2: n = 25, difference = 1.5, p = 0.08

Step 1: Analyze statistical significance

Study 1: p = 0.02 < 0.05 → Statistically significant

Study 2: p = 0.08 > 0.05 → Not statistically significant

Step 2: Analyze practical significance

Study 1: 0.001 units may be too small to matter practically

Study 2: 1.5 units is likely practically important

Step 3: Discuss sample size effects

Larger samples can detect tiny differences as significant

Smaller samples may miss practically important differences

Step 4: Draw conclusions

Statistical significance ≠ Practical significance

Always consider effect size and context

Final Answer:

Study 1: Statistically significant but likely not practically significant

Study 2: Not statistically significant but practically significant

Both statistical and practical significance matter for interpretation

Applied Rules:

• Sample Size Impact: Large samples increase statistical power

• Effect Size Importance: Consider magnitude, not just significance

• Contextual Interpretation: Statistical results must make practical sense

Comprehensive P-value and Significance Guide
p-value = P(observed data | H₀ true)
P-value Formula
Two-tailed
p = 2 × P(Z ≥ |z|)
Tests for any difference
Right-tailed
p = P(Z ≥ z)
Tests for increase
Left-tailed
p = P(Z ≤ z)
Tests for decrease
Key Significance Concepts:

Type I Error: Rejecting H₀ when it's true (false positive), probability = α

Type II Error: Failing to reject H₀ when it's false (false negative), probability = β

Power: Probability of correctly rejecting a false H₀ = 1 - β

Effect Size: Magnitude of difference or relationship being tested

5-Step P-value Decision Process:
  1. Calculate test statistic: z, t, or other appropriate statistic
  2. Determine test type: Two-tailed, right-tailed, or left-tailed
  3. Find p-value: Using appropriate distribution tables
  4. Compare to α: p-value vs significance level
  5. Make decision: Reject or fail to reject H₀
Memory Tip: "If p is low, H₀ must go" - Low p-value rejects null
Memory Tip: P-value is conditional probability given H₀ is true
Memory Tip: Statistical significance doesn't guarantee practical importance
Common Misconceptions: P-value ≠ probability H₀ is true, significance ≠ importance
Best Practices: Report p-value, effect size, and confidence intervals together

Questions & Answers

Question: What does it really mean when we say "p-value = 0.03"? I've heard people say it means "there's a 3% chance that H₀ is true" - is that correct?

Answer: That's a very common misconception! A p-value of 0.03 does NOT mean "there's a 3% chance that H₀ is true." Here's the correct interpretation:

Correct meaning: If H₀ were true, there would be a 3% chance of observing data as extreme or more extreme than what we actually observed.

The p-value is a conditional probability: P(data | H₀ is true), not P(H₀ is true | data).

Think of it this way: Imagine H₀ is true and we repeatedly collect samples. Only 3% of those samples would produce results as extreme as ours just by random chance. This suggests our observed result is unusual under H₀, providing evidence against it.

The p-value tells us about the compatibility of our data with H₀, not about the probability that H₀ is correct.

Question: Why do we sometimes see p-values reported as "p < 0.001" instead of exact values? And why is 0.05 such a common threshold?

Answer: Great questions about reporting conventions and historical choices:

About "p < 0.001": When p-values are extremely small, we report them as inequalities because:

  • The exact value becomes less meaningful due to computational precision limits
  • The focus is on the strength of evidence rather than the precise probability
  • It emphasizes that the result is highly significant

About α = 0.05: This threshold was popularized by statistician Ronald Fisher in the early 20th century. He suggested that a 1 in 20 (5%) chance of observing results by random variation was a reasonable threshold for considering results "significant enough" to investigate further.

It became the convention partly because it represents a reasonable balance between Type I and Type II errors in many contexts. However, the choice should depend on the context and consequences of errors.

Question: I'm still confused about the difference between statistical significance and practical significance. Can you explain with a real-world example?

Answer: This is a crucial distinction! Here's a clear example:

Medical Drug Example:

  • Statistical Significance: A clinical trial with 10,000 participants finds a new drug reduces blood pressure by 0.1 mmHg compared to placebo, with p = 0.001
  • Practical Significance: A 0.1 mmHg reduction is clinically meaningless - doctors wouldn't change treatment for such a tiny difference

Another Example: A study of 100 people shows a new exercise program increases strength by 15 lbs with p = 0.08 (not statistically significant at α = 0.05), but this could be very meaningful in practice.

Statistical significance depends on sample size and variability, while practical significance depends on the real-world impact. Always consider both the p-value AND the effect size when interpreting results.

The goal is to find results that are both statistically significant and practically meaningful.