Solved Exercises on Standard Error and Bias-Variance Tradeoff in Grade 12

Master standard error and bias-variance tradeoff: sampling distributions, estimation accuracy, model selection, and statistical inference through these 5 detailed exercises.

Solution: Exercises 1 to 3
1 Standard Error Calculation
Exercise 1
A population has standard deviation σ = 12. Calculate the standard error of the sample mean for sample sizes n = 16, n = 36, and n = 144. Explain how sample size affects precision.
Definition:

Standard Error: The standard deviation of the sampling distribution of a statistic, typically the sample mean. Formula: SE = σ/√n

Method:
  1. Identify the population standard deviation σ
  2. Identify the sample sizes n
  3. Apply the standard error formula: SE = σ/√n
  4. Compare standard errors to assess precision
Given
σ = 12
n=16
SE = 3
n=36
SE = 2
n=144
SE = 1
Step 1: Calculate standard error for n = 16

SE₁₆ = σ/√n = 12/√16 = 12/4 = 3

Step 2: Calculate standard error for n = 36

SE₃₆ = σ/√n = 12/√36 = 12/6 = 2

Step 3: Calculate standard error for n = 144

SE₁₄₄ = σ/√n = 12/√144 = 12/12 = 1

Step 4: Analyze the relationship

As sample size increases, standard error decreases, indicating improved precision

SE₁₆ = 3, SE₃₆ = 2, SE₁₄₄ = 1
Final answer:

Standard errors are: n=16 → SE=3, n=36 → SE=2, n=144 → SE=1. Larger samples provide more precise estimates.

Applied rules:

• Standard error formula: SE = σ/√n shows inverse relationship with √n

• Precision improvement: Quadrupling sample size halves the standard error

• Law of large numbers: Larger samples yield more stable estimates

Key notes:

• The relationship between sample size and precision follows the square root law

• To reduce standard error by 50%, you need to quadruple the sample size

• This demonstrates why larger samples are preferred in statistical studies

2 Bias and Variance Analysis
Exercise 2
Two estimators θ₁ and θ₂ are used to estimate parameter θ = 10. Estimator θ₁ has bias = 0.5 and variance = 2. Estimator θ₂ has bias = 0 and variance = 4. Calculate the Mean Squared Error (MSE) for each estimator and determine which is better.
Definition:

Bias: E[θ̂] - θ (expected value minus true parameter). Variance: E[(θ̂ - E[θ̂])²]. MSE: E[(θ̂ - θ)²] = Bias² + Variance

Estimator 1
Bias=0.5, Var=2
Estimator 2
Bias=0, Var=4
MSE
MSE₁=2.25, MSE₂=4
Step 1: Apply MSE decomposition formula

MSE = Bias² + Variance for any estimator

Step 2: Calculate MSE for estimator θ₁

MSE₁ = (Bias₁)² + Var₁ = (0.5)² + 2 = 0.25 + 2 = 2.25

Step 3: Calculate MSE for estimator θ₂

MSE₂ = (Bias₂)² + Var₂ = (0)² + 4 = 0 + 4 = 4

Step 4: Compare and conclude

Since MSE₁ < MSE₂, estimator θ₁ is better despite having some bias

MSE₁ = 2.25, MSE₂ = 4, θ₁ is better
Final answer:

Estimator θ₁ has MSE = 2.25, estimator θ₂ has MSE = 4. Despite its bias, θ₁ is the better estimator because it has lower MSE.

Applied rules:

• MSE decomposition: MSE = Bias² + Variance

• Bias-variance tradeoff: Sometimes biased estimators have lower overall error

• Optimality criterion: Lower MSE indicates better estimator performance

Key notes:

• An unbiased estimator (zero bias) isn't necessarily the best if it has high variance

• The bias-variance tradeoff is fundamental in statistical modeling

• Sometimes accepting a small amount of bias leads to significant variance reduction

3 Bias-Variance Tradeoff Example
Exercise 3
Consider a dataset where we want to predict Y from X. Model A is a simple linear regression (low complexity) with bias = 0.8 and variance = 0.2. Model B is a complex polynomial regression (high complexity) with bias = 0.1 and variance = 1.5. Which model has lower prediction error?
Definition:

Bias-Variance Tradeoff: In predictive modeling, simpler models tend to have higher bias but lower variance, while complex models have lower bias but higher variance

Model A
Bias=0.8, Var=0.2
Model B
Bias=0.1, Var=1.5
MSE
MSE_A=0.84, MSE_B=1.51
Step 1: Understand the bias-variance decomposition

Expected prediction error = Bias² + Variance + Irreducible error

Step 2: Calculate MSE for Model A

MSE_A = (Bias_A)² + Var_A = (0.8)² + 0.2 = 0.64 + 0.2 = 0.84

Step 3: Calculate MSE for Model B

MSE_B = (Bias_B)² + Var_B = (0.1)² + 1.5 = 0.01 + 1.5 = 1.51

Step 4: Make decision based on MSE

Model A has lower MSE (0.84 < 1.51), so it provides better predictions despite higher bias

MSE_A = 0.84, MSE_B = 1.51, Model A is better
Final answer:

Model A has MSE = 0.84 and Model B has MSE = 1.51. Model A is better for prediction despite its higher bias because the variance reduction compensates for the increased bias.

Applied rules:

• Tradeoff principle: Balance bias and variance to minimize total error

• Model selection: Consider both bias and variance, not just one component

• Overfitting prevention: Complex models may have low bias but high variance

Key notes:

• Simple models often generalize better to new data

• Complex models may fit training data well but perform poorly on new data

• The optimal model balances bias and variance to achieve lowest prediction error

Standard Error and Bias-Variance Rules and Methods
\( SE = \frac{\sigma}{\sqrt{n}} \)
Standard Error Formula
\( MSE = Bias^2 + Variance \)
MSE Decomposition
\( Bias = E[\hat{\theta}] - \theta \)
Bias Formula
Property
SE ↓ as n ↑
Standard Error Decreases with Sample Size
Tradeoff
Low Bias ↔ High Variance
Complex Models Relationship
Optimality
Minimize MSE
Best Estimator Criterion
Key definitions:

Standard Error: Measure of the variability of a sample statistic across different samples

Bias: Systematic deviation of expected estimator value from true parameter value

Variance: Random fluctuation of estimator around its expected value

MSE: Mean Squared Error combines both bias and variance

Complete methodology:
  1. Identify parameters: σ (population SD), n (sample size), or estimator properties
  2. Calculate standard error: SE = σ/√n for sample means
  3. Assess bias and variance: Determine E[θ̂] and Var(θ̂)
  4. Compute MSE: Use MSE = Bias² + Variance
  5. Make comparisons: Choose estimator/model with lowest MSE
Tip 1: Standard error measures precision of estimates; smaller SE means more precise estimates.
Tip 2: Unbiased estimators aren't always optimal; consider both bias and variance.
Tip 3: Simple models often generalize better due to lower variance.
Tip 4: Always evaluate both bias and variance when comparing estimators.
Key insight: The bias-variance tradeoff is fundamental to statistical modeling and machine learning.
Practical application: Used in model selection, regularization, and cross-validation.
Key Properties:

• Standard error: SE = σ/√n decreases as sample size increases

• MSE decomposition: MSE = Bias² + Variance

• Bias-variance tradeoff: Complex models have low bias, high variance

• Optimal estimator: Minimizes MSE, balancing bias and variance

Solution: Exercises 4 to 5
4 Sample Size Planning
Exercise 4
A researcher wants to estimate a population mean with standard deviation σ = 15. What sample size is needed to achieve a standard error of at most 2? How does this relate to the desired precision?
Definition:

Sample size planning: Determining n to achieve a desired level of precision in estimation

Given
σ = 15, SE ≤ 2
Formula
n ≥ (σ/SE)²
Calculation
n ≥ 56.25
Step 1: Start with standard error formula

SE = σ/√n

Step 2: Solve for minimum sample size

We want SE ≤ 2, so σ/√n ≤ 2

Therefore, √n ≥ σ/2, which gives n ≥ (σ/2)²

Step 3: Substitute known values

n ≥ (15/2)² = (7.5)² = 56.25

Step 4: Round up to nearest integer

Since n must be an integer, we need n ≥ 57

n ≥ 57 to achieve SE ≤ 2
Final answer:

A sample size of at least 57 is needed to achieve a standard error of at most 2. This ensures the desired level of precision in estimation.

Applied rules:

• Sample size formula: n ≥ (σ/SE)² for desired standard error

• Precision requirement: Smaller standard error requires larger sample size

• Rounding rule: Always round up to ensure requirements are met

Key notes:

• The relationship between precision and sample size is quadratic

• Halving the standard error requires quadrupling the sample size

• Researchers must balance precision needs with resource constraints

5 Estimator Comparison
Exercise 5
Three estimators are available for parameter θ: θ₁ (bias=0.3, var=1.2), θ₂ (bias=0.1, var=2.0), θ₃ (bias=0.5, var=0.8). Which estimator is best according to MSE? Discuss the bias-variance tradeoff implications.
Definition:

Estimator comparison: Using MSE as a comprehensive measure of estimator quality that considers both bias and variance

Estimator 1
MSE₁ = 1.29
Estimator 2
MSE₂ = 2.01
Estimator 3
MSE₃ = 1.05
Step 1: Calculate MSE for estimator θ₁

MSE₁ = (0.3)² + 1.2 = 0.09 + 1.2 = 1.29

Step 2: Calculate MSE for estimator θ₂

MSE₂ = (0.1)² + 2.0 = 0.01 + 2.0 = 2.01

Step 3: Calculate MSE for estimator θ₃

MSE₃ = (0.5)² + 0.8 = 0.25 + 0.8 = 1.05

Step 4: Compare and select best estimator

MSE₃ = 1.05 < MSE₁ = 1.29 < MSE₂ = 2.01, so θ₃ is best

Step 5: Analyze bias-variance tradeoff

θ₃ has moderate bias but low variance, achieving the best overall balance

MSE₁ = 1.29, MSE₂ = 2.01, MSE₃ = 1.05, θ₃ is best
Final answer:

Estimator θ₃ is best with MSE = 1.05. This demonstrates the bias-variance tradeoff: θ₃ has higher bias than θ₂ but much lower variance, resulting in lower overall MSE.

Applied rules:

• Comprehensive evaluation: MSE captures both bias and variance simultaneously

• Tradeoff principle: Sometimes accepting more bias leads to better overall performance

• Optimization: Best estimator minimizes MSE, not just bias or variance separately

Key notes:

• Estimator θ₂ had the lowest bias but highest variance, leading to poor overall performance

• Estimator θ₃ achieved the optimal balance between bias and variance

• The bias-variance tradeoff is crucial for selecting effective statistical procedures

Standard Error and Bias-Variance Tradeoff Summary

Comprehensive Summary: Standard Error and Bias-Variance Tradeoff

Key Definitions:

Standard Error (SE): The standard deviation of the sampling distribution of a statistic, measuring the precision of an estimator. For the sample mean: SE = σ/√n

Bias: The difference between the expected value of an estimator and the true parameter value: Bias = E[θ̂] - θ. An estimator is unbiased if bias = 0.

Variance: The variability of an estimator around its expected value: Var(θ̂) = E[(θ̂ - E[θ̂])²]

Mean Squared Error (MSE): A comprehensive measure of estimator quality: MSE = E[(θ̂ - θ)²] = Bias² + Variance

Core Principles and Rules:
  1. Standard Error Formula: SE = σ/√n, showing that precision improves with larger samples
  2. MSE Decomposition: MSE = Bias² + Variance, revealing the fundamental components of estimation error
  3. Bias-Variance Tradeoff: There's an inverse relationship between bias and variance in many estimation scenarios
  4. Optimality Criterion: The best estimator minimizes MSE, balancing both bias and variance
  5. Sample Size Effect: Increasing sample size reduces standard error proportionally to √n
Simple Example: Sample Mean

Population: μ = 100, σ = 20

Sample size n = 25: SE = 20/√25 = 4

Sample size n = 100: SE = 20/√100 = 2

Conclusion: Quadrupling sample size halves the standard error, doubling precision.

Advanced Example: Model Selection

Simple Model: Bias = 0.5, Variance = 0.3 → MSE = 0.25 + 0.3 = 0.55

Complex Model: Bias = 0.1, Variance = 1.2 → MSE = 0.01 + 1.2 = 1.21

Despite higher bias, the simple model has lower MSE and is therefore preferred.

Tip 1: Standard error measures how much sample statistics vary from sample to sample. Smaller SE means more consistent estimates.
Tip 2: Unbiased doesn't mean optimal. A slightly biased estimator with low variance can outperform an unbiased estimator with high variance.
Tip 3: The bias-variance tradeoff appears everywhere in statistics: from choosing sample sizes to selecting models to regularization techniques.
Tip 4: Remember the quadratic relationship: to halve standard error, you need to quadruple sample size.
Common Pitfall: Focusing only on bias when evaluating estimators. Both bias and variance contribute to total error.
Common Mistake: Assuming that unbiased estimators are always better. Sometimes accepting small bias leads to significant variance reduction.
Key Takeaway: The bias-variance tradeoff is fundamental to statistical inference and guides optimal decision-making in estimation and prediction.
Essential Formulas:

• Standard Error: SE = σ/√n

• Bias: Bias = E[θ̂] - θ

• Variance: Var(θ̂) = E[(θ̂ - E[θ̂])²]

• MSE Decomposition: MSE = Bias² + Variance

• Required Sample Size: n ≥ (σ/SE)²

Memory Techniques:

  • SE ∝ 1/√n: "Standard Error is Inversely Proportional to Square Root of n"
  • MSE = B² + V: "Mean Squared Error Equals Bias Squared Plus Variance"
  • Tradeoff: "Simple models: High Bias, Low Variance; Complex models: Low Bias, High Variance"
  • Optimization: "Always minimize MSE, not just bias or variance alone"

Questions & Answers

Question: I'm confused about the difference between bias and variance. Can you explain this with a practical example?

Answer: Think of shooting arrows at a target:

  • Bias: On average, are your arrows centered on the bullseye? If they're consistently off to the left, you have high bias.
  • Variance: Are your arrows clustered together or scattered widely? High variance means wide scatter.

An archer with high bias but low variance shoots arrows that are consistently off-target but grouped together. An archer with low bias but high variance shoots arrows that average around the target but are widely scattered. The best archer has both low bias (accurate) and low variance (precise).

In statistics: an estimator with high bias systematically misses the true value, while an estimator with high variance gives inconsistent results across different samples.

Question: Why would anyone want a biased estimator? Isn't unbiased always better?

Answer: That's a common misconception! Unbiased isn't always better. Consider this example:

Suppose we have two estimators for a parameter θ = 5:

  • Estimator A: Unbiased (E[A] = 5) but high variance (Var(A) = 10)
  • Estimator B: Biased (E[B] = 6) but low variance (Var(B) = 1)

MSE(A) = 0² + 10 = 10

MSE(B) = 1² + 1 = 2

Even though B is biased, it has lower MSE and is therefore better overall. The bias-variance tradeoff means we should optimize total error (MSE), not just bias.

This principle is used in regularization techniques like Ridge Regression, where we accept some bias to dramatically reduce variance.

Question: How does the bias-variance tradeoff apply to real-world situations beyond statistics?

Answer: The bias-variance tradeoff is everywhere! Here are some real-world examples:

  • Education: A simple study method (high bias, low variance) might consistently yield average grades, while cramming (low bias potential, high variance) could lead to very high or very low grades.
  • Investment: Conservative investments (high bias toward lower returns, low variance) vs. aggressive stocks (potential for high returns, high variance).
  • Medicine: A simple diagnostic rule (high bias, low variance) vs. a complex algorithm (low bias, high variance) for predicting disease.

In machine learning, this tradeoff determines whether to use simple linear models or complex neural networks. The optimal choice depends on the specific context and data availability.

Understanding this tradeoff helps make better decisions in any situation involving prediction or estimation.

Question: How does increasing sample size affect bias and variance?

Answer: Increasing sample size has different effects on bias and variance:

Effect on Bias: Generally, sample size does NOT affect bias. If an estimator is biased with n=10, it remains biased with n=1000. Bias is a property of the estimator itself, not the sample size.

Effect on Variance: Increasing sample size REDUCES variance. Specifically, the variance of sample means decreases as 1/n. This is why larger samples give more consistent results.

For the sample mean specifically:

  • Bias of sample mean: Always 0 (unbiased estimator), regardless of n
  • Variance of sample mean: σ²/n, which decreases as n increases
  • Standard error of sample mean: σ/√n, which decreases as √n increases

So larger samples improve precision (reduce variance) but don't fix systematic bias in the estimation procedure.

Question: How do I decide between two estimators when one has lower bias and the other has lower variance?

Answer: This is exactly where the MSE (Mean Squared Error) comes in! Never compare bias and variance separately - always combine them using the MSE formula:

MSE = Bias² + Variance

The estimator with the lower MSE is better overall. Let me illustrate:

Estimator A: Bias = 0.1, Variance = 2.0 → MSE = (0.1)² + 2.0 = 0.01 + 2.0 = 2.01

Estimator B: Bias = 0.5, Variance = 0.8 → MSE = (0.5)² + 0.8 = 0.25 + 0.8 = 1.05

Even though A has lower bias, B is better overall because its lower variance more than compensates for higher bias.

Always use MSE to make optimal decisions when comparing estimators!