Solved Exercises on Bias in Sampling in Grade 10 Statistics

Master bias in sampling: selection bias, response bias, non-response bias, undercoverage, and voluntary response bias through these 5 detailed exercises with comprehensive explanations.

Solution: Exercises 1 to 3
1 Selection Bias
Exercise 1
A political pollster surveys voters by calling landline phones during business hours. What type of bias is present, and how does it affect the results? Propose a solution to minimize this bias.
Definition:

Selection Bias: Occurs when the sample systematically excludes certain groups from the population, leading to a sample that is not representative of the target population.

Identification Process:
  1. Identify the target population (all eligible voters)
  2. Determine who could potentially be sampled (landline phone users during business hours)
  3. Compare the two to identify excluded groups
  4. Assess how exclusions might affect the results
Step 1: Identify the target population

All eligible voters in the area being surveyed

Step 2: Determine sampling method limitations

Only landline phone users available during business hours (9 AM - 5 PM)

Step 3: Identify excluded groups

Young people (more likely to have only cell phones), workers (not home during business hours), low-income households (may not have landlines)

Step 4: Assess impact on results

Results may overrepresent older, higher-income, stay-at-home voters

Solution:

This is selection bias due to undercoverage. Include cell phone users and conduct surveys at various times of day to reach a more representative sample.

Types of Selection Bias:

• Undercoverage: Systematically excluding part of the population

• Voluntary response bias: People choose to participate

• Convenience sampling: Easiest to reach people

⚠️ Bias Impact
This sampling method would likely overestimate support for candidates favored by older, higher-income voters and underestimate support among younger demographics.
Tip: Always consider who might be systematically excluded from your sample.
Key Note: Selection bias leads to systematic errors that cannot be reduced by increasing sample size.
2 Response Bias
Exercise 2
A survey asks, "Do you agree that our excellent mayor has done a wonderful job?" What type of bias is present? How would you reword the question to reduce bias? Explain the psychological factors involved.
Definition:

Response Bias: Occurs when respondents systematically answer questions differently than their true beliefs or feelings, often due to question wording, social desirability, or interviewer influence.

Step 1: Identify problematic wording

Words like "excellent" and "wonderful" are loaded and lead respondents toward positive responses

Step 2: Recognize psychological factors

Respondents may feel pressured to agree with positive descriptors due to social desirability

Step 3: Analyze the bias mechanism

The question assumes the mayor is excellent and wonderful, pressuring agreement

Step 4: Propose unbiased alternatives

Reword to be neutral and allow for all responses

Original: "Do you agree that our excellent mayor has done a wonderful job?"
Better: "How would you rate the mayor's job performance?"
Best: "Rate the mayor's job performance from 1 (poor) to 5 (excellent)"
Final answer:

This is response bias caused by leading question wording. The words "excellent" and "wonderful" prime respondents to give positive answers. Better alternatives: "How would you rate the mayor's job performance?" or "Rate the mayor's job performance from 1-5."

Causes of Response Bias:

• Leading questions: Words that suggest desired answer

• Order effects: Question order influences responses

• Wording effects: Complex or emotionally charged language

• Interviewer effects: Characteristics influencing responses

🧠 Psychological Factors
People tend to agree with positive statements (acquiescence bias), want to appear socially acceptable (social desirability bias), and may be influenced by authority figures or perceived expectations.
3 Non-response Bias
Exercise 3
A mail survey about workplace satisfaction has a 25% response rate. Of those who responded, 80% reported high satisfaction. What concerns should researchers have about these results? How might non-response bias affect the findings?
Definition:

Non-response Bias: Occurs when individuals who do not respond to a survey differ systematically from those who do respond, leading to unrepresentative results.

Step 1: Analyze the response rate

Only 25% responded, meaning 75% did not participate in the survey

Step 2: Consider who might not respond

Unhappy employees might avoid responding to avoid attention or retaliation

Step 3: Assess the potential bias

If dissatisfied employees are less likely to respond, the actual satisfaction rate is likely lower than 80%

Step 4: Calculate potential impact

True satisfaction rate could be significantly lower than reported 80%

Reported: 80% satisfied (from 25% responders)
Actual: Likely lower (if non-responders were less satisfied)
Potential range: 40-70% depending on non-responder satisfaction
Final answer:

Non-response bias is a major concern. The low 25% response rate suggests the results may not be representative. If dissatisfied employees were less likely to respond, the true satisfaction rate is likely much lower than the reported 80%.

Factors Contributing to Non-response Bias:
Factor Effect on Response Impact on Results
Topic sensitivity Lower response Biased toward less sensitive views
Survey length Lower response Biased toward more committed participants
Time demands Lower response Biased toward available participants
📊 Impact Assessment
With only 25% response rate, if non-responders were significantly less satisfied, the true satisfaction rate could be 40-60% instead of the reported 80%. This represents a substantial bias in the findings.
Bias in Sampling: Overview
\(\text{Bias} = \text{Expected Value of Statistic} - \text{True Parameter Value}\)
Bias Formula
🎯 Selection Bias
Systematic exclusion of part of the population during sampling process.
Example: Surveying only college students about voting preferences excludes younger non-students and older adults.
Prevention: Use random sampling methods that give everyone in population a chance to be selected.
💬 Response Bias
Systematic difference between responses given and true values due to question wording or context.
Example: Leading questions like "Don't you think the policy is beneficial?" encourage agreement.
Prevention: Use neutral, clear wording and avoid loaded terms or assumptions.
📞 Non-response Bias
Difference between responders and non-responders leads to unrepresentative sample.
Example: Only satisfied customers return product surveys, inflating satisfaction ratings.
Prevention: Follow up with non-responders, offer incentives, keep surveys brief.
📋 Undercoverage Bias
Sampling frame doesn't include entire target population.
Example: Using telephone directory to sample households excludes those without listed numbers.
Prevention: Ensure sampling frame includes entire target population.
Key Principle: Bias creates systematic errors that don't decrease with larger sample sizes.
Detection Challenge: Bias is often difficult to detect after data collection.
Prevention Focus: Design phase is crucial for minimizing bias.
Solution: Exercises 4 to 5
4 Undercoverage Bias
Exercise 4
A health survey uses a database of registered voters to select participants. What population is being studied? What groups are systematically excluded? How might this affect the results regarding health behaviors?
Definition:

Undercoverage Bias: Occurs when the sampling frame (list of potential participants) does not include all members of the target population, systematically excluding certain groups.

Step 1: Identify the target population

Target: All residents in the geographic area (general population health behaviors)

Step 2: Identify the sampling frame

Sampling frame: Registered voters database

Step 3: Determine excluded groups

Excluded: Non-citizens, under-18 residents, unregistered citizens, recently moved residents

Step 4: Assess potential impact on health behaviors

Excluded groups may have different health behaviors, leading to biased results

Target: All residents
Sample: Registered voters
Excluded: Non-citizens, youth, recent movers
Impact: May miss vulnerable populations with different health needs
Final answer:

This is undercoverage bias. The target population is all residents, but only registered voters are included. Excluded groups include non-citizens, young adults under 18, and recently moved residents. These groups may have different health behaviors and needs, leading to biased results that underrepresent vulnerable populations.

Common Undercoverage Examples:

• Phone surveys: Exclude those without phones

• Online surveys: Exclude those without internet access

• Mail surveys: Exclude those with unstable housing

• Voter databases: Exclude non-citizens and non-registered voters

🏥 Health Implications
Using voter registration as a sampling frame for health surveys could systematically exclude vulnerable populations who may have different health risks, access barriers, and health outcomes. This could lead to inadequate resource allocation for underserved communities.
5 Voluntary Response Bias
Exercise 5
A radio station asks listeners to call in to vote on whether taxes should be increased. Of 1,000 callers, 75% oppose the tax increase. Evaluate the validity of this result. What type of bias is present and how does it affect the interpretation?
Definition:

Voluntary Response Bias: Occurs when people choose themselves to participate in a study, typically because they have strong opinions about the topic, leading to unrepresentative results.

Step 1: Identify the sampling method

This is voluntary response sampling where participants self-select

Step 2: Analyze who is likely to participate

People with strong opinions (especially opposition) are more likely to call in

Step 3: Assess representativeness

The sample consists only of motivated callers, not representative of all listeners

Step 4: Evaluate validity

Results are highly suspect due to systematic bias in participant selection

Method: Voluntary response
Participants: Motivated callers
Bias: Strong opinion holders
Validity: Very low
Final answer:

This result is not valid due to voluntary response bias. Only motivated listeners chose to call in, and those with strong opinions (especially opposition to taxes) are more likely to participate. The true public opinion could be significantly different from the 75% opposition observed.

Critical Issues with Voluntary Response:

• No randomization: Participants self-select

• Extreme views: Strong opinions overrepresented

• Non-representative: Not reflective of target population

• Invalid inference: Cannot generalize to broader population

Tip: Never trust results from voluntary response surveys for scientific conclusions.
Key Note: Voluntary response bias makes results completely unreliable for population inference.
Warning: Media polls using voluntary response are entertainment, not science.
Comprehensive Summary: Bias in Sampling
\(\text{Total Survey Error} = \text{Sampling Error} + \text{Non-sampling Error}\)
Total Survey Error Framework
📚 Comprehensive Summary: Bias in Sampling

🔍 Definitions of Key Concepts

Bias: Systematic error in a statistic that causes it to consistently overestimate or underestimate the true parameter value. Unlike random error, bias does not decrease with larger sample sizes.
Sampling Error: Natural variability that occurs because we sample only part of the population. This decreases as sample size increases.
Non-sampling Error: Errors that occur during data collection, processing, or analysis, including all forms of bias, measurement errors, and processing mistakes.

⚖️ Core Rules and Principles

Rule 1 - Bias vs. Variability: Bias affects the center of the sampling distribution (accuracy), while sample size affects the spread (precision). Increasing sample size reduces variability but does NOT reduce bias.
Rule 2 - Prevention vs. Correction: Bias is nearly impossible to detect or correct after data collection. Prevention through careful design is essential.
Rule 3 - Multiple Biases: Studies can suffer from multiple biases simultaneously, compounding the problem.

⚙️ Step-by-Step Methods for Bias Detection

  1. Define the target population: Clearly specify who you want to study
  2. Examine the sampling frame: List of potential participants
  3. Compare frame to population: Identify who is systematically excluded
  4. Analyze response patterns: Look for differences between responders and non-responders
  5. Review question wording: Check for leading or loaded language
  6. Assess data collection method: Consider how method might influence responses

🧪 Examples: From Simple to Advanced

Simple Example: Asking "Do you like ice cream?" - seems unbiased but may have response bias if asked by an ice cream vendor.
Intermediate Example: Surveying shopping mall visitors about spending habits - undercoverage bias excludes those who don't visit malls.
Advanced Example: Political polling that combines landline and cell phone surveys but fails to weight properly for demographic differences - multiple biases combined.

💡 Tips, Tricks, and Common Pitfalls

Tip 1: Always pilot test your survey with a small group to identify potential biases before full deployment.
Tip 2: Use multiple modes of contact (phone, mail, online) to reduce non-response bias.
Common Pitfall: Assuming that a large sample size automatically makes results valid - large biased samples are still biased.
Common Pitfall: Confusing correlation with causation in observational studies affected by bias.

📝 Key Notes for Memorization

Memory Device: "BURN" - Bias Undermines Representative Numbers (selection, undercoverage, response, non-response).
Key Equation: Bias = Expected Value of Statistic - True Parameter Value
Golden Rule: A representative sample of 1,000 is better than a biased sample of 10,000.
Design Priority: Eliminate bias first, then worry about sample size for precision.

Questions & Answers

Question: I'm confused about the difference between selection bias and undercoverage bias. Aren't they the same thing?

Answer: Great question! Undercoverage bias is actually a specific type of selection bias:

  • Selection bias: A broad category that includes any systematic exclusion of parts of the population during the sampling process.
  • Undercoverage bias: A specific form of selection bias that occurs when the sampling frame (the list of potential participants) doesn't include all members of the target population.

Analogy: Think of selection bias as the umbrella term (like "vehicle"), and undercoverage bias as a specific type (like "car"). Other types of selection bias include voluntary response bias and convenience sampling.

Example: Using a phone book to survey households is undercoverage bias (excludes unlisted numbers, cell-only households). Asking friends to participate is convenience sampling, another type of selection bias.

Question: How can I tell if a survey has bias just by reading about it? Are there warning signs?

Answer: Yes, there are several red flags that indicate potential bias:

  • Voluntary response: "Call our number," "Click here to vote," or "Tell us your opinion"
  • Convenience sampling: Surveying only at malls, universities, or other specific locations
  • Leading questions: Questions that assume facts or use emotional language
  • Low response rates: Less than 30% response rates suggest non-response bias
  • Unclear sampling method: Vague descriptions like "randomly selected" without details

Red flag example: "A survey of 1,000 people found that 90% support the new policy." Without knowing how they were selected or contacted, this result is suspect.

Good sign: Surveys that clearly describe their sampling method, response rates, and acknowledge limitations.

Question: Can bias ever be completely eliminated? Is it possible to have a perfectly unbiased study?

Answer: In practice, completely eliminating all bias is nearly impossible, but we can minimize it significantly:

  • Perfect randomization: Requires complete sampling frames and cooperation from all selected participants
  • Resource constraints: Time and money limit our ability to achieve perfect conditions
  • Human factors: Interviewer effects and respondent behavior introduce subtle biases

Reality: The goal is to minimize bias to levels where it doesn't meaningfully affect conclusions, rather than achieving theoretical perfection.

Best practices: Use random sampling, multiple contact attempts, neutral question wording, and acknowledge limitations in reporting.

Key insight: A well-designed study with acknowledged limitations is better than a poorly designed study claiming perfection.