Selection Bias: Occurs when the sample systematically excludes certain groups from the population, leading to a sample that is not representative of the target population.
- Identify the target population (all eligible voters)
- Determine who could potentially be sampled (landline phone users during business hours)
- Compare the two to identify excluded groups
- Assess how exclusions might affect the results
All eligible voters in the area being surveyed
Only landline phone users available during business hours (9 AM - 5 PM)
Young people (more likely to have only cell phones), workers (not home during business hours), low-income households (may not have landlines)
Results may overrepresent older, higher-income, stay-at-home voters
This is selection bias due to undercoverage. Include cell phone users and conduct surveys at various times of day to reach a more representative sample.
• Undercoverage: Systematically excluding part of the population
• Voluntary response bias: People choose to participate
• Convenience sampling: Easiest to reach people
Response Bias: Occurs when respondents systematically answer questions differently than their true beliefs or feelings, often due to question wording, social desirability, or interviewer influence.
Words like "excellent" and "wonderful" are loaded and lead respondents toward positive responses
Respondents may feel pressured to agree with positive descriptors due to social desirability
The question assumes the mayor is excellent and wonderful, pressuring agreement
Reword to be neutral and allow for all responses
Better: "How would you rate the mayor's job performance?"
Best: "Rate the mayor's job performance from 1 (poor) to 5 (excellent)"
This is response bias caused by leading question wording. The words "excellent" and "wonderful" prime respondents to give positive answers. Better alternatives: "How would you rate the mayor's job performance?" or "Rate the mayor's job performance from 1-5."
• Leading questions: Words that suggest desired answer
• Order effects: Question order influences responses
• Wording effects: Complex or emotionally charged language
• Interviewer effects: Characteristics influencing responses
Non-response Bias: Occurs when individuals who do not respond to a survey differ systematically from those who do respond, leading to unrepresentative results.
Only 25% responded, meaning 75% did not participate in the survey
Unhappy employees might avoid responding to avoid attention or retaliation
If dissatisfied employees are less likely to respond, the actual satisfaction rate is likely lower than 80%
True satisfaction rate could be significantly lower than reported 80%
Actual: Likely lower (if non-responders were less satisfied)
Potential range: 40-70% depending on non-responder satisfaction
Non-response bias is a major concern. The low 25% response rate suggests the results may not be representative. If dissatisfied employees were less likely to respond, the true satisfaction rate is likely much lower than the reported 80%.
| Factor | Effect on Response | Impact on Results |
|---|---|---|
| Topic sensitivity | Lower response | Biased toward less sensitive views |
| Survey length | Lower response | Biased toward more committed participants |
| Time demands | Lower response | Biased toward available participants |
Undercoverage Bias: Occurs when the sampling frame (list of potential participants) does not include all members of the target population, systematically excluding certain groups.
Target: All residents in the geographic area (general population health behaviors)
Sampling frame: Registered voters database
Excluded: Non-citizens, under-18 residents, unregistered citizens, recently moved residents
Excluded groups may have different health behaviors, leading to biased results
Sample: Registered voters
Excluded: Non-citizens, youth, recent movers
Impact: May miss vulnerable populations with different health needs
This is undercoverage bias. The target population is all residents, but only registered voters are included. Excluded groups include non-citizens, young adults under 18, and recently moved residents. These groups may have different health behaviors and needs, leading to biased results that underrepresent vulnerable populations.
• Phone surveys: Exclude those without phones
• Online surveys: Exclude those without internet access
• Mail surveys: Exclude those with unstable housing
• Voter databases: Exclude non-citizens and non-registered voters
Voluntary Response Bias: Occurs when people choose themselves to participate in a study, typically because they have strong opinions about the topic, leading to unrepresentative results.
This is voluntary response sampling where participants self-select
People with strong opinions (especially opposition) are more likely to call in
The sample consists only of motivated callers, not representative of all listeners
Results are highly suspect due to systematic bias in participant selection
Participants: Motivated callers
Bias: Strong opinion holders
Validity: Very low
This result is not valid due to voluntary response bias. Only motivated listeners chose to call in, and those with strong opinions (especially opposition to taxes) are more likely to participate. The true public opinion could be significantly different from the 75% opposition observed.
• No randomization: Participants self-select
• Extreme views: Strong opinions overrepresented
• Non-representative: Not reflective of target population
• Invalid inference: Cannot generalize to broader population
🔍 Definitions of Key Concepts
⚖️ Core Rules and Principles
⚙️ Step-by-Step Methods for Bias Detection
- Define the target population: Clearly specify who you want to study
- Examine the sampling frame: List of potential participants
- Compare frame to population: Identify who is systematically excluded
- Analyze response patterns: Look for differences between responders and non-responders
- Review question wording: Check for leading or loaded language
- Assess data collection method: Consider how method might influence responses
🧪 Examples: From Simple to Advanced
💡 Tips, Tricks, and Common Pitfalls
📝 Key Notes for Memorization
Questions & Answers
Question: I'm confused about the difference between selection bias and undercoverage bias. Aren't they the same thing?
Answer: Great question! Undercoverage bias is actually a specific type of selection bias:
- Selection bias: A broad category that includes any systematic exclusion of parts of the population during the sampling process.
- Undercoverage bias: A specific form of selection bias that occurs when the sampling frame (the list of potential participants) doesn't include all members of the target population.
Analogy: Think of selection bias as the umbrella term (like "vehicle"), and undercoverage bias as a specific type (like "car"). Other types of selection bias include voluntary response bias and convenience sampling.
Example: Using a phone book to survey households is undercoverage bias (excludes unlisted numbers, cell-only households). Asking friends to participate is convenience sampling, another type of selection bias.
Question: How can I tell if a survey has bias just by reading about it? Are there warning signs?
Answer: Yes, there are several red flags that indicate potential bias:
- Voluntary response: "Call our number," "Click here to vote," or "Tell us your opinion"
- Convenience sampling: Surveying only at malls, universities, or other specific locations
- Leading questions: Questions that assume facts or use emotional language
- Low response rates: Less than 30% response rates suggest non-response bias
- Unclear sampling method: Vague descriptions like "randomly selected" without details
Red flag example: "A survey of 1,000 people found that 90% support the new policy." Without knowing how they were selected or contacted, this result is suspect.
Good sign: Surveys that clearly describe their sampling method, response rates, and acknowledge limitations.
Question: Can bias ever be completely eliminated? Is it possible to have a perfectly unbiased study?
Answer: In practice, completely eliminating all bias is nearly impossible, but we can minimize it significantly:
- Perfect randomization: Requires complete sampling frames and cooperation from all selected participants
- Resource constraints: Time and money limit our ability to achieve perfect conditions
- Human factors: Interviewer effects and respondent behavior introduce subtle biases
Reality: The goal is to minimize bias to levels where it doesn't meaningfully affect conclusions, rather than achieving theoretical perfection.
Best practices: Use random sampling, multiple contact attempts, neutral question wording, and acknowledge limitations in reporting.
Key insight: A well-designed study with acknowledged limitations is better than a poorly designed study claiming perfection.