Error identification: The process of recognizing incorrect, inconsistent, or invalid data entries in a dataset
Note: Errors can be typos, impossible values, or outliers that don't fit the expected pattern.
- Examine each data point individually
- Look for values that are significantly different from others
- Check for impossible or unrealistic values
- Identify the problematic entry
Look at: 5, 7, 8, 100, 9
Notice that 100 is much larger than other values (5-9)
100 is likely a typo for 10 (based on context)
Replace 100 with 10 to get [5, 7, 8, 10, 9]
The value 100 is likely an error and should be corrected to 10
• Pattern recognition: Look for values that don't fit the expected pattern
• Context awareness: Consider what values make sense in the given context
• Common errors: Typos often result in significantly different values
• Practice Tip: Always examine data for values that seem too high or too low
- [10, 12, 15, 1000, 14] - 1000 is likely a typo for 10
- [5, 6, 7, 500, 8] - 500 is probably an error
- [20, 22, 21, 200, 23] - 200 stands out as unusual
- Look for values that are much larger or smaller than others
- Consider the context of the data (age, height, etc.)
- Check for obvious typos (missing zeros, extra digits)
Q: How do I know if a value is really an error?
A: Look at the context and pattern. If it's significantly different from other values, it's likely an error.
Q: What if I can't determine the correct value?
A: Mark it as questionable or remove it if possible, noting that it was suspect.
Missing values: Entries in a dataset that are absent, empty, or marked as NULL, NaN, or other placeholders
Note: Missing values need to be handled appropriately to avoid errors in analysis.
- Identify all missing values in the dataset
- Decide how to handle the missing data
- Choose between removal or imputation
- Apply the chosen method
Found NULL at position 3 in [5, 7, NULL, 8, 9]
Can remove the entry or fill with an estimated value
Average of known values: (5+7+8+9) ÷ 4 = 7.25 ≈ 7
Replace NULL with 7: [5, 7, 7, 8, 9]
Replaced the missing value with 7 (the rounded average of other values)
• Identification: Look for NULL, empty, or placeholder values
• Handling options: Remove or impute (fill with estimated value)
• Imputation: Use mean, median, or logical estimate
• Practice Tip: Document when you've imputed missing values
- [10, 15, _, 20, 25] - Fill with 17.5 (average)
- [2, 4, -, 8, 10] - Fill with 6 (logical pattern)
- [30, 35, *, 45, 50] - Replace with 40 (pattern)
- Look for empty spaces, hyphens, or special characters indicating missing data
- Use the average of surrounding values as a common imputation method
- Document when you've filled in missing values
Q: Should I always fill in missing values?
A: It depends on your analysis. Sometimes it's better to remove incomplete entries.
Q: What's the best way to fill missing values?
A: Use the average, median, or a logical estimate based on the pattern.
Outlier detection: Identifying data points that are significantly different from other observations in the dataset
Note: Outliers can be legitimate data points or errors that need investigation.
- Calculate the average (mean) of the dataset
- Look for values that are much higher or lower than average
- Examine the context and pattern of values
- Determine if the outlier is an error or legitimate
(12+15+14+16+100+13+14) ÷ 7 = 174 ÷ 7 = 24.9
Most values (12-16) are much lower than average (24.9)
100 is significantly higher than other values (12-16)
100 is likely an error if other values represent similar measurements
The value 100 is an outlier, significantly different from the other values (12-16)
• Visual inspection: Look for values that stand out
• Statistical approach: Compare to average and range
• Context consideration: Determine if outlier is legitimate
• Practice Tip: Outliers often indicate data entry errors
- [5, 6, 7, 8, 50, 6, 7] - 50 is an outlier
- [20, 21, 22, 23, 200, 21, 22] - 200 stands out
- [1, 2, 3, 4, 100, 2, 3] - 100 is clearly an outlier
- Outliers are values that don't fit the general pattern
- Calculate the average to help identify extreme values
- Consider whether the outlier makes sense in context
Q: Are all outliers errors?
A: No, some outliers represent legitimate extreme values that are part of the data.
Q: How do I decide whether to keep or remove an outlier?
A: If it's clearly an error, correct or remove it. If legitimate, consider keeping it.
Validation rules: Criteria used to check if data values are reasonable, accurate, and within acceptable ranges
Note: Validation rules help identify impossible or incorrect values in datasets.
- Establish appropriate validation rules for the data type
- Check each value against the rules
- Identify values that fail validation
- Correct or flag invalid entries
For ages: must be between 0 and 110 (reasonable range)
-3: negative, invalid; 120: too high, invalid
-3 and 120 fail the validation test
Flag -3 and 120 as invalid entries
Values -3 and 120 are invalid: negative age and impossible age respectively
• Range validation: Check if values fall within acceptable bounds
• Logical validation: Ensure values make sense in context
• Domain-specific rules: Apply rules relevant to data type (age, height, etc.)
• Practice Tip: Establish validation rules before analyzing data
- Heights: [40, 45, -5, 50] - negative height is invalid
- Temperatures: [32, 33, 200, 34] - 200°F is impossible for human temp
- Scores: [85, 90, 150, 88] - 150% is impossible if max is 100%
- Establish reasonable ranges for each data type
- Check for negative values when inappropriate (ages, heights)
- Verify values don't exceed logical maximums
Q: What are common validation rules for different data types?
A: Ages: 0-110, Heights: 12-96 inches, Temperatures: 95-105°F, Scores: 0-100%.
Q: How strict should validation rules be?
A: Make them reasonable but allow for genuine extreme values.
Data consistency: Ensuring that the same information is recorded in the same format throughout the dataset
Note: Inconsistent formatting can lead to errors in analysis and counting.
- Identify the data element that should be consistent
- Look for variations in spelling, capitalization, or format
- Standardize the format across all entries
- Apply the same standard to all similar entries
All entries represent the same name but in different capitalizations
Found: John, JON, jon, Jon, JOHN - same name, different formats
Standardize to proper capitalization: "John"
Change all variations to "John"
Standardized all variations to "John" to ensure data consistency
• Standardization: Apply consistent format to all entries
• Capitalization: Use proper capitalization (names, places)
• Spelling consistency: Ensure same entities use same spelling
• Practice Tip: Create standard formats for common data types
- States: [CA, California, calif, CA] → [CA, CA, CA, CA]
- Colors: [red, RED, Red, rEd] → [Red, Red, Red, Red]
- Days: [Monday, MONDAY, monday, Mon] → [Monday, Monday, Monday, Monday]
- Standardize capitalization for names and proper nouns
- Use consistent abbreviations throughout
- Apply the same format to all similar data entries
Q: How do I handle names with unusual spellings?
A: Respect the original spelling but standardize capitalization.
Q: Should I always standardize to uppercase or lowercase?
A: Use proper capitalization (names, places) for readability.