Handle outliers by deciding whether they are errors or valid extremes, then correcting, removing, transforming, or retaining them. Handle missing data using justified deletion or imputation, and document decisions to limit bias.
Outliers
An outlier is a value substantially different from the rest of a dataset. Detect one using domain rules or the interquartile range (IQR) rule: values below the lower quartile minus , or above the upper quartile plus , may be outliers.
An outlier is not automatically wrong. A large transaction could be a recording error or genuine fraud that a model should detect.
| Situation | Appropriate treatment |
|---|---|
| Obvious recording error | Correct it from a reliable source or remove it |
| Valid extreme value | Retain it, especially when relevant |
| Value distorts a model | Consider transformation, capping, or a robust algorithm |
| Cause uncertain | Investigate and document the decision |
Missing Data
First determine why values are absent and how many records are affected. Choose a method that preserves information without creating misleading patterns.
| Method | Appropriate use |
|---|---|
| Remove record | Few records are affected, and deletion will not bias the sample |
| Remove attribute | It has extensive missing data and limited analytical value |
| Mean or median imputation | Numerical values need simple replacement; median is less affected by outliers |
| Mode imputation | A categorical attribute has a meaningful most frequent value |
| Model-based imputation | Other attributes can predict a plausible value |
A common misconception is that every incomplete or unusual record should be deleted. This can shrink the dataset and cause sampling bias; careless imputation can also reduce variation artificially.
Exam technique: For A4.2 Data preprocessing (HL only), state the method, justify it from the dataset's context, and explain one effect on accuracy, bias, or model performance. Writing only "clean the data" is too vague for full credit.
