Data cleaning is important because a machine-learning model learns patterns from the data it receives. If the data is incomplete, inconsistent, duplicated, or incorrect, the model may learn misleading patterns and produce inaccurate or unfair predictions.
In data preprocessing, cleaning improves the quality of data before it is used to train a model. This supports the principle often summarized as “garbage in, garbage out”: even a sophisticated algorithm cannot reliably compensate for poor-quality training data.
Common cleaning operations include:
| Data-quality problem | Cleaning action and importance |
|---|---|
| Missing values | Remove affected records or replace values using an appropriate method. This prevents errors and reduces incomplete learning. |
| Duplicate records | Remove repeated entries so particular cases are not given disproportionate influence. |
| Inconsistent formats | Standardize values, such as converting all dates to one format or using one unit of measurement. |
| Invalid or incorrect values | Correct or remove impossible values, such as a negative age. |
| Outliers | Investigate unusually extreme values and remove them only if they are errors or unsuitable for the task. |
| Inconsistent labels | Standardize categories, such as treating “UK”, “U.K.”, and “United Kingdom” as one label. |
Cleaning can improve model accuracy because the model identifies patterns that better represent the real population. It can also improve fairness: systematic errors affecting one demographic group may otherwise create or reinforce algorithmic bias.
However, cleaning must be justified. A common misconception is that every unusual value should be deleted. A valid outlier may contain important information, so automatic removal could reduce representativeness and introduce bias.
For an IB Computer Science HL examination response on A4.2 Data preprocessing, identify a specific data-quality problem, describe the cleaning method, and explain its effect on the model. Do not merely state that cleaning “makes data better”; link it to consequences such as improved accuracy, reduced bias, greater consistency, or more reliable predictions.
