A4.2.1 Significance of Data Cleaning (HL only) Notes

Machine Learning Lifecycle

1. Define the Problem: Identify the task and desired outcome.
2. Gather Data: Collect relevant data from various sources.
3. Data Preprocessing: Clean and prepare data (handle outliers, missing values, formatting).
4. Exploratory Data Analysis (EDA): Visualise and understand the patterns and relationships in the data.
5. Feature Engineering & Selection: Create and select the most relevant features to improve model performance.
6. Choose a Model: Select a suitable machine learning algorithm.
7. Split the Data: Divide the data into training and testing (or validation) sets.
8. Train the Model: Fit the model to the training data.
9. Model Evaluation: Assess the model’s performance using test/validation data.
10. Parameter Tuning: Optimise hyperparameters to improve accuracy.
11. Deployment: Integrate the model into a real-world environment.
12. Monitor and Maintain: Track performance over time and retrain as needed.

The Significance of Data Cleaning

The Impact of Data Quality on Model Performance

1. Accuracy and Reliability: High-quality data enables models to make precise predictions. Conversely, poor-quality data leads to inaccurate and unreliable outcomes.
2. Generalization: Models trained on clean data generalize better to unseen data, performing well in real-world scenarios.
3. Bias and Fairness: Ensuring data is representative and unbiased prevents models from perpetuating or amplifying biases.

Techniques for Data Cleaning

Handling Outliers

1. Outliers are data points that are significantly different from other observations.
2. Techniques:
   1. Trim: Remove outliers from the dataset.
   2. Cap: Replace outliers with the nearest acceptable value.
   3. Transform: Apply transformations like log or square root to reduce the impact of extreme values.

Removing or Consolidating Duplicate Data

1. Duplicates can skew analysis and model training.
2. Techniques:
   1. Identify and Remove: Use software or database queries to find and delete duplicates.
   2. Consolidate: Merge partially varying duplicates by averaging numerical values or choosing the most frequent category.