A4.2.1 Significance of Data Cleaning (HL only) Notes

Machine Learning Lifecycle

1. Define the Problem: Identify the task and desired outcome.
2. Gather Data: Collect relevant data from various sources.
3. Data Preprocessing: Clean and prepare data (handle outliers, missing values, formatting).
4. Exploratory Data Analysis (EDA): Visualise and understand the patterns and relationships in the data.
5. Feature Engineering & Selection: Create and select the most relevant features to improve model performance.
6. Choose a Model: Select a suitable machine learning algorithm.
7. Split the Data: Divide the data into training and testing (or validation) sets.
8. Train the Model: Fit the model to the training data.
9. Model Evaluation: Assess the model’s performance using test/validation data.
10. Parameter Tuning: Optimise hyperparameters to improve accuracy.
11. Deployment: Integrate the model into a real-world environment.
12. Monitor and Maintain: Track performance over time and retrain as needed.

The Significance of Data Cleaning

The Impact of Data Quality on Model Performance

1. Accuracy and Reliability: High-quality data enables models to make precise predictions. Conversely, poor-quality data leads to inaccurate and unreliable outcomes.
2. Generalization: Models trained on clean data generalize better to unseen data, performing well in real-world scenarios.
3. Bias and Fairness: Ensuring data is representative and unbiased prevents models from perpetuating or amplifying biases.

Techniques for Data Cleaning

Handling Outliers

1. Outliers are data points that are significantly different from other observations.
2. Techniques:
   1. Trim: Remove outliers from the dataset.
   2. Cap: Replace outliers with the nearest acceptable value.
   3. Transform: Apply transformations like log or square root to reduce the impact of extreme values.

Removing or Consolidating Duplicate Data

1. Duplicates can skew analysis and model training.
2. Techniques:
   1. Identify and Remove: Use software or database queries to find and delete duplicates.
   2. Consolidate: Merge partially varying duplicates by averaging numerical values or choosing the most frequent category.

Identifying and Correcting Incorrect Data

Techniques:

1. Data Validation: Implement rules based on known ranges or formats to identify anomalies.
2. Cross-Referencing: Use external sources to validate and correct data.
3. Anomaly Detection: Employ machine learning techniques to identify potential inaccuracies.

Filtering Irrelevant Data

Techniques:

1. Feature Selection: Use methods like correlation matrices or random forest importance to remove irrelevant features.
2. Domain Expertise: Consult experts to identify unnecessary data elements.

Transforming Improperly Formatted Data

Techniques:

1. Parsing: Reformat data into a usable structure, such as standardizing date formats.
2. Regular Expressions: Use patterns to identify and transform data formats.

Handling Missing Data

Techniques:

1. Imputation: Replace missing values with the mean, median, or mode.
2. Deletion: Remove records with missing data (listwise or pairwise deletion).
3. Predictive Modeling: Use algorithms like regression or decision trees to predict missing values.

Normalization and Standardization

1. Normalization
   1. Definition: Rescales features to a specific range, typically [0, 1] or [-1, 1].
   2. Use Cases: Beneficial for algorithms sensitive to input scale, such as gradient descent-based methods.
2. Standardization
   1. Definition: Rescales features to have a mean of 0 and a standard deviation of 1.
   2. Use Cases: Useful for data with unknown minimum and maximum values or when outliers are present.