Dimensionality reduction is a data preprocessing technique that decreases the number of features, or variables, used to represent each data item while preserving as much useful information as possible. It can make a dataset easier and faster for a machine learning model to process.
A dataset with features represents each item in dimensions. Dimensionality reduction produces a representation with dimensions, where .
The process generally works as follows:
- Identify features that are irrelevant, redundant, or strongly correlated.
- Remove unnecessary features or combine existing features into fewer new ones.
- Evaluate whether the reduced dataset retains enough information for accurate analysis or prediction.
For example, a dataset predicting student performance might contain attendance rate, homework completion, study time, and many repeated measures of similar behaviour. Reducing overlapping variables can lower processing requirements without substantially reducing predictive accuracy.
| Method | Mechanism | Example |
|---|---|---|
| Feature selection | Keeps the most useful original features and removes others. | Removing a student ID field because it has no predictive value. |
| Feature extraction | Transforms several original features into fewer new features. | Principal component analysis (PCA) combines correlated variables into principal components that retain as much variance as possible. |
Dimensionality reduction may reduce storage requirements, training time, noise, and overfitting. It can also help visualize high-dimensional data by reducing it to two or three dimensions.
However, information may be lost. Feature extraction can also reduce interpretability because transformed features may not have an obvious real-world meaning.
A common misconception is that dimensionality reduction means deleting data records. It normally reduces the number of features, not the number of observations.
For an IB Computer Science HL question on A4.2 Data preprocessing, define the technique, distinguish feature selection from feature extraction, and explain at least one benefit and one limitation. If the command term is evaluate, reach a justified conclusion about whether reduced complexity outweighs possible information loss.
