The curse of dimensionality describes the problems that arise when a dataset has a very large number of features, or dimensions. As dimensionality increases, data become sparse, computation increases, and machine-learning models may struggle to identify reliable patterns.
In A4.2 Data preprocessing (HL only), a dimension is normally an input feature, such as age, income, location, or examination score. Increasing the number of features greatly increases the possible feature space in which data points can occur.
Suppose each feature is divided into 10 intervals. One feature produces intervals, two features produce cells, and ten features produce cells. Therefore, the amount of data needed to represent the feature space adequately grows exponentially with the number of dimensions.
| Consequence | Why it occurs |
|---|---|
| Data sparsity | Data points are spread across an increasingly large feature space. |
| Less meaningful distances | In high dimensions, distances between the nearest and furthest points can become increasingly similar, weakening distance-based methods. |
| Overfitting | A model may learn noise or accidental patterns from too many features rather than generalizable relationships. |
| Greater computational cost | More features require additional storage and processing time. |
Feature selection reduces dimensionality by retaining only relevant features. Feature extraction creates a smaller set of new features from the original data; for example, principal component analysis (PCA) transforms correlated features into fewer principal components that preserve as much variance as possible.
A common misconception is that every additional feature improves a model. Extra features help only when they contain useful information; irrelevant or redundant features can reduce model performance.
For an IB exam, define the term and then explain the mechanism: more dimensions enlarge the feature space, making data sparse and learning more difficult. For an “explain” question, link this explicitly to a preprocessing response such as feature selection or feature extraction.
