Dimensionality is the number of features or variables in a data set, which is the number of columns rather than the number of records.
Each feature records one aspect of the data, and how many there are depends on the kind of data being stored.
Customer data: age, income and location, so a few dozen features at most.
Medical images: every pixel is a feature, so a 256 by 256 greyscale scan already has 65,536 of them.
Text data: one feature per word in the vocabulary, so a modest corpus runs to tens of thousands.
Note
Dimensionality reduction is the process of reducing the number of features in a data set while preserving the relevant aspects of the data.
The aim is a narrower data set that still answers the same question, not simply a smaller one.
The Curse of Dimensionality
As features are added, the space they describe grows exponentially, so a fixed number of records spreads ever more thinly across it.
The syllabus names seven consequences of that.
Overfitting: with enough features a model can find a rule that separates the training data perfectly and means nothing on new data.
Computational complexity: every extra feature adds calculations to every step of training and prediction.
Data sparsity: to hold the same density, 10 points along one feature become 100 across two and 1,000 across three, so real data sets become mostly empty space.
Effectiveness of distance metrics: in high dimensions the nearest and the farthest point end up almost equally far away, so Euclidean distance stops separating them and k-nearest neighbours degrades.
Data visualization: a scatter plot can show two or three features, so beyond that the data cannot be inspected by eye.
Sample size increases: keeping the same coverage needs exponentially more records as features are added, and that data usually does not exist.
Memory usage: a wide data set has to be held in memory during training, and the requirement rises with every column.
Note
The term "curse of dimensionality" was coined by Richard Bellman in 1957 while working on dynamic programming.
He was describing how the work needed to search a space grows exponentially with the number of dimensions in it.
Two Routes to Fewer Features
Feature selection: keep a subset of the original columns and drop the rest, so every surviving feature still means exactly what it meant before.
Feature extraction: build new features that combine several originals, so the data set gets narrower but the new columns no longer correspond to anything a person can name.
The choice is a trade: selection keeps the model explainable, and extraction usually compresses further.
Note
Statistical extraction techniques such as principal component analysis (PCA) and linear discriminant analysis (LDA) are beyond the scope of this course.
You are expected to know that feature extraction exists and what it does, not how these techniques compute their new features.
Worked Scenario: Predicting Home Sale Prices
Initial data set: bedrooms, bathrooms, floor area and location, which is four features across a few thousand sales.
High-dimensional expansion: adding appliance brands, wall colours, window types and fittings pushes it past two hundred features on the same number of sales.
Three problems follow immediately.
Data sparsity: almost no two houses now share a combination of values, so each sale sits isolated in the feature space.
Overfitting: the model starts predicting from wall colour because that happened to separate the training sales.
Increased complexity: training slows and memory use rises, for features that carry no information about price.
Example
A data set with 1,000 features is cut to the 100 that carry the information.
Training time and memory use fall sharply, and accuracy on unseen data usually rises, because the 900 discarded columns were contributing noise rather than signal.
Benefits of Dimensionality Reduction
Reduces overfitting: removing features the model could latch onto by chance keeps it fitting the pattern rather than the noise.
Speeds up training and cuts memory use: fewer columns means fewer calculations per step and a smaller data set to hold.
Restores meaningful visualization: cutting to two or three dimensions makes a scatter plot possible again, so patterns can be seen rather than only measured.
Improves accuracy on unseen data: with sparsity and the distance problem reduced, distance-based algorithms such as k-nearest neighbours and k-means behave as intended.
Common Mistake
Reducing dimensions is a balance, not an automatic improvement.
Every feature removed is information removed, and cutting too far discards attributes the model needed.
The syllabus wording is to reduce the variables while preserving the relevant aspects of the data, so the test of a reduction is whether performance on unseen data holds up afterwards.
What Examiners Expect on Dimensionality Reduction
A4.2.3 uses the command term Describe, so state a consequence of high dimensionality and say what causes it, rather than listing the seven terms.
The seven curse-of-dimensionality considerations are the examinable list, so learn them by name.
PCA and LDA are out of scope, so naming them earns nothing; describe the effect of reducing dimensions instead of the algorithm that does it.
Active recall
Define dimensionality and state what a single dimension corresponds to in a data set of medical images.
Name four of the seven considerations that make up the curse of dimensionality.
Why do distance metrics such as Euclidean distance become less effective as the number of features grows?
Distinguish between feature selection and feature extraction as routes to fewer features.
Give one situation in which reducing dimensions makes a model worse rather than better.
