Loading subject…
Curse of dimensionality
The set of problems that appear as the number of features grows, because the space the data describes expands exponentially while the number of records does not.
Example
Splitting each of four features into ten bands gives 10,000 cells, so 3,200 sales average less than one sale per cell.
Data cleaning
The stage of data preprocessing where errors, gaps, duplicates and inconsistencies are removed from a data set before a model is trained.
Dimensionality
The number of features, or columns, in a data set. It is counted separately from the number of records the data set holds.
Example
A table of 3,200 property sales recording bedrooms, bathrooms, floor area and postcode district has a dimensionality of four.
Dimensionality reduction
The process of reducing the number of variables in a data set while preserving the relevant aspects of the data.
Example
Cutting a property data set from 214 columns to the 12 that carry information about price is a dimensionality reduction.
Embedded method
A feature selection strategy in which the selection happens inside the training process of a single model, so the model decides which features matter as it fits.
Feature extraction
A route to fewer variables that builds new features by combining several original ones, so the data set narrows but the new features no longer correspond to a named original attribute.
Example
Blending floor area, room count and plot size into one new column reduces three features to one, and that column has no name or unit of its own.
Feature selection
Feature selection is the process of identifying and retaining the most informative attributes of a data set while removing those that are redundant or irrelevant.
Filter method
A feature selection strategy that scores each feature on its own statistical relationship with the target variable, before any model is trained, and keeps the highest scoring features.
Imputation
Filling a missing value with an estimate calculated from the data that is present, most often the mean, median or mode of the same column.
Normalization (data preprocessing)
Rescaling a numeric feature into a fixed range, usually 0 to 1, so that features measured in different units contribute to a model on a comparable scale. This is a different process from normalization in relational database design.
Outlier
A data point whose value sits far away from the rest of the observations in its column, which can pull a model's fit toward a value that is not typical of the data.
Overfitting
A model that fits the data it was trained on so closely that it captures accidental patterns in that data and performs poorly on records it has not seen.
Example
A price model given 214 columns may split the training sales on wall colour, which says nothing about the price of the next house.
Standardization
Rescaling a numeric feature so that it has a mean of 0 and a standard deviation of 1, which expresses every value as the number of standard deviations it sits from the mean.
Wrapper method
A feature selection strategy that evaluates a subset of features by training a model on that subset and measuring the resulting performance, repeating the search until the best performing subset is found.