Filter, wrapper, and embedded methods differ in when and how they select features. Filter methods rank features independently of a model; wrapper methods test feature subsets using a model; embedded methods select features during model training.
How the Methods Work
In feature selection, irrelevant or redundant input variables are removed before or during machine learning. This can reduce training time, limit overfitting, and improve interpretability while retaining useful information.
| Method | Mechanism | Main trade-offs |
|---|---|---|
| Filter method | Uses statistical measures such as correlation or information gain to score features before training. | Fast and inexpensive, but may miss feature interactions or model-specific relationships. |
| Wrapper method | Trains and evaluates a model repeatedly with different feature subsets, including forward selection and backward elimination. | Can identify model-specific combinations, but is computationally expensive and may overfit validation data. |
| Embedded method | Performs selection during model training; decision trees choose informative split features, while regularized models can reduce coefficients. | More efficient than wrappers, but dependent on the learning algorithm. |
For example, suppose a dataset contains age, income, height, and spending history for predicting loan repayment. A filter method might retain features with the strongest statistical association with repayment. A wrapper method would compare predictive performance across subsets. An embedded decision-tree method would prioritize features producing the most informative splits during training.
A common misconception is that all three methods alter feature values. They do not: they select or remove entire features. Feature extraction, instead, creates new features by transforming existing ones.
Exam Technique
For an IB Computer Science HL compare question in A4.2 Data preprocessing, identify similarities and differences. State that all three reduce the feature set, then compare their relationship to training, computational cost, model dependence, and overfitting risk. Avoid listing definitions without direct comparisons.
