Hyperparameter tuning is the process of testing different predefined settings for a machine-learning algorithm and selecting the combination that produces the best model performance. Unlike model parameters, hyperparameters are normally chosen before training begins.
The Reasoning and Mechanism
A machine-learning model contains parameters learned from training data. For example, a linear regression algorithm learns the coefficients that determine its fitted line.
By contrast, hyperparameters control the model's structure or learning process. Examples include the learning rate of a neural network, the maximum depth of a decision tree, and the value of in a k-nearest neighbours algorithm.
The usual tuning process is:
- Select possible hyperparameter values.
- Train a separate model using each configuration.
- Evaluate each model using validation data and an appropriate performance metric, such as accuracy.
- Select the configuration with the best validation performance.
- Use unseen test data for the final evaluation.
For example, a k-nearest neighbours model might be trained with , , , and . If produces the highest validation accuracy, it may be selected as the tuned hyperparameter.
| Feature | Parameter | Hyperparameter |
|---|---|---|
| How it is obtained | Learned automatically during training | Chosen before training or by a tuning procedure |
| Example | A neural-network weight | Learning rate or number of layers |
| Main role | Represents patterns learned from data | Controls model structure or training behaviour |
Common tuning methods include grid search, which tests every specified combination, and random search, which samples combinations. The validation set must remain separate from the training set; otherwise, data leakage may produce an unreliable performance estimate.
IB Exam Technique
For A4.3 Machine learning approaches (HL only), distinguish hyperparameters from learned parameters and describe the training, validation, and testing sequence. A common misconception is that tuning changes weights directly; in fact, training learns the weights, while tuning selects the settings that govern training or model structure.
