Understanding Hyperparameters
Definition
Hyperparameters
Parameters set before training that cannot be learned from data.
Their role is to control the learning algorithm’s behaviour and have a major impact on model performance.
HintThink of hyperparameters as the “dials” you set before training begins.
Confusing hyperparameters with model parameters (weights/biases learned during training).
ExampleK-Nearest Neighbors (K-NN):
- K (neighbors): How many points to consider when classifying.
Decision Trees:
- Maximum Depth: Prevents trees from growing too deep.
- Minimum Samples per Leaf: Controls when a leaf node is created.
Neural Networks:
- Learning Rate: Step size for updating weights.
- Hidden Layers/Neurons: Controls network complexity.
Always link the hyperparameter to the problem type in examples.
Choosing K in K-NN too small (overfitting) or too large (underfitting).
Why Hyperparameter Tuning Matters
- Improves Performance: Boosts accuracy, precision, recall, F1 score.
- Controls Overfitting/Underfitting:
- Overfitting: Too complex (e.g., deep trees, too many layers).
- Underfitting: Too simple (e.g., shallow tree, small K).
Balance model complexity: not too simple, not too complex.
Assuming defaults are always optimal.
Evaluation Metrics in Supervised Learning
- Accuracy: Correct predictions ÷ all predictions.
- Precision: True Positives ÷ Predicted Positives.
- Recall: True Positives ÷ Actual Positives.
