Understanding Hyperparameters
Definition
Hyperparameters
Parameters set before training that cannot be learned from data.
Their role is to control the learning algorithm’s behaviour and have a major impact on model performance.
Hint
Think of hyperparameters as the “dials” you set before training begins.
Common Mistake
Confusing hyperparameters with model parameters (weights/biases learned during training).
Example
K-Nearest Neighbors (K-NN):
- K (neighbors): How many points to consider when classifying.
Decision Trees:
- Maximum Depth: Prevents trees from growing too deep.
- Minimum Samples per Leaf: Controls when a leaf node is created.
Neural Networks:
- Learning Rate: Step size for updating weights.
- Hidden Layers/Neurons: Controls network complexity.
Hint
Always link the hyperparameter to the problem type in examples.
Common Mistake
Choosing K in K-NN too small (overfitting) or too large (underfitting).
Why Hyperparameter Tuning Matters
- Improves Performance: Boosts accuracy, precision, recall, F1 score.
- Controls Overfitting/Underfitting:
- Overfitting: Too complex (e.g., deep trees, too many layers).
- Underfitting: Too simple (e.g., shallow tree, small K).
Hint
Balance model complexity: not too simple, not too complex.
Common Mistake
Assuming defaults are always optimal.
Evaluation Metrics in Supervised Learning
- Accuracy: Correct predictions ÷ all predictions.
- Precision: True Positives ÷ Predicted Positives.
- Recall: True Positives ÷ Actual Positives.
- F1 Score: Harmonic mean of precision & recall (useful for imbalanced data).
Hint
Use F1 when precision and recall matter more than raw accuracy.
Common Mistake
Reporting only accuracy on imbalanced datasets.
Techniques for Hyperparameter Tuning
- Grid Search:
- Exhaustively tests all parameter combinations.
- Thorough
- Very slow.
- Random Search:
- Randomly samples combinations.
- Faster,
- May miss best option.
- Bayesian Optimization:
- Uses probability models to predict good hyper parameters.
- Efficient for complex models,
- Harder to implement.
