Loading subject…
Confusion matrix
A table of the four outcome counts of a two-class prediction: true positives, false positives, false negatives and true negatives.
Example
On 1000 admissions a sepsis model records 80 true positives, 10 false positives, 20 false negatives and 890 true negatives.
Cross-validation
A way of estimating performance that splits the data several times, uses a different slice as the validation set each round, and averages the results.
Example
Five-fold cross-validation trains five times on 800 records and validates on the remaining 200, then averages the five scores.
F1 score
The harmonic mean of precision and recall, used as one number when a false positive and a false negative both carry real cost.
Example
A precision of 0.89 and a recall of 0.80 give an F1 score of 0.84.
Grid search
A hyperparameter search that trains and evaluates the model once for every combination of the values listed for each hyperparameter.
Example
Five values of K crossed with three distance measures is 15 combinations, so 15 training runs.
Hyperparameters
Parameters set before training that cannot be learned from data.
Overfitting
A model that fits the data it was trained on so closely that it captures accidental patterns in that data and performs poorly on records it has not seen.
Example
A price model given 214 columns may split the training sales on wall colour, which says nothing about the price of the next house.
Validation set
A slice of the data held back from training and used to compare hyperparameter settings, so the test set stays untouched until the final measurement.
Example
Calderwood Hospital fits each candidate value of K on 700 admissions and compares them on a separate 150.