Evaluating Classifier Performance

Confusion Matrix for Two-Class Classifiers

		Predicted Class
		+	-
Actual Class	+	TP	FN
Actual Class	-	FP	TN

One challenge in machine learning is skew (percentage of positive samples): \[\alpha = \frac{P}{P + N}\]
If the \(\alpha\) isn’t near .5, we describe it as a skewed data set.
Skewed data creates challenges both for training and evaluating classifiers.
How might we address it???

Imbalance During Training

Oversampling
Undersampling
SMOTE - Synthetic Minority Oversampling Technique

Evaluation Metrics

Confusion matrix tells the whole story BUT, we often want a single number to summarize our performance.
- Accuracy \({} = \frac{\text{Correct Predictions}}{\text{Total Predictions}} = \frac{TP + TN}{TP + TN + FP + FN}\)
  - Sensitive to skew, doesn’t distinguish between types of error:
    - Type I Error: False Positive
    - Type II Error: False Negative
- True Positive Rate (TPR)/Recall = Percentage of positives that are classified as positives = \(\frac{TP}{TP + FN}\)
- False Positive Rate (FPR) = Percentage of negatives that are classified as positives = \(\frac{FP}{FP + TN}\)
  - These distinguish error type, but don’t help with skew
- Precision = Correct predictions over the total number of positive predictions = \(\frac{TP}{TP + FP}\)
  - Meaningful even for highly skewed data, since it only considers positive predictions.
- There are also more complicated measures that attempt to combine strengths of those above. E.g.:
  - F1 Score = \(\frac{2 \times precision \times recall}{precision + recall}\)
  - Both precision and recall must be high for the F1 score to be high.

Classifiers Scores + ROC Curves + AUC

Many classifiers are able to produce a score \(s(\mathbf{x})\) that indicates the model’s confidence that \(\mathbf{x}\) is in the positive class.
This allows us to adjust our classifier’s behavior by changing the classification threshold.
Receiver Operating Characteristic (ROC) curve shows the impact of different threshold values:

(Image credit: https://glassboxmedicine.com/2019/02/23/measuring-performance-auc-auroc/
The Area Under the Curve (AUC) provides a single summary value (generally between .5 and 1.0) of the behavior across all possible thresholds