• Accuracy $=\frac{\text{Correct Predictions}}{\text{Total Predictions}}=\frac{TP+TN}{TP+TN+FP+FN}$
  • Sensitive to skew, doesn’t distinguish between types of error:
    • Type I Error: False Positive
    • Type II Error: False Negative
• True Positive Rate (TPR)/Recall = Percentage of positives that are classified as positives = $\frac{TP}{TP+FN}$
• False Positive Rate (FPR) = Percentage of negatives that are classified as positives = $\frac{FP}{FP+TN}$
  • These distinguish error type, but don’t help with skew
• Precision = Correct predictions over the total number of positive predictions = $\frac{TP}{TP+FP}$
  • Meaningful even for highly skewed data, since it only considers positive predictions.
• There are also more complicated measures that attempt to combine strengths of those above. E.g.:
  • F1 Score = $\frac{2\times precision\times recall}{precision+recall}$
  • Both precision and recall must be high for the F1 score to be high.

Many classifiers are able to produce a score $s(\mathbf{x})$ that indicates the model’s confidence that $\mathbf{x}$ is in the positive class.

This allows us to adjust our classifier’s behavior by changing the classification threshold.

Receiver Operating Characteristic (ROC) curve shows the impact of different threshold values:

(Image credit: https://glassboxmedicine.com/2019/02/23/measuring-performance-auc-auroc/

The Area Under the Curve (AUC) provides a single summary value (generally between .5 and 1.0) of the behavior across all possible thresholds