Model Evaluation, ROC Curves & Calibration

A machine learning model is only as effective as the metric used to measure its generalization performance. Relying uncritically on accuracy can lead to disastrous deployments in imbalanced regimes such as fraud detection, cancer diagnosis, or rare event prediction.

Rigorous evaluation demands understanding threshold-dependent metric tradeoffs, the probabilistic rank-order interpretation of ROC-AUC, and post-hoc probability calibration.

1. Confusion Matrix & The Harmonic Mean of F1#

Given binary ground truth $y \in \{0, 1\}$ and binary prediction $\hat{y} \in \{0, 1\}$, all predictions partition into a $2 \times 2$ contingency table:

Why the Harmonic Mean for F1?#

Let $P = \frac{TP}{TP + FP}$ denote Precision and $R = \frac{TP}{TP + FN}$ denote Recall.

Why does $F_1$ employ the Harmonic Mean rather than the standard Arithmetic Mean $\frac{P + R}{2}$?

The Generalized $F_\beta$ Score Derivation#

In domains where recall is prioritized over precision (e.g. disease screening), the parameter $\beta$ weights Recall $\beta$ times as heavily as Precision:

Inverting yields the standard $F_\beta$ formula:

2. ROC-AUC as a Probabilistic Rank Statistic#

The Receiver Operating Characteristic (ROC) curve plots the True Positive Rate ($TPR = \frac{TP}{TP + FN}$) against the False Positive Rate ($FPR = \frac{FP}{TN + FP}$) across all classification thresholds $\tau \in [0, 1]$.

Mathematical Equivalence to the Wilcoxon-Mann-Whitney Statistic#

The Area Under the Curve (AUC) has an exact probabilistic interpretation:

Where $x^+$ is a randomly chosen positive sample ($y = 1$) and $x^-$ is a randomly chosen negative sample ($y = 0$).

For $N_+$ positive samples and $N_-$ negative samples with continuous model scores $s_i = f(x_i)$:

Where $\mathbb{I}(\cdot)$ is the binary indicator function. An AUC of $0.50$ denotes random guessing, while $1.0$ represents perfect separation.

3. Probability Calibration: Expected Calibration Error#

A model outputting $\hat{p} = 0.90$ should be correct exactly 90% of the time. Modern deep neural networks with high parameter counts often produce overconfident, uncalibrated probabilities.

To quantify calibration across $M$ confidence bins $B_1, \dots, B_M$:

Where $\text{acc}(B_m)$ is empirical accuracy and $\text{conf}(B_m)$ is average confidence within bin $B_m$.

Post-Hoc Temperature Scaling#

To calibrate modern deep networks without altering top-1 accuracy rankings, Temperature Scaling divides logit scores $z$ by a learned scalar temperature $T > 0$:

• If $T > 1$: Logits are softened, reducing overconfidence.
• If $T < 1$: Logits are sharpened.
• Because temperature scaling is a strictly monotonic transformation, it preserves all argmax class predictions while aligning probabilities with true empirical frequency.

4. From-Scratch Implementation: Zero Dependencies#

The following implementations compute a complete classification evaluation suite (Confusion Matrix, Precision, Recall, Specificity, F1, and ROC-AUC via the trapezoidal rule) from scratch in Python and Rust.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
from typing import List, Dict, Tuple

class ModelEvaluator:
    @staticmethod
    def confusion_matrix(y_true: List[int], y_pred: List[int]) -> Dict[str, int]:
        tp = sum(1 for yt, yp in zip(y_true, y_pred) if yt == 1 and yp == 1)
        fp = sum(1 for yt, yp in zip(y_true, y_pred) if yt == 0 and yp == 1)
        tn = sum(1 for yt, yp in zip(y_true, y_pred) if yt == 0 and yp == 0)
        fn = sum(1 for yt, yp in zip(y_true, y_pred) if yt == 1 and yp == 0)
        return {"TP": tp, "FP": fp, "TN": tn, "FN": fn}

    @classmethod
    def classification_report(cls, y_true: List[int], y_pred: List[int]) -> Dict[str, float]:
        cm = cls.confusion_matrix(y_true, y_pred)
        tp, fp, tn, fn = cm["TP"], cm["FP"], cm["TN"], cm["FN"]

        precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        specificity = tn / (tn + fp) if (tn + fp) > 0 else 0.0
        accuracy = (tp + tn) / len(y_true) if y_true else 0.0
        f1 = (2 * precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0

        return {
            "Accuracy": accuracy,
            "Precision": precision,
            "Recall": recall,
            "Specificity": specificity,
            "F1_Score": f1,
        }

    @staticmethod
    def roc_auc_score(y_true: List[int], y_scores: List[float]) -> float:
        # Separate positive and negative sample scores
        pos_scores = [score for yt, score in zip(y_true, y_scores) if yt == 1]
        neg_scores = [score for yt, score in zip(y_true, y_scores) if yt == 0]

        n_pos = len(pos_scores)
        n_neg = len(neg_scores)
        if n_pos == 0 or n_neg == 0:
            return 0.5

        # Compute Wilcoxon-Mann-Whitney rank statistic
        concordant_pairs = 0.0
        for pos in pos_scores:
            for neg in neg_scores:
                if pos > neg:
                    concordant_pairs += 1.0
                elif pos == neg:
                    concordant_pairs += 0.5

        return concordant_pairs / (n_pos * n_neg)

if __name__ == "__main__":
    y_true = [1, 0, 1, 1, 0, 0, 1, 0, 1, 0]
    y_scores = [0.92, 0.15, 0.78, 0.85, 0.42, 0.23, 0.65, 0.55, 0.89, 0.11]
    y_pred = [1 if s >= 0.5 else 0 for s in y_scores]

    evaluator = ModelEvaluator()
    report = evaluator.classification_report(y_true, y_pred)
    auc = evaluator.roc_auc_score(y_true, y_scores)

    print("Classification Report:")
    for metric, val in report.items():
        print(f"  {metric}: {val:.4f}")
    print(f"  ROC-AUC: {auc:.4f}")

5. Active Knowledge Checks#