Unsupervised Validation & Evaluation Metrics | AI & Machine Learning Engineering

In supervised learning, evaluating performance is straightforward because ground truth labels $y$ are known. In unsupervised clustering and manifold discovery, external labels are absent in production, making model evaluation inherently challenging.

Validation requires mathematical criteria that evaluate the intrinsic geometry of partitions: compactness (cohesion) within clusters and isolation (separation) between distinct clusters.

Silhouette RangeCluster Metric

[-1.0, +1.0]

Evaluates whether samples belong to their assigned cluster versus neighboring clusters.

1. Internal Validation: The Silhouette Coefficient#

The Silhouette Coefficient (Rousseeuw, 1987) measures how well each individual point has been clustered by contrasting intra-cluster tightness against inter-cluster separation.

Architecture Diagram

Rendering diagram...

Step 1: Intra-Cluster Dissimilarity $a(i)$

Let point $i$ belong to cluster $C_I$ . Compute the average distance between $i$ and all other points in the same cluster $C_I$ :

a(i) = \frac{1}{|C_I| - 1} \sum_{j \in C_I, j \ne i} \|x_i - x_j\|_2

$a(i)$ quantifies the compactness of cluster $C_I$ around point $i$ . Small $a(i)$ denotes high cluster cohesion.

Step 2: Nearest-Cluster Dissimilarity $b(i)$

For any other cluster $C_J \ne C_I$ , compute the average distance from point $i$ to all points in $C_J$ :

d(i, C_J) = \frac{1}{|C_J|} \sum_{j \in C_J} \|x_i - x_j\|_2

Define $b(i)$ as the minimum average distance across all alternative clusters:

b(i) = \min_{J \ne I} d(i, C_J)

The cluster $C_{\text{neighbor}}$ that achieves this minimum is the "neighboring cluster" of point $i$ .

Step 3: Normalize the Discrepancy

The Silhouette score $s(i)$ is the normalized difference between inter-cluster separation and intra-cluster cohesion:

s(i) = \frac{b(i) - a(i)}{\max\Big(a(i), b(i)\Big)}

Interpretation of Values:#

$s(i) \approx +1$ : $a(i) \ll b(i)$ . Point $i$ is tightly clustered and far from its nearest neighboring cluster.
$s(i) \approx 0$ : $a(i) \approx b(i)$ . Point $i$ sits directly on the boundary between two clusters.
$s(i) \approx -1$ : $a(i) \gg b(i)$ . Point $i$ is closer to neighboring cluster points than its assigned cluster, indicating a severe misassignment.

2. The Davies-Bouldin Index#

The Davies-Bouldin (DB) Index evaluates clustering quality by computing the maximum ratio of intra-cluster dispersion to inter-cluster centroid separation.

Let $C_i$ have centroid $\mu_i$ . The cluster dispersion $S_i$ is the average distance of points to their centroid:

S_i = \frac{1}{|C_i|} \sum_{x \in C_i} \|x - \mu_i\|_2

The separation between clusters $i$ and $j$ is the Euclidean distance between centroids $M_{ij} = \|\mu_i - \mu_j\|_2$ .

The similarity ratio $R_{ij}$ evaluates the overlap between clusters $i$ and $j$ :

R_{ij} = \frac{S_i + S_j}{M_{ij}} = \frac{S_i + S_j}{\|\mu_i - \mu_j\|_2}

The Davies-Bouldin Index is the average over all clusters of the worst-case similarity ratio:

DB = \frac{1}{K} \sum_{i=1}^K \max_{j \ne i} R_{ij}

Optimization Direction

Unlike the Silhouette score where higher is better, lower Davies-Bouldin values indicate superior clustering. A small DB score requires both minimal intra-cluster scatter (

S_i \to 0

) and maximal inter-centroid separation (

M_{ij} \to \infty

3. External Validation: Adjusted Rand Index (ARI)#

When ground truth class labels $Y^* = \{y_1^*, \dots, y_N^*\}$ are available for benchmark validation, we can measure the agreement between predicted partitions $C$ and true partitions $Y^*$ .

The Rand Index (RI) considers all $\binom{N}{2}$ unique pairs of points $(x_i, x_j)$ :

$a$ : Number of pairs assigned to the same cluster in both $Y^*$ and $C$ .
$b$ : Number of pairs in the same cluster in $Y^*$ , but different in $C$ .
$c$ : Number of pairs in different clusters in $Y^*$ , but same in $C$ .
$d$ : Number of pairs assigned to different clusters in both $Y^*$ and $C$ .

\text{RI} = \frac{a + d}{a + b + c + d} = \frac{a + d}{\binom{N}{2}}

Chance Correction#

The raw Rand Index does not approach zero for random cluster assignments. The Adjusted Rand Index (ARI) applies Hubert and Arabie's generalized chance correction:

\text{ARI} = \frac{\text{Index} - \mathbb{E}[\text{Index}]}{\text{Max(Index)} - \mathbb{E}[\text{Index}]}

Under a generalized hypergeometric null model of random clusterings:

\text{ARI} = \frac{\sum_{i,j} \binom{n_{ij}}{2} - \left[ \sum_i \binom{a_i}{2} \sum_j \binom{b_j}{2} \right] / \binom{N}{2}}{\frac{1}{2}\left[ \sum_i \binom{a_i}{2} + \sum_j \binom{b_j}{2} \right] - \left[ \sum_i \binom{a_i}{2} \sum_j \binom{b_j}{2} \right] / \binom{N}{2}}

$\text{ARI} = 1.0$ : Identical partitions.
$\text{ARI} = 0.0$ : Cluster assignments no better than random guessing.
$\text{ARI} < 0.0$ : Cluster assignments actively disagree with ground truth.

4. From-Scratch Implementation: Zero Dependencies#

The following implementations compute the Silhouette Score and Davies-Bouldin Index from scratch in pure Python and pure Rust.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
import math
from typing import List, Dict

class UnsupervisedEvaluator:
    @staticmethod
    def _euclidean_dist(a: List[float], b: List[float]) -> float:
        return math.sqrt(sum((x - y) ** 2 for x, y in zip(a, b)))

    @classmethod
    def silhouette_score(cls, X: List[List[float]], labels: List[int]) -> float:
        num_samples = len(X)
        unique_labels = list(set(labels))
        if len(unique_labels) <= 1:
            return 0.0

        # Partition points by cluster label
        clusters: Dict[int, List[int]] = {l: [] for l in unique_labels}
        for idx, l in enumerate(labels):
            clusters[l].append(idx)

        silhouette_values = []

        for i in range(num_samples):
            current_label = labels[i]
            same_cluster_indices = clusters[current_label]

            # 1. Compute a(i)
            if len(same_cluster_indices) <= 1:
                silhouette_values.append(0.0)
                continue

            a_i = sum(
                cls._euclidean_dist(X[i], X[j])
                for j in same_cluster_indices if j != i
            ) / (len(same_cluster_indices) - 1)

            # 2. Compute b(i)
            b_i = float("inf")
            for other_label, other_indices in clusters.items():
                if other_label == current_label:
                    continue
                mean_dist_other = sum(
                    cls._euclidean_dist(X[i], X[j]) for j in other_indices
                ) / len(other_indices)
                b_i = min(b_i, mean_dist_other)

            # 3. Compute s(i)
            denom = max(a_i, b_i)
            s_i = (b_i - a_i) / denom if denom > 0 else 0.0
            silhouette_values.append(s_i)

        return sum(silhouette_values) / num_samples

    @classmethod
    def davies_bouldin_index(cls, X: List[List[float]], labels: List[int]) -> float:
        unique_labels = list(set(labels))
        k = len(unique_labels)
        if k <= 1:
            return 0.0

        num_features = len(X[0])
        # Compute centroids and scatters
        centroids: Dict[int, List[float]] = {}
        scatters: Dict[int, float] = {}

        for l in unique_labels:
            indices = [idx for idx, label in enumerate(labels) if label == l]
            size = len(indices)
            centroid = [
                sum(X[i][feat] for i in indices) / size
                for feat in range(num_features)
            ]
            centroids[l] = centroid
            scatter = sum(cls._euclidean_dist(X[i], centroid) for i in indices) / size
            scatters[l] = scatter

        # Compute max R_ij per cluster
        db_sum = 0.0
        for l1 in unique_labels:
            max_r = 0.0
            for l2 in unique_labels:
                if l1 == l2:
                    continue
                m_ij = cls._euclidean_dist(centroids[l1], centroids[l2])
                if m_ij > 0:
                    r_ij = (scatters[l1] + scatters[l2]) / m_ij
                    max_r = max(max_r, r_ij)
            db_sum += max_r

        return db_sum / k

if __name__ == "__main__":
    X = [
        [1.0, 1.0], [1.1, 1.2], [0.9, 1.0],  # Cluster 0
        [8.0, 8.0], [8.2, 8.1], [7.9, 8.0],  # Cluster 1
    ]
    labels = [0, 0, 0, 1, 1, 1]

    sil = UnsupervisedEvaluator.silhouette_score(X, labels)
    db = UnsupervisedEvaluator.davies_bouldin_index(X, labels)

    print(f"Silhouette Score (Higher => Better): {sil:.4f}")
    print(f"Davies-Bouldin Index (Lower => Better): {db:.4f}")

5. Active Knowledge Checks#

Concept check

What does a negative Silhouette score s(i) < 0 indicate for a specific data point i?

Concept check

Why does the Davies-Bouldin Index evaluate to a lower score for superior cluster partitions?