Multimodal Alignment & Contrastive Learning (CLIP)

Traditional artificial intelligence systems treat sensory modalities in strict isolation: Computer Vision models classify spatial pixel matrices, Natural Language Processing models process discrete text token sequences, and Speech architectures analyze acoustic frequency spectrograms.

Multimodal AI bridges these heterogeneous perception streams into a single unified geometric vector space. The breakthrough architecture enabling open-vocabulary zero-shot perception is OpenAI's Contrastive Language-Image Pre-training (CLIP).

Symmetric InfoNCE LossMultimodal Vision-Language

L_InfoNCE = -log(exp(s_ii / tau) / sum_j exp(s_ij / tau))

Maximizes cosine similarity between matching image-text pairs while repelling non-matching negative pairs across the batch.

1. The Multimodal Vector Alignment Problem#

Given an image $x_I \in \mathbb{R}^{H \times W \times C}$ and a text caption $x_T$ , single-modality encoders output features residing in completely distinct coordinate systems:

v = f_I(x_I) \in \mathbb{R}^{d_I}, \quad t = f_T(x_T) \in \mathbb{R}^{d_T}

Because $d_I$ and $d_T$ inhabit separate vector spaces, Euclidean distance or inner products between $v$ and $t$ are mathematically undefined.

To resolve this, CLIP introduces linear projection layers $W_I \in \mathbb{R}^{d_I \times d_e}$ and $W_T \in \mathbb{R}^{d_T \times d_e}$ that map both representations into a shared latent embedding space of dimension $d_e$ :

z_I = \frac{W_I^T v}{\|W_I^T v\|_2}, \quad z_T = \frac{W_T^T t}{\|W_T^T t\|_2}

Both vectors are $L_2$ -normalized to lie on the unit hypersphere $\mathbb{S}^{d_e - 1}$ . Their semantic compatibility is measured by the cosine similarity:

s(z_I, z_T) = z_I \cdot z_T = \cos(\theta)

Architecture Diagram

Rendering diagram...

2. Derivation of the InfoNCE Contrastive Loss#

Consider a mini-batch of $N$ paired image-text examples:

\{(I_1, T_1), (I_2, T_2), \dots, (I_N, T_N)\}

There are $N$ positive matching pairs on the diagonal: $(I_i, T_i)$ . The remaining $N(N - 1)$ off-diagonal pairs $(I_i, T_j)$ for $i \ne j$ serve as negative distractors.

Let $s_{i,j} = z_{I,i} \cdot z_{T,j}$ denote the cosine similarity between image $i$ and text $j$ . Let $\tau > 0$ denote a learnable temperature parameter controlling the sharpness of the probability distribution.

Step 1: Image-to-Text Classification Loss ( $\mathcal{L}_{I \to T}$ )#

For image $i$ , we formulate a categorical distribution over all $N$ candidate text captions in the batch using the Softmax function:

p_{i,j}^{I \to T} = \frac{\exp(s_{i,j} / \tau)}{\sum_{k=1}^N \exp(s_{i,k} / \tau)}

The ground-truth label is the matching diagonal caption $j = i$ . The categorical cross-entropy loss for image $i$ is:

\ell_i^{I \to T} = -\log p_{i,i}^{I \to T} = -\log \left( \frac{\exp(s_{i,i} / \tau)}{\sum_{k=1}^N \exp(s_{i,k} / \tau)} \right)

Averaging across all $N$ images in the mini-batch yields:

\mathcal{L}_{I \to T} = -\frac{1}{N} \sum_{i=1}^N \log \left( \frac{\exp(s_{i,i} / \tau)}{\sum_{k=1}^N \exp(s_{i,k} / \tau)} \right)

Step 2: Text-to-Image Classification Loss ( $\mathcal{L}_{T \to I}$ )#

Symmetrically, for text caption $j$ , we predict which of the $N$ candidate images corresponds to it:

p_{j,i}^{T \to I} = \frac{\exp(s_{i,j} / \tau)}{\sum_{k=1}^N \exp(s_{k,j} / \tau)}

The cross-entropy loss averaged across all $N$ texts is:

\mathcal{L}_{T \to I} = -\frac{1}{N} \sum_{j=1}^N \log \left( \frac{\exp(s_{j,j} / \tau)}{\sum_{k=1}^N \exp(s_{k,j} / \tau)} \right)

Step 3: Total Symmetric InfoNCE Objective#

The total CLIP loss is the arithmetic mean of the two directional contrastive objectives:

\mathcal{L}_{\text{CLIP}} = \frac{1}{2} \left( \mathcal{L}_{I \to T} + \mathcal{L}_{T \to I} \right)

The Role of Batch Size in InfoNCE

Notice the denominator in

\mathcal{L}_{I \to T}

sums over

N

items. If batch size

N

is small (e.g.

N = 32

), distinguishing the true positive from 31 negatives is relatively easy. In contrast, OpenAI trained CLIP with an immense batch size of

N = 32,768

, forcing the model to discriminate the true caption against 32,767 fine-grained distractors simultaneously.

3. Alternative Alignment: SigLIP (Sigmoid Loss)#

In 2023, Xiaohua Zhai et al. introduced SigLIP (Sigmoid Loss for Language-Image Pre-training).

Standard Softmax InfoNCE requires global denominator normalization across all items in the batch ( $\sum_k \exp(s_{i,k}/\tau)$ ). In multi-GPU distributed data parallel setups, this requires expensive All-Gather collective communications.

SigLIP reframes multimodal contrastive learning as $N^2$ independent binary classification problems using the sigmoid function $\sigma(z) = \frac{1}{1 + e^{-z}}$ :

\mathcal{L}_{\text{SigLIP}} = -\frac{1}{N} \sum_{i=1}^N \sum_{j=1}^N \log \sigma\left( y_{i,j} (t \cdot s_{i,j} + b) \right)

where $y_{i,j} = 1$ if $i = j$ (positive match) and $y_{i,j} = -1$ if $i \ne j$ (negative pair), with learnable temperature $t$ and bias $b$ .

Because every cell in the $N \times N$ similarity matrix is evaluated independently without batch-wide normalization, SigLIP scales to arbitrarily large batch sizes with reduced communication overhead.

4. Zero-Shot Classification Mechanics#

Once trained, CLIP operates as an open-vocabulary classifier without requiring any fine-tuning on downstream datasets:

Architecture Diagram

Rendering diagram...

5. From-Scratch Implementation: CLIP Contrastive Engine & Zero-Shot Retrieval#

We implement the complete CLIP dual-encoder embedding projection, cosine similarity matrix computation, symmetric InfoNCE loss, and zero-shot retrieval ranking from scratch in Python and Rust without external libraries.

Python Implementation (Pure Standard Library)#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
"""
CLIP Dual-Encoder Embedding Alignment & Symmetric InfoNCE Loss
from scratch using only the Python standard library.
"""
import math
from typing import List, Tuple

def l2_normalize(vec: List[float]) -> List[float]:
    norm = math.sqrt(sum(x * x for x in vec))
    if norm < 1e-12:
        return vec[:]
    return [x / norm for x in vec]

def dot_product(v1: List[float], v2: List[float]) -> float:
    return sum(a * b for a, b in zip(v1, v2))

def matmul_vec(matrix: List[List[float]], vec: List[float]) -> List[float]:
    # matrix: [out_dim, in_dim], vec: [in_dim]
    return [dot_product(row, vec) for row in matrix]

class SimpleCLIPAlignment:
    def __init__(self, vis_dim: int, text_dim: int, embed_dim: int, temperature: float = 0.07):
        self.vis_dim = vis_dim
        self.text_dim = text_dim
        self.embed_dim = embed_dim
        self.temperature = temperature

        # Projection matrices: W_vis [embed_dim, vis_dim], W_text [embed_dim, text_dim]
        # Initialized with simple deterministic scaling
        scale_v = 1.0 / math.sqrt(vis_dim)
        self.w_vis = [[((i + j) % 7 - 3) * scale_v * 0.1 for j in range(vis_dim)] for i in range(embed_dim)]

        scale_t = 1.0 / math.sqrt(text_dim)
        self.w_text = [[((i * j) % 5 - 2) * scale_t * 0.1 for j in range(text_dim)] for i in range(embed_dim)]

    def project_image(self, raw_vis: List[float]) -> List[float]:
        projected = matmul_vec(self.w_vis, raw_vis)
        return l2_normalize(projected)

    def project_text(self, raw_text: List[float]) -> List[float]:
        projected = matmul_vec(self.w_text, raw_text)
        return l2_normalize(projected)

    def compute_infonce_loss(self, img_embeds: List[List[float]], txt_embeds: List[List[float]]) -> float:
        N = len(img_embeds)
        # Compute cosine similarity matrix S: [N, N]
        sim_matrix = [[dot_product(img_embeds[i], txt_embeds[j]) for j in range(N)] for i in range(N)]

        # Image-to-Text Cross Entropy
        loss_i2t = 0.0
        for i in range(N):
            scaled_logits = [sim_matrix[i][j] / self.temperature for j in range(N)]
            max_logit = max(scaled_logits)
            exp_sum = sum(math.exp(l - max_logit) for l in scaled_logits)
            log_prob = (scaled_logits[i] - max_logit) - math.log(exp_sum)
            loss_i2t += -log_prob
        loss_i2t /= N

        # Text-to-Image Cross Entropy
        loss_t2i = 0.0
        for j in range(N):
            scaled_logits = [sim_matrix[i][j] / self.temperature for i in range(N)]
            max_logit = max(scaled_logits)
            exp_sum = sum(math.exp(l - max_logit) for l in scaled_logits)
            log_prob = (scaled_logits[j] - max_logit) - math.log(exp_sum)
            loss_t2i += -log_prob
        loss_t2i /= N

        return 0.5 * (loss_i2t + loss_t2i)

    def zero_shot_classify(self, img_embed: List[float], class_text_embeds: List[List[float]]) -> List[float]:
        # Compute similarities with all class embeddings
        logits = [dot_product(img_embed, class_emb) / self.temperature for class_emb in class_text_embeds]
        max_l = max(logits)
        exps = [math.exp(l - max_l) for l in logits]
        sum_exp = sum(exps)
        return [e / sum_exp for e in exps]

if __name__ == "__main__":
    clip = SimpleCLIPAlignment(vis_dim=4, text_dim=4, embed_dim=3, temperature=0.1)

    # Batch of 3 paired synthetic features
    raw_images = [
        [1.0, 0.5, -0.2, 0.8],   # dog image
        [-0.5, 1.2, 0.3, -0.1],  # sports car image
        [0.2, -0.8, 1.5, 0.4]    # sunset beach image
    ]
    raw_texts = [
        [0.9, 0.4, -0.1, 0.7],   # "a photo of a dog"
        [-0.4, 1.1, 0.2, -0.2],  # "a photo of a sports car"
        [0.1, -0.7, 1.4, 0.5]    # "a photo of a sunset beach"
    ]

    img_emb = [clip.project_image(img) for img in raw_images]
    txt_emb = [clip.project_text(txt) for txt in raw_texts]

    loss = clip.compute_infonce_loss(img_emb, txt_emb)
    print(f"InfoNCE Loss for batch: {loss:.4f}")

    # Zero-shot classification test for Image 0 (dog)
    probs = clip.zero_shot_classify(img_emb[0], txt_emb)
    for c_idx, p in enumerate(probs):
        print(f"Class {c_idx} Probability: {p * 100:.2f}%")

Rust Implementation (Pure Standard Library)#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
// Multimodal Contrastive Alignment & Symmetric InfoNCE Loss
// Implemented strictly from scratch in pure Rust std.

fn l2_normalize(vec: &[f64]) -> Vec<f64> {
    let norm_sq: f64 = vec.iter().map(|&x| x * x).sum();
    let norm = norm_sq.sqrt();
    if norm < 1e-12 {
        vec.to_vec()
    } else {
        vec.iter().map(|&x| x / norm).collect()
    }
}

fn dot_product(a: &[f64], b: &[f64]) -> f64 {
    a.iter().zip(b.iter()).map(|(&x, &y)| x * y).sum()
}

struct SimpleCLIP {
    embed_dim: usize,
    temperature: f64,
    w_vis: Vec<Vec<f64>>,  // [embed_dim x vis_dim]
    w_text: Vec<Vec<f64>>, // [embed_dim x text_dim]
}

impl SimpleCLIP {
    fn new(vis_dim: usize, text_dim: usize, embed_dim: usize, temperature: f64) -> Self {
        let scale_v = 1.0 / (vis_dim as f64).sqrt();
        let mut w_vis = vec![vec![0.0; vis_dim]; embed_dim];
        for i in 0..embed_dim {
            for j in 0..vis_dim {
                w_vis[i][j] = (((i + j) % 7) as f64 - 3.0) * scale_v * 0.1;
            }
        }

        let scale_t = 1.0 / (text_dim as f64).sqrt();
        let mut w_text = vec![vec![0.0; text_dim]; embed_dim];
        for i in 0..embed_dim {
            for j in 0..text_dim {
                w_text[i][j] = (((i * j) % 5) as f64 - 2.0) * scale_t * 0.1;
            }
        }

        Self {
            embed_dim,
            temperature,
            w_vis,
            w_text,
        }
    }

    fn project_image(&self, raw: &[f64]) -> Vec<f64> {
        let proj: Vec<f64> = self.w_vis.iter().map(|row| dot_product(row, raw)).collect();
        l2_normalize(&proj)
    }

    fn project_text(&self, raw: &[f64]) -> Vec<f64> {
        let proj: Vec<f64> = self.w_text.iter().map(|row| dot_product(row, raw)).collect();
        l2_normalize(&proj)
    }

    fn compute_loss(&self, img_embeds: &[Vec<f64>], txt_embeds: &[Vec<f64>]) -> f64 {
        let n = img_embeds.len();
        let mut sim = vec![vec![0.0; n]; n];
        for i in 0..n {
            for j in 0..n {
                sim[i][j] = dot_product(&img_embeds[i], &txt_embeds[j]);
            }
        }

        // Image to Text Loss
        let mut loss_i2t = 0.0;
        for i in 0..n {
            let scaled: Vec<f64> = (0..n).map(|j| sim[i][j] / self.temperature).collect();
            let max_val = scaled.iter().cloned().fold(f64::NEG_INFINITY, f64::max);
            let sum_exp: f64 = scaled.iter().map(|&x| (x - max_val).exp()).sum();
            let log_prob = (scaled[i] - max_val) - sum_exp.ln();
            loss_i2t -= log_prob;
        }
        loss_i2t /= n as f64;

        // Text to Image Loss
        let mut loss_t2i = 0.0;
        for j in 0..n {
            let scaled: Vec<f64> = (0..n).map(|i| sim[i][j] / self.temperature).collect();
            let max_val = scaled.iter().cloned().fold(f64::NEG_INFINITY, f64::max);
            let sum_exp: f64 = scaled.iter().map(|&x| (x - max_val).exp()).sum();
            let log_prob = (scaled[j] - max_val) - sum_exp.ln();
            loss_t2i -= log_prob;
        }
        loss_t2i /= n as f64;

        0.5 * (loss_i2t + loss_t2i)
    }

    fn zero_shot_classify(&self, img_emb: &[f64], candidate_texts: &[Vec<f64>]) -> Vec<f64> {
        let logits: Vec<f64> = candidate_texts.iter().map(|txt| dot_product(img_emb, txt) / self.temperature).collect();
        let max_val = logits.iter().cloned().fold(f64::NEG_INFINITY, f64::max);
        let exps: Vec<f64> = logits.iter().map(|&x| (x - max_val).exp()).collect();
        let sum_exp: f64 = exps.iter().sum();
        exps.iter().map(|&e| e / sum_exp).collect()
    }
}

fn main() {
    let clip = SimpleCLIP::new(4, 4, 3, 0.1);

    let raw_images = vec![
        vec![1.0, 0.5, -0.2, 0.8],
        vec![-0.5, 1.2, 0.3, -0.1],
        vec![0.2, -0.8, 1.5, 0.4],
    ];

    let raw_texts = vec![
        vec![0.9, 0.4, -0.1, 0.7],
        vec![-0.4, 1.1, 0.2, -0.2],
        vec![0.1, -0.7, 1.4, 0.5],
    ];

    let img_embeds: Vec<Vec<f64>> = raw_images.iter().map(|img| clip.project_image(img)).collect();
    let txt_embeds: Vec<Vec<f64>> = raw_texts.iter().map(|txt| clip.project_text(txt)).collect();

    let loss = clip.compute_loss(&img_embeds, &txt_embeds);
    println!("InfoNCE Loss (Rust std): {:.4}", loss);

    let probs = clip.zero_shot_classify(&img_embeds[0], &txt_embeds);
    for (i, p) in probs.iter().enumerate() {
        println!("Class {} Prediction: {:.2}%", i, p * 100.0);
    }
}

6. Architectural Comparison: Multimodal Paradigms#

Paradigm	Architecture	Alignment Level	Computational Efficiency	Downstream Flexibility
Dual Encoder (CLIP)	Independent Vision & Text Encoders + Dot Product	Global representation vectors	Extremely fast: embeddings pre-computable for instant index search	Strong zero-shot classification and text-to-image retrieval
Cross-Attention Resampler (Flamingo)	Frozen Vision Encoder + Perceiver Resampler into LLM	Fine-grained patch embeddings conditioned via cross-attention	Moderate: requires full forward pass through large decoder	Multi-image interleaved in-context conversational reasoning
Linear Projection Adapter (LLaVA)	Vision Encoder + 2-layer MLP projection directly into token space	Patches mapped directly as visual tokens	High throughput: visual tokens treated identically to words	Document OCR, visual question answering, spatial grounding
Independent Sigmoid (SigLIP)	Dual Encoder with pairwise Sigmoid classification	Global vectors without batch-wide normalization	Scalable to ultra-large batch sizes with zero multi-GPU all-gather latency	Open-vocabulary vision-language tasks at Google scale

7. Knowledge Check#

Concept check

Question

question: "Why must image and text feature embeddings be L2-normalized before computing cosine similarity in CLIP?"

options:

"(Recommended) To constrain all vectors to the unit hypersphere S^(d-1), preventing vector magnitudes from artificially dominating the dot product similarity score"
"To eliminate the need for matrix multiplication"
"To convert floating point numbers to integers"
"To prevent the text encoder from learning grammar"

explanation: "L2 normalization ensures that ||z_I|| = ||z_T|| = 1. Consequently, z_I . z_T equals the cosine of the angle between them, bounded in [-1, 1]. Without normalization, encoders could arbitrarily increase vector norms to manipulate the Softmax probability distribution."

:::