Diffusion Models & DDPM Mathematics

Generative Adversarial Networks (GANs) rely on an adversarial minimax equilibrium between generator and discriminator, which is prone to training instability, mode collapse, and vanishing gradients.

Denoising Diffusion Probabilistic Models (DDPM) (Sohl-Dickstein et al. 2015, Ho et al. 2020) model generation as a gradual stochastic thermodynamic process: systematically destroying signal via a forward Gaussian diffusion process and training a neural network to invert the trajectory step by step.

1. The Forward (Noising) Diffusion Process#

Given clean data distribution $x_0 \sim q(x_0)$, the forward diffusion process adds small Gaussian noise across $T$ discrete timesteps according to variance schedule $\beta_1, \beta_2, \dots, \beta_T \in (0, 1)$:

At timestep $T \approx 1000$, all original structure is eradicated, and $x_T$ is indistinguishable from standard isotropic Gaussian noise $\mathcal{N}(0, I)$.

2. Derivation of the Closed-Form Arbitrary Timestep Sample#

Simulating $t$ forward transitions step by step would require $t$ sequential computations. We derive how to sample $x_t$ directly from $x_0$ in $O(1)$ constant time.

Let $\alpha_t = 1 - \beta_t$ and define cumulative product $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$.

This establishes that $q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t} x_0, (1 - \bar{\alpha}_t) I)$, allowing instantaneous noising at arbitrary timesteps during training.

3. Reverse Process & The Simplified Loss Objective#

The true reverse conditional distribution $q(x_{t-1} | x_t, x_0)$ is a Gaussian whose mean evaluates via Bayes' Rule to:

Substituting $x_0 = \frac{x_t - \sqrt{1 - \bar{\alpha}_t}\epsilon}{\sqrt{\bar{\alpha}_t}}$ eliminates $x_0$, leaving the reverse mean as a function of the injected noise $\epsilon$:

Rather than predicting the clean image $x_0$ directly, the neural network $\epsilon_\theta(x_t, t)$ is trained to predict the noise vector $\epsilon$ added at step $t$:

Ho et al. proved that omitting the complex variational weighting coefficients produces superior sample visual quality, yielding the Simplified DDPM Loss:

4. Classifier-Free Guidance (CFG)#

To direct generation using text conditioning $c$ without training a separate noisy image classifier, Ho and Salimans (2021) introduced Classifier-Free Guidance (CFG).

During training, condition $c$ is dropped randomly (replaced by null condition $\emptyset$) with probability $p_{\text{uncond}} \approx 10\%$.

During inference sampling, the predicted noise is computed as an extrapolation between conditional and unconditional estimates:

Where $s \ge 1.0$ is the Guidance Scale:

• When $s = 1.0$: Standard conditional generation.
• When $s > 1.0$: Amplifies alignment with prompt $c$ by pushing generation along the gradient direction $\epsilon_\theta(x_t, c) - \epsilon_\theta(x_t, \emptyset)$ away from unconditional space.

5. From-Scratch Implementation: Zero Dependencies#

The following implementations compute the DDPM Closed-Form Forward Transition and the Reverse Sampling Step from scratch in pure Python and pure Rust.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
import math
import random
from typing import List, Tuple

class DDPMPrimitives:
    def __init__(self, timesteps: int = 100, beta_start: float = 0.0001, beta_end: float = 0.02):
        self.timesteps = timesteps
        # Linear variance schedule
        self.betas = [
            beta_start + (beta_end - beta_start) * (t / (timesteps - 1))
            for t in range(timesteps)
        ]
        self.alphas = [1.0 - b for b in self.betas]

        # Cumulative products: alpha_bar_t
        self.alpha_bars = []
        prod = 1.0
        for a in self.alphas:
            prod *= a
            self.alpha_bars.append(prod)

    def q_sample(self, x_0: List[float], t: int) -> Tuple[List[float], List[float]]:
        # x_t = sqrt(alpha_bar_t) * x_0 + sqrt(1 - alpha_bar_t) * epsilon
        alpha_bar = self.alpha_bars[t]
        sqrt_alpha_bar = math.sqrt(alpha_bar)
        sqrt_one_minus_alpha_bar = math.sqrt(1.0 - alpha_bar)

        noise = [random.gauss(0.0, 1.0) for _ in range(len(x_0))]
        x_t = [
            sqrt_alpha_bar * x_0[i] + sqrt_one_minus_alpha_bar * noise[i]
            for i in range(len(x_0))
        ]
        return x_t, noise

    def p_sample_step(
        self,
        x_t: List[float],
        t: int,
        predicted_noise: List[float]
    ) -> List[float]:
        # Reverse step: mu_theta = (1 / sqrt(alpha_t)) * (x_t - (beta_t / sqrt(1 - alpha_bar_t)) * eps_pred)
        beta_t = self.betas[t]
        alpha_t = self.alphas[t]
        alpha_bar_t = self.alpha_bars[t]

        coeff = 1.0 / math.sqrt(alpha_t)
        noise_coeff = beta_t / math.sqrt(1.0 - alpha_bar_t)

        mean = [
            coeff * (x_t[i] - noise_coeff * predicted_noise[i])
            for i in range(len(x_t))
        ]

        if t == 0:
            return mean

        # Add stochastic noise sigma_t * z
        sigma_t = math.sqrt(beta_t)
        return [mean[i] + sigma_t * random.gauss(0.0, 1.0) for i in range(len(x_t))]

if __name__ == "__main__":
    random.seed(42)
    ddpm = DDPMPrimitives(timesteps=100)

    # 4-dimensional clean representation x_0
    x_clean = [1.0, -1.0, 0.5, -0.5]
    print("Original Clean Signal x_0:", x_clean)

    # Directly jump to timestep 50 in O(1) time
    x_50, noise = ddpm.q_sample(x_clean, t=50)
    print("\nNoisy Sample at t=50 (Closed-Form):", [round(v, 4) for v in x_50])

    # Simulate reverse step
    x_49 = ddpm.p_sample_step(x_50, t=50, predicted_noise=noise)
    print("Denoised Step to t=49:", [round(v, 4) for v in x_49])

6. Active Knowledge Checks#