Convolutional Networks & ResNet Skip Connections

Fully connected Multi-Layer Perceptrons treat each pixel in an image independently, ignoring spatial translation equivariance and incurring an explosion in weight parameters ($O(H \cdot W \cdot C)$).

Convolutional Neural Networks (CNNs) enforce spatial locality and weight sharing. To scale depth past dozens of layers without optimization degradation, He et al. introduced Residual Networks (ResNet), opening an unattenuated gradient highway via identity skip connections.

1. 2D Discrete Convolution & Spatial Geometry#

In deep learning, the "convolution" operation is technically 2D discrete cross-correlation between input feature map $I \in \mathbb{R}^{H \times W}$ and kernel filter $K \in \mathbb{R}^{k_h \times k_w}$:

Derivation of Output Spatial Dimensions#

Let $W$ be input width, $K$ be kernel filter width, $P$ be zero-padding applied to both borders, and $S$ be stride.

2. Receptive Field Arithmetic#

The Receptive Field (RF) of a unit in layer $l$ is the spatial region in input image $x_0$ that affects that unit's activation.

Let:

• $r_l$: Receptive field size at layer $l$ (with $r_0 = 1$)
• $k_l$: Kernel filter size at layer $l$
• $s_l$: Stride at layer $l$
• $j_l$: Feature jump (cumulative stride) at layer $l$, where $j_l = j_{l-1} \cdot s_l$ (with $j_0 = 1$)

The receptive field expands recursively:

This explains why stacking two $3 \times 3$ conv layers (each with stride 1) yields an effective receptive field of $1 + (3-1) + (3-1) = 5 \times 5$, while using $2 \cdot (3^2) = 18$ parameters rather than $5^2 = 25$ parameters.

3. Mathematical Proof: The ResNet Gradient Highway#

Prior to 2015, stacking additional layers in deep networks led to the Degradation Problem: a 56-layer network exhibited higher training error and test error than a 20-layer network, despite having higher expressive capacity.

He et al. (2015) hypothesized that optimizing an identity mapping $\mathcal{H}(x) = x$ across multiple non-linear layers is difficult. They reformulated the problem into fitting a residual mapping $\mathcal{F}(x) \equiv \mathcal{H}(x) - x$:

Derivation of Unattenuated Gradient Flow#

Consider a general residual block from layer $l$ to a deeper layer $L$:

Let $\mathcal{E}$ denote the scalar loss function. Applying the multivariate chain rule to differentiate with respect to earlier activation $x_l$:

Differentiating $x_L$ with respect to $x_l$:

Substituting into the chain rule expression:

4. From-Scratch Implementation: Zero Dependencies#

The following implementations construct a 2D Convolution Operator and an Additive Residual Block from scratch in pure Python and pure Rust.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
from typing import List

class Conv2DFromScratch:
    @staticmethod
    def conv2d(
        image: List[List[float]],
        kernel: List[List[float]],
        padding: int = 0,
        stride: int = 1
    ) -> List[List[float]]:
        h = len(image)
        w = len(image[0])
        kh = len(kernel)
        kw = len(kernel[0])

        # Apply zero-padding
        padded_h = h + 2 * padding
        padded_w = w + 2 * padding
        padded = [[0.0] * padded_w for _ in range(padded_h)]
        for r in range(h):
            for c in range(w):
                padded[r + padding][c + padding] = image[r][c]

        # Calculate output spatial shape
        out_h = (h - kh + 2 * padding) // stride + 1
        out_w = (w - kw + 2 * padding) // stride + 1
        output = [[0.0] * out_w for _ in range(out_h)]

        # Slide convolution kernel
        for i in range(out_h):
            for j in range(out_w):
                r_start = i * stride
                c_start = j * stride
                val = 0.0
                for kr in range(kh):
                    for kc in range(kw):
                        val += padded[r_start + kr][c_start + kc] * kernel[kr][kc]
                output[i][j] = val

        return output

    @classmethod
    def residual_block(
        cls,
        x: List[List[float]],
        kernel: List[List[float]]
    ) -> List[List[float]]:
        # F(x): same-padding convolution (padding=1, kernel 3x3)
        fx = cls.conv2d(x, kernel, padding=1, stride=1)
        h = len(x)
        w = len(x[0])

        # Residual Addition: H(x) = ReLU(F(x) + x)
        output = [[0.0] * w for _ in range(h)]
        for r in range(h):
            for c in range(w):
                output[r][c] = max(0.0, fx[r][c] + x[r][c])
        return output

if __name__ == "__main__":
    img = [
        [1.0, 2.0, 3.0, 0.0],
        [4.0, 5.0, 6.0, 1.0],
        [7.0, 8.0, 9.0, 0.0],
        [1.0, 1.0, 1.0, 1.0]
    ]

    # Edge detection kernel
    sobel_kernel = [
        [-1.0, 0.0, 1.0],
        [-2.0, 0.0, 2.0],
        [-1.0, 0.0, 1.0]
    ]

    conv_out = Conv2DFromScratch.conv2d(img, sobel_kernel, padding=0, stride=1)
    print("Conv2D Output (2x2):")
    for row in conv_out:
        print(" ", [round(v, 2) for v in row])

    res_out = Conv2DFromScratch.residual_block(img, sobel_kernel)
    print("\nResidual Block Output (4x4):")
    for row in res_out:
        print(" ", [round(v, 2) for v in row])

5. Active Knowledge Checks#