Recurrent Modeling: BPTT, LSTM Gates & GRU | AI & Machine Learning Engineering

Sequential data (natural language, time series, financial trajectories) possesses temporal dependencies where the probability of token $x_t$ depends on preceding history $(x_1, \dots, x_{t-1})$ .

Standard Recurrent Neural Networks (RNNs) suffer exponential gradient degradation during Backpropagation Through Time. To capture long-range temporal horizons, Hochreiter and Schmidhuber introduced the Constant Error Carousel (CEC) inside the Long Short-Term Memory (LSTM) architecture.

Constant Error CarouselRecurrent State

dC_t / dC_(t-1) = f_t

The linear cell state preserves backpropagated gradients over hundreds of timesteps without exponential decay.

1. Backpropagation Through Time (BPTT) Gradient Decay#

In a standard vanilla RNN, hidden state $h_t \in \mathbb{R}^d$ evolves according to:

h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b)

Architecture Diagram

Rendering diagram...

Let total loss over $T$ steps be $\mathcal{L} = \sum_{t=1}^T \mathcal{L}_t$ .

Applying the chain rule to compute the gradient with respect to recurrence weight matrix $W_{hh}$ :

\frac{\partial \mathcal{L}_T}{\partial W_{hh}} = \sum_{k=1}^T \frac{\partial \mathcal{L}_T}{\partial h_T} \frac{\partial h_T}{\partial h_k} \frac{\partial h_k}{\partial W_{hh}}

Evaluating the temporal state Jacobian product:

\frac{\partial h_T}{\partial h_k} = \prod_{j=k+1}^T \frac{\partial h_j}{\partial h_{j-1}} = \prod_{j=k+1}^T \text{diag}\Big( 1 - h_j^2 \Big) W_{hh}^T

The Exponential Vanishing Trap

The term

\text{diag}(1 - h_j^2)

represents the derivative of the

\tanh

activation, which is strictly bounded in

(0, 1]

. If the largest eigenvalue of weight matrix

W_{hh}

satisfies

\lambda_{\max} < 1

, the continuous matrix multiplication

\prod_{j=k+1}^T W_{hh}^T

collapses to zero exponentially:

\lim_{T - k \to \infty} \left\| \frac{\partial h_T}{\partial h_k} \right\| = 0

Early token inputs

x_k

receive zero gradient updates, blinding the model to long-range context.

2. LSTM Architecture & The Constant Error Carousel#

The Long Short-Term Memory (LSTM) cell resolves vanishing gradients by separating working memory from long-term memory:

Hidden State ( $h_t$ ): Short-term working memory.
Cell State ( $C_t$ ): Long-term memory flowing through a linear additive highway.

Architecture Diagram

Rendering diagram...

The Six Governing Equations#

Let $[h_{t-1}, x_t] \in \mathbb{R}^{d_h + d_x}$ denote vector concatenation.

Step 1: Forget Gate Vector ($f_t$)

Decides what fraction of historical memory to discard:

f_t = \sigma\Big( W_f [h_{t-1}, x_t] + b_f \Big) \in (0, 1)^{d_h}

Step 2: Input Gate Vector ($i_t$)

Decides which coordinates of incoming information to update:

i_t = \sigma\Big( W_i [h_{t-1}, x_t] + b_i \Big) \in (0, 1)^{d_h}

Step 3: Candidate Cell State ($\tilde{C}_t$)

Generates new candidate memory values bounded in $(-1, 1)$ :

\tilde{C}_t = \tanh\Big( W_c [h_{t-1}, x_t] + b_c \Big) \in (-1, 1)^{d_h}

Step 4: Cell State Accumulation ($C_t$)

Combines preserved past memory with modulated new candidates:

C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t

Step 5: Output Gate Vector ($o_t$)

Filters which components of cell state to emit to the hidden output:

o_t = \sigma\Big( W_o [h_{t-1}, x_t] + b_o \Big) \in (0, 1)^{d_h}

Step 6: Final Hidden State ($h_t$)

Emits the activated working representation:

h_t = o_t \odot \tanh(C_t)

Proof of the Constant Error Carousel (CEC)#

Consider the partial derivative of current cell state $C_t$ with respect to previous cell state $C_{t-1}$ :

\frac{\partial C_t}{\partial C_{t-1}} = f_t

Unlike vanilla RNNs where gradients pass through non-linear matrix products, the gradient through the cell state is simply the value of the forget gate $f_t$ .

When $f_t \approx 1$ , the gradient propagates completely unaltered ( $\frac{\partial C_t}{\partial C_{t-1}} \approx 1$ ) across arbitrary time gaps without diminishing.

3. Gated Recurrent Unit (GRU)#

Cho et al. (2014) introduced the Gated Recurrent Unit (GRU), merging cell state $C_t$ and hidden state $h_t$ into a single state vector to reduce parameter count:

Reset Gate ( $r_t$ ): Controls how much past history to forget:

r_t = \sigma\Big( W_r [h_{t-1}, x_t] + b_r \Big)

Update Gate ( $z_t$ ): Acts simultaneously as both forget and input gate:

z_t = \sigma\Big( W_z [h_{t-1}, x_t] + b_z \Big)

Candidate Hidden State ( $\tilde{h}_t$ ):

\tilde{h}_t = \tanh\Big( W_h [r_t \odot h_{t-1}, x_t] + b_h \Big)

Linear Interpolation Update:

h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t

4. From-Scratch Implementation: Zero Dependencies#

The following implementations build a full LSTM Cell Forward Pass from scratch in pure Python and pure Rust.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
import math
from typing import List, Tuple

class LSTMCellFromScratch:
    def __init__(self, input_dim: int, hidden_dim: int):
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        concat_dim = input_dim + hidden_dim

        # Initialize weights for all 4 gates with small values
        scale = 0.1
        self.Wf = [[scale for _ in range(concat_dim)] for _ in range(hidden_dim)]
        self.Wi = [[scale for _ in range(concat_dim)] for _ in range(hidden_dim)]
        self.Wc = [[scale for _ in range(concat_dim)] for _ in range(hidden_dim)]
        self.Wo = [[scale for _ in range(concat_dim)] for _ in range(hidden_dim)]

        # Initialize forget bias to +1.0 (Jozefowicz et al. 2015 recommendation)
        self.bf = [1.0] * hidden_dim
        self.bi = [0.0] * hidden_dim
        self.bc = [0.0] * hidden_dim
        self.bo = [0.0] * hidden_dim

    @staticmethod
    def _sigmoid(x: float) -> float:
        if x < -700.0: return 0.0
        if x > 700.0: return 1.0
        return 1.0 / (1.0 + math.exp(-x))

    @staticmethod
    def _tanh(x: float) -> float:
        return math.tanh(x)

    def _mat_vec_add(self, W: List[List[float]], v: List[float], b: List[float]) -> List[float]:
        return [sum(W[i][j] * v[j] for j in range(len(v))) + b[i] for i in range(len(W))]

    def forward(
        self,
        x_t: List[float],
        h_prev: List[float],
        c_prev: List[float]
    ) -> Tuple[List[float], List[float]]:
        # Concatenate [h_(t-1), x_t]
        concat_input = h_prev + x_t

        # 1. Forget Gate: f_t = sigma(W_f * input + b_f)
        f_logits = self._mat_vec_add(self.Wf, concat_input, self.bf)
        f_t = [self._sigmoid(val) for val in f_logits]

        # 2. Input Gate: i_t = sigma(W_i * input + b_i)
        i_logits = self._mat_vec_add(self.Wi, concat_input, self.bi)
        i_t = [self._sigmoid(val) for val in i_logits]

        # 3. Candidate Cell: c_tilde = tanh(W_c * input + b_c)
        c_logits = self._mat_vec_add(self.Wc, concat_input, self.bc)
        c_tilde = [self._tanh(val) for val in c_logits]

        # 4. Cell State: C_t = f_t * C_(t-1) + i_t * c_tilde
        c_t = [f_t[j] * c_prev[j] + i_t[j] * c_tilde[j] for j in range(self.hidden_dim)]

        # 5. Output Gate: o_t = sigma(W_o * input + b_o)
        o_logits = self._mat_vec_add(self.Wo, concat_input, self.bo)
        o_t = [self._sigmoid(val) for val in o_logits]

        # 6. Hidden State: h_t = o_t * tanh(C_t)
        h_t = [o_t[j] * self._tanh(c_t[j]) for j in range(self.hidden_dim)]

        return h_t, c_t

if __name__ == "__main__":
    lstm = LSTMCellFromScratch(input_dim=3, hidden_dim=2)
    h_init = [0.0, 0.0]
    c_init = [0.0, 0.0]

    seq = [[1.0, 0.5, -0.2], [0.8, -0.4, 0.9]]
    h, c = h_init, c_init
    for step, x in enumerate(seq):
        h, c = lstm.forward(x, h, c)
        print(f"Step {step+1}:")
        print(f"  Hidden State: {[round(v, 4) for v in h]}")
        print(f"  Cell State:   {[round(v, 4) for v in c]}")

5. Active Knowledge Checks#

Concept check

Why does the derivative dC_t / dC_(t-1) = f_t in an LSTM prevent the vanishing gradient problem that plagues vanilla RNNs?

Concept check

Why did Jozefowicz et al. (2015) famously recommend initializing the LSTM forget gate bias b_f to +1.0 or +2.0 at the beginning of training?