Linear Classification & Logistic Regression

Linear classification forms the bedrock of parametric discriminative modeling in modern machine learning. While linear regression directly predicts an unbounded real-valued response $\hat{y} \in \mathbb{R}$, classification requires constraining predictions to a bounded probability interval $\hat{p} \in [0, 1]$.

Logistic Regression bridges this gap not by fitting non-linear curves directly, but by asserting that the log-odds of the target class is a linear combination of input features.

1. The Log-Odds Ratio and Sigmoid Derivation#

To understand why the Sigmoid function appears ubiquitous in machine learning, we derive it directly from first principles by modeling event odds.

Step-by-Step Derivation of the Sigmoid Function#

Let $p = P(y = 1 | x)$ denote the conditional probability that an instance belongs to the positive class.

This establishes that the standard Sigmoid function $\sigma(z) = \frac{1}{1 + e^{-z}}$ is the algebraic inverse of the linear log-odds model.

2. Mathematical Derivative of the Sigmoid Function#

During gradient descent, we must compute the derivative of $\sigma(z)$ with respect to $z$. We derive this using the quotient rule:

Applying the power and chain rules:

Substitute the internal derivative back into the expression:

Decompose the fraction into two factors:

Observe that $\frac{e^{-z}}{1 + e^{-z}} = \frac{(1 + e^{-z}) - 1}{1 + e^{-z}} = 1 - \frac{1}{1 + e^{-z}} = 1 - \sigma(z)$. Therefore:

3. Deriving Binary Cross-Entropy via Maximum Likelihood#

Why do we not optimize Logistic Regression using Mean Squared Error (MSE)? Under MSE with non-linear sigmoid activations, the loss surface becomes non-convex, with saturated plateaus where gradients vanish.

Instead, we formulate the loss function using Maximum Likelihood Estimation (MLE).

Step-by-Step MLE Derivation#

Assume binary labels $y_i \in \{0, 1\}$ drawn independently from a Bernoulli distribution conditioned on input $x_i$:

Where $\hat{y}_i = \sigma(w^T x_i + b)$.

4. Analytical Gradient Vector Derivation#

To update parameters via gradient descent:

We apply the multivariate chain rule for a single training example $(x_i, y_i)$ with linear logit $z_i = w^T x_i + b$:

Let us compute each partial derivative individually:

1. Loss with respect to activation:

1. Activation with respect to logit:

1. Logit with respect to weight parameter:

Multiply the three partial derivatives:

Notice how the non-linear denominator $\hat{y}_i(1 - \hat{y}_i)$ cancels out:

Summing over all $N$ training samples yields the exact gradient vector:

5. From-Scratch Implementation: Zero Dependencies#

The following implementations demonstrate pure mathematical logistic regression in Python and Rust without relying on any external packages like NumPy, PyTorch, or ndarray.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
import math
from typing import List, Tuple

class LogisticRegressionFromScratch:
    def __init__(self, learning_rate: float = 0.1, num_iterations: int = 1000):
        self.learning_rate = learning_rate
        self.num_iterations = num_iterations
        self.weights: List[float] = []
        self.bias: float = 0.0

    @staticmethod
    def _sigmoid(z: float) -> float:
        # Prevent math overflow for extreme negative logits
        if z < -700.0:
            return 0.0
        if z > 700.0:
            return 1.0
        return 1.0 / (1.0 + math.exp(-z))

    def fit(self, X: List[List[float]], y: List[float]) -> None:
        num_samples = len(X)
        num_features = len(X[0])
        
        # Initialize parameters to zeros
        self.weights = [0.0] * num_features
        self.bias = 0.0

        for _ in range(self.num_iterations):
            # Gradient accumulators
            dw = [0.0] * num_features
            db = 0.0

            for i in range(num_samples):
                # Compute linear score: z = w^T x + b
                z = sum(self.weights[j] * X[i][j] for j in range(num_features)) + self.bias
                y_hat = self._sigmoid(z)
                error = y_hat - y[i]

                # Accumulate gradients
                for j in range(num_features):
                    dw[j] += error * X[i][j]
                db += error

            # Apply gradient descent update
            for j in range(num_features):
                self.weights[j] -= (self.learning_rate / num_samples) * dw[j]
            self.bias -= (self.learning_rate / num_samples) * db

    def predict_proba(self, X: List[List[float]]) -> List[float]:
        num_features = len(self.weights)
        probabilities = []
        for row in X:
            z = sum(self.weights[j] * row[j] for j in range(num_features)) + self.bias
            probabilities.append(self._sigmoid(z))
        return probabilities

    def predict(self, X: List[List[float]], threshold: float = 0.5) -> List[int]:
        return [1 if p >= threshold else 0 for p in self.predict_proba(X)]

# Verification on toy separable dataset
if __name__ == "__main__":
    X_train = [[0.5, 1.5], [1.0, 1.0], [1.5, 2.5], [3.0, 3.5], [3.5, 4.0], [4.0, 3.0]]
    y_train = [0.0, 0.0, 0.0, 1.0, 1.0, 1.0]

    model = LogisticRegressionFromScratch(learning_rate=0.5, num_iterations=500)
    model.fit(X_train, y_train)
    preds = model.predict(X_train)
    print(f"Learned weights: {model.weights}")
    print(f"Learned bias: {model.bias:.4f}")
    print(f"Training predictions: {preds}")

6. Active Knowledge Checks#

Test your comprehension of the mathematical derivations and optimization landscape: