All StoriesDeep Learning Core

From Chain Rule to Transformers: The 5 Equations Training Trillion-Parameter AI

Step-by-step mathematical anatomy of how Matrix Multiplication, Dot Product, Softmax, Cross-Entropy, and the Chain Rule power modern AI.

\text{Attention}(Q,K,V) = \operatorname{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right)V, \quad \frac{\partial \mathcal{L}}{\partial W_l} = \frac{\partial \mathcal{L}}{\partial a_L}\cdots\frac{\partial a_l}{\partial W_l}

Forward Pass uses Matrix Multiplication and scaled Dot-Product Attention to compute contextual token representations.

Cross-Entropy Loss measures the Kullback-Leibler divergence between predicted token probabilities and ground truth.

Backpropagation applies the Calculus Chain Rule backward through layers to compute exact parameter gradients.

⚡ Formulas Featured in This Article

#L02Dot Product & Cosine Similarity #L04Matrix Multiplication & Linear Transformation #C03Product Rule & Chain Rule (Backpropagation Core)#C08Partial Derivatives & Gradient Vector (∇f)#P09Shannon Entropy & Cross-Entropy Loss #P10Softmax Function & Stochastic Gradient Descent

1. Why Divide by √d_k in Scaled Dot-Product Attention?

\operatorname{Var}(q \cdot k) = d_k \implies \operatorname{Var}\!\left(\frac{q \cdot k}{\sqrt{d_k}}\right) = 1

If q and k have independent unit-variance components, their dot product has variance d_k. Dividing by √d_k restores unit variance, preventing Softmax saturation and vanishing gradients.

2. The Miraculous Derivative of Softmax + Cross-Entropy

\frac{\partial \mathcal{L}_{\mathrm{CE}}}{\partial z_i} = \sigma(\mathbf{z})_i - y_i

Differentiating Cross-Entropy Loss with respect to raw logits z_i cancels out the exponentials and logarithms via the Chain Rule, leaving the remarkably clean error signal (predicted_prob - true_label).