返回故事列表Deep Learning Core

從鏈式法則到Transformer：訓練萬億參數AI的5個核心數學公式

Step-by-step mathematical anatomy of how Matrix Multiplication, Dot Product, Softmax, Cross-Entropy, and the Chain Rule power modern AI.

Attention(Q,K,V)=softmax⁡ ⁣(QKTdk)V,∂L∂Wl=∂L∂aL⋯∂al∂Wl\text{Attention}(Q,K,V) = \operatorname{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right)V, \quad \frac{\partial \mathcal{L}}{\partial W_l} = \frac{\partial \mathcal{L}}{\partial a_L}\cdots\frac{\partial a_l}{\partial W_l}
Forward Pass uses Matrix Multiplication and scaled Dot-Product Attention to compute contextual token representations.
Cross-Entropy Loss measures the Kullback-Leibler divergence between predicted token probabilities and ground truth.
Backpropagation applies the Calculus Chain Rule backward through layers to compute exact parameter gradients.
⚡ 本文涉及的核心數學公式
#L02向量點積(內積)與餘弦相似度#L04矩陣乘法與線性變換複合 (GEMM)#C03乘積法則與複合函數鏈式法則 (Chain Rule)#C08偏導數與梯度向量 (∇f 多元微積分)#P09香農資訊熵與交叉熵損失 (Cross-Entropy)#P10Softmax函數與隨機梯度下降 (SGD)

1. Why Divide by √d_k in Scaled Dot-Product Attention?

Var⁡(q⋅k)=dk  ⟹  Var⁡ ⁣(q⋅kdk)=1\operatorname{Var}(q \cdot k) = d_k \implies \operatorname{Var}\!\left(\frac{q \cdot k}{\sqrt{d_k}}\right) = 1

If q and k have independent unit-variance components, their dot product has variance d_k. Dividing by √d_k restores unit variance, preventing Softmax saturation and vanishing gradients.

2. The Miraculous Derivative of Softmax + Cross-Entropy

∂LCE∂zi=σ(z)i−yi\frac{\partial \mathcal{L}_{\mathrm{CE}}}{\partial z_i} = \sigma(\mathbf{z})_i - y_i

Differentiating Cross-Entropy Loss with respect to raw logits z_i cancels out the exponentials and logarithms via the Chain Rule, leaving the remarkably clean error signal (predicted_prob - true_label).