#P10 · Probability, Statistics & AI · Lv.5View in 3D Mind-Map

Softmax Function & Stochastic Gradient Descent

Augustin-Louis Cauchy, Ludwig Boltzmann & Geoffrey Hinton · 1986 CE

\sigma(\mathbf{z})_i = \frac{e^{z_i / T}}{\sum_{j=1}^K e^{z_j / T}}, \quad \theta_{t+1} = \theta_t - \eta \nabla_\theta \mathcal{L}(\theta_t)

\frac{\partial \mathcal{L}_{\mathrm{CE}}}{\partial z_i} = \sigma(\mathbf{z})_i - y_i

Converts raw neural network logits into a normalized probability distribution (Softmax) and iteratively updates weights opposite to the loss gradient (Gradient Descent).

🧮 Live Formula Calculator & Plotter

Logit z₁ (Target Class)

Logit z₂ (Rival Class)

Learning Rate η

\sigma(z_1) = 81.8\%, \; \frac{\partial \mathcal{L}}{\partial z_1} = -0.182, \; z_1^{(t+1)} = 2.536

정답 소프트맥스 확률 σ(z₁)

81.76%

기울기 ∂L/∂z₁ = σ(z₁)-1

-0.1824

1스텝 업데이트 후 z₁'

2.50 → 2.536

Prerequisite Path (22 STEP)

Full Path Analyzer

Click steps in order to follow the 3D neon path from basics to target.

STEP 1#G03Pi, Radian Measure & Circular Sector

l = r\theta

STEP 2#A02Polynomial Expansion & Factoring

(a+b)^2 = a^2 + 2ab + b^2

STEP 3#G01Pythagorean Theorem

a^2 + b^2 = c^2 \implies c = \sqrt{a^2 + b^2}

STEP 4#P01Permutations & Combinations

{}_nP_r = \frac{n!}{(n-r)!}

STEP 5#A03Laws of Exponents & Radicals

a^m \cdot a^n = a^{m+n}

STEP 6#L01Vector Operations, L1/L2 Norms & Normalization

\|\mathbf{v}\|_2 = \sqrt{\sum_{i=1}^{n} v_i^2}

STEP 7#A01Linear Equation & Balance Axiom

ax + b = 0 \implies x = -\frac{b}{a}

STEP 8#G04Unit Circle Trigonometry & Identities

\cos^2\theta + \sin^2\theta = 1

STEP 9#G022D/3D Euclidean Distance Formula

d(P

STEP 10#G05Law of Sines & Law of Cosines

c^2 = a^2 + b^2 - 2ab\cos C

STEP 11#A06Logarithm Laws & Change of Base

\log_a(xy) = \log_a x + \log_a y

STEP 12#A08Binomial Theorem & Pascal's Triangle

(a+b)^n = \sum_{k=0}^{n} \binom{n}{k} a^{n-k} b^k

STEP 13#C02Derivative Definition & Instantaneous Rate

f'(x) = \frac{df}{dx} = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}

STEP 14#L02Dot Product & Cosine Similarity

\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^{n} a_i b_i = \|\mathbf{a}\|\|\mathbf{b}\|\cos\theta

STEP 15#P04Expected Value & Variance

E[X] = \sum_i x_i p_i

STEP 16#P02Conditional Probability & Chain Rule

P(A \mid B) = \frac{P(A \cap B)}{P(B)}

STEP 17#C01Limits, Continuity & Euler's Number e

e = \lim_{n \to \infty} \left(1 + \frac{1}{n}\right)^n \approx 2.718281828

STEP 18#L04Matrix Multiplication & Linear Transformation

(AB)_{ij} = \sum_{k=1}^{n} A_{ik} B_{kj}

STEP 19#C03Product Rule & Chain Rule (Backpropagation Core)

\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}

STEP 20#C08Partial Derivatives & Gradient Vector (∇f)

\nabla f = \left(\frac{\partial f}{\partial x_1}

STEP 21#P09Shannon Entropy & Cross-Entropy Loss

H(P

STEP 22#P10Softmax Function & Stochastic Gradient Descent

\sigma(\mathbf{z})_i = \frac{e^{z_i / T}}{\sum_{j=1}^K e^{z_j / T}}

Geometric & Intuitive Insight

소프트맥스와 교차 엔트로피를 합성해 미분하면 복잡한 지수·로그가 기적처럼 약분되어 '예측확률 - 정답(ŷ - y)'이라는 가장 아름다운 오차 신호가 나옵니다.

Step-by-Step Derivation

Step 1.로짓 벡터 z에 지수함수를 취해 양수로 만든 뒤 총합 ∑e^(z_j/T)로 나누어 총합 1인 확률분포 σ(z)를 만듭니다.
Step 2.교차 엔트로피 L = -∑ y_k ln σ(z)_k를 z_i로 편미분(연쇄법칙)하면 ∂L/∂z_i = σ(z)_i - y_i가 유도됩니다.
Step 3.이 기울기에 학습률 η를 곱해 가중치에서 빼주면 θ_{t+1} = θ_t - η∇L(θ_t)로 손실이 감소합니다.

AI & Coding Real-World Use

The beating heart of Transformer Self-Attention, LLM Temperature sampling, and all deep learning optimizers (SGD, AdamW).

#Transformer Attention#Deep Learning Optimizer#LLM Temperature#Backpropagation

const softmax = (zs, T=1) => { const exps = zs.map(z => Math.exp(z/T)); const s = exps.reduce((a,b)=>a+b,0); return exps.map(e => e/s); };

Probability, Statistics & AI — Related Formulas in This Cluster

Permutations & Combinations

Lv.1 · Combinatorics

#P01

Conditional Probability & Chain Rule

Lv.2 · LLM Next-Token

Expected Value & Variance

Lv.2 · Reinforcement Learning

#P04

Binomial Distribution

Lv.2 · A/B Testing

#P05

Normal (Gaussian) Distribution & Z-Score

Lv.3 · Diffusion Models

#P06