#P09 · Probability, Statistics & AI · Lv.4View in 3D Mind-Map

Shannon Entropy & Cross-Entropy Loss

Claude Shannon, Solomon Kullback & Richard Leibler · 1948 CE

H(P, Q) = -\sum_x P(x) \log Q(x) = H(P) + D_{\mathrm{KL}}(P \parallel Q)

\mathcal{L}_{\mathrm{BCE}} = -\big[y \ln \hat{y} + (1-y)\ln(1-\hat{y})\big]

Measures average information uncertainty (Shannon entropy) and the divergence between true distribution P and predicted model distribution Q.

🧮 Live Formula Calculator & Plotter

True Prob p

Predicted Prob q

H(P,Q) = 0.4927, \quad D_{\mathrm{KL}}(P\parallel Q) = 0.1676\text{ nats}

교차 엔트로피 손실 H(P,Q)

0.4927

KL 발산 D_KL(P||Q)

0.1676

자체 엔트로피 H(P)

0.3251

Prerequisite Path (9 STEP)

Full Path Analyzer

Click steps in order to follow the 3D neon path from basics to target.

STEP 1#A01Linear Equation & Balance Axiom

ax + b = 0 \implies x = -\frac{b}{a}

STEP 2#A02Polynomial Expansion & Factoring

(a+b)^2 = a^2 + 2ab + b^2

STEP 3#A03Laws of Exponents & Radicals

a^m \cdot a^n = a^{m+n}

STEP 4#P01Permutations & Combinations

{}_nP_r = \frac{n!}{(n-r)!}

STEP 5#A08Binomial Theorem & Pascal's Triangle

(a+b)^n = \sum_{k=0}^{n} \binom{n}{k} a^{n-k} b^k

STEP 6#A06Logarithm Laws & Change of Base

\log_a(xy) = \log_a x + \log_a y

STEP 7#P04Expected Value & Variance

E[X] = \sum_i x_i p_i

STEP 8#P02Conditional Probability & Chain Rule

P(A \mid B) = \frac{P(A \cap B)}{P(B)}

STEP 9#P09Shannon Entropy & Cross-Entropy Loss

H(P

🚀 Next Unlocks (Derived Formulas) (1)

#P10Softmax Function & Stochastic Gradient Descent

Geometric & Intuitive Insight

드문 사건이 일어날수록 놀람의 정도(정보량 -log P)가 크며, AI가 정답에 낮은 확률 Q를 부여할수록 교차 엔트로피 벌점이 로그 함수로 폭발합니다.

Step-by-Step Derivation

Step 1.독립 사건의 정보량이 덧셈(I(p₁p₂) = I(p₁)+I(p₂))을 만족하려면 정보량은 I(p) = -log(p) 형태여야 합니다.
Step 2.실제 분포 P로 예측 분포 Q의 정보량 기댓값을 구하면 교차 엔트로피 H(P,Q) = -∑ P(x) log Q(x)가 됩니다.
Step 3.H(P,Q) - H(P) = ∑ P(x) log(P(x)/Q(x)) = D_KL(P || Q) ≥ 0 이므로 Q = P일 때 손실이 최소가 됩니다.

AI & Coding Real-World Use

The universal training loss function (Cross-Entropy Loss) for LLMs, image classifiers, and decision tree Information Gain.

#Cross-Entropy Loss#LLM Pretraining#Information Theory#KL Divergence

const bceLoss = (y, yHat) => -(y * Math.log(yHat + 1e-9) + (1 - y) * Math.log(1 - yHat + 1e-9));

Probability, Statistics & AI — Related Formulas in This Cluster

Permutations & Combinations

Lv.1 · Combinatorics

Conditional Probability & Chain Rule

Lv.2 · LLM Next-Token

Lv.3 · Spam Filter

Expected Value & Variance

Lv.2 · Reinforcement Learning

Binomial Distribution

Lv.2 · A/B Testing

Normal (Gaussian) Distribution & Z-Score

Lv.3 · Diffusion Models