CalculusLogarithm#007

Logarithm

로그 함수

The log, the inverse of exp, turns products into sums. Multiplying 0.9, 0.5, 0.01 again and again in float64 hits 0 at factor 414; the log sum stays −745.7. Covers the log rules, ln vs. log, and log-sum-exp.

Key symbol
log
Reading time
13 min
Prerequisites
Exponential Function

Translated by an agent and not reviewed yet. Original: 한국어

Intuition — turning multiplication into adding counts of 0s

Intuition

For 100 × 1,000 you only need to count the 0s. Multiply a number with two 0s by one with three and you get 100,000, with five 0s. Multiplication has turned into adding up counts of 0s.

Here the number of 0s is the logarithm, and 10 is the base of that logarithm. The logarithm extends this 'number of 0s' to every positive number. For 2 the count of 0s is about 0.301 (log⁡102≈0.301\log_{10} 2 \approx 0.301), and 0.01, which is 1 divided by 100, has had two 0s erased, so its count is −2. In AI the base is e≈2.718e \approx 2.718 instead of 10, and this is called the natural logarithm.

The logarithm undoes the exponential function

log⁡bx\log_b x asks to what power the base bb must be raised to give xx (xx is called the argument). So the logarithm is the function that undoes the exponential function byb^y (its inverse function).

y=log⁡bx  ⟺  by=x(b>0, b≠1, x>0)(1)y = \log_b x \iff b^y = x \qquad (b > 0,\ b \ne 1,\ x > 0) \tag{1}

At first, the top five rows of the table below are enough. The remaining rows are used in later sections.

Definition
SymbolNameMeaning
log⁡bx\log_b xlogarithm with base bbThe power to which bb must be raised to give xx. Defined only for x>0x > 0. Viewed as a function of xx, it is the logarithmic function
bbbaseUsually ee, 2, or 10
log⁡x\log xnatural logarithmThe logarithm with base ee. Also written ln⁡x\ln x. This is the log of this site and of NumPy and PyTorch; for base 2 or 10 we write log⁡2\log_2 or log⁡10\log_{10}
eeEuler's numbere≈2.718e \approx 2.718
exe^x, exp⁡(x)\exp(x)exponential functionee raised to the power xx. The inverse function of log⁡\log
pp, pip_iprobabilityThe probability the model gave to the correct answer, and that probability for the iith example. 0<p≤10 < p \le 1
nnnumber of examplesHow many probabilities are multiplied or added
LLlossThe mean of −log⁡-\log of the correct-answer probabilities. Smaller is better
z\mathbf{z}, zjz_j, KKlogit, number of classesThe KK per-class scores that go into the softmax
ss, tt, rrpositive number, real numberAny positive numbers ss, tt and any real number rr put into Eqs. (2) and (3)
mmmaximum logitm=max⁡jzjm = \max_j z_j. Subtracted first in Eq. (8)
θ\theta, f(θ)f(\theta)parameter, likelihoodIn the coin example, the probability of heads, and the probability that the observed result, with the order of the tosses fixed, comes out at that θ\theta

In many school textbooks, log⁡x\log x with no base is the common logarithm (base 10), and the natural logarithm is written ln⁡x\ln x. On this page, log⁡\log is the natural logarithm.

With b=eb = e, log⁡\log and exp⁡\exp undo each other: log⁡ex=x\log e^x = x and elog⁡x=xe^{\log x} = x. So the two curves in the left panel of Figure 1 face each other like mirror images across the diagonal y=xy = x. The points (0,1)(0, 1) and (1,e)(1, e) on exe^x become (1,0)(1, 0) and (e,1)(e, 1) on log⁡x\log x.

Left: the exponential function e^x (gray) and the natural logarithm log x (blue) are symmetric about y = x (dashed). The points (0, 1) and (1, 0), and (1, e) and (e, 1), are pairs. Right: -log p for 0 < p ≤ 1. For p = 0.9, 0.5, 0.01 it is 0.105, 0.693, 4.605; it is 0 at p = 1 and grows without bound as p approaches 0.
Figure 1 Left: the exponential function exe^x (gray) and the natural logarithm log⁡x\log x (blue) are symmetric about y=xy = x (dashed). The points (0,1)(0, 1) and (1,0)(1, 0), and (1,e)(1, e) and (e,1)(e, 1), are pairs. Right: −log⁡p-\log p for 0<p≤10 < p \le 1. For p=0.9p = 0.9, 0.50.5, 0.010.01 it is 0.105, 0.693, 4.605; it is 0 at p=1p = 1 and grows without bound as pp approaches 0. figures/log_curves.py

Three things are visible right away in the figure. The logarithm is defined only for x>0x > 0, log⁡1=0\log 1 = 0, and as xx approaches 0 it falls to −∞-\infty. So a probability between 0 and 1 has a negative log, and −log⁡p-\log p, with the sign flipped, is 0 or more.

The logarithm turns products into sums

The logarithm turns products into sums, division into subtraction, and powers into multiplication (the 'laws of logarithms' in textbooks). Each of these is the exponent law euev=eu+ve^u e^v = e^{u+v} (multiplying by ee uu times and then vv more times is multiplying by it u+vu+v times) read backward. In the equations below, ss and tt are positive numbers and rr is a real number.

log⁡(st)=log⁡s+log⁡t(2)\log(st) = \log s + \log t \tag{2} log⁡st=log⁡s−log⁡t,log⁡sr=rlog⁡s(3)\log \frac{s}{t} = \log s - \log t, \qquad \log s^r = r \log s \tag{3}
Why Eq. (2) holds

Set log⁡s=u\log s = u and log⁡t=v\log t = v; then by Eq. (1), s=eus = e^u and t=evt = e^v. So st=euev=eu+vst = e^u e^v = e^{u+v}, and by Eq. (1) again, log⁡(st)=u+v\log(st) = u + v. Division works the same way with eu/ev=eu−ve^u / e^v = e^{u-v}, and powers with (eu)r=eru(e^u)^r = e^{ru}.

Let's check with three probabilities, 0.9, 0.5, 0.01. The product is 0.9×0.5×0.01=0.00450.9 \times 0.5 \times 0.01 = 0.0045, and log⁡0.0045≈−5.404\log 0.0045 \approx -5.404. Adding the individual logs −0.105-0.105, −0.693-0.693, −4.605-4.605, using the values before rounding, also gives −5.4037-5.4037, that is, about −5.404-5.404. With nn terms, applying Eq. (2) again and again turns the product ∏\prod into the sum ∑\sum (the symbols from the Sigma Notation page).

log⁡∏i=1npi=∑i=1nlog⁡pi(4)\log \prod_{i=1}^{n} p_i = \sum_{i=1}^{n} \log p_i \tag{4}

Why add logs instead of multiplying probabilities

Because when you multiply many probabilities, the computer rounds the result to 0 (underflow). Multiply 0.9, 0.5, 0.01 over and over in the computer's 64-bit floating-point numbers (float64), and the product becomes 0 at the 414th factor, while the sum of logs remains, at −745.7. The absolute value of the sum of logs only grows as n×n \times (the mean −log⁡p-\log p), so it does not vanish.

Say the probabilities the model gave to the correct answers of nn examples are p1,…,pnp_1, \dots, p_n. If the examples are taken to be independent, the probability that the model gets all of them right (the likelihood) is the product ∏ipi\prod_i p_i. Figure 2 measures this product and the log sum over nn examples in which the three probabilities 0.9, 0.5, 0.01 repeat in turn.

Over n examples in which the three probabilities 0.9, 0.5, 0.01 repeat in turn, the sum of logs (solid) and the log taken after multiplying in float64 (dashed). The two go down together until the product drops below the smallest number that keeps full precision (the smallest normal number; shaded, below −708.4 in log), and then at n = 414 the product becomes 0 and the dashed line breaks off. The log sum is still −1080.7 at n = 600.
Figure 2 Over nn examples in which the three probabilities 0.9, 0.5, 0.01 repeat in turn, the sum of logs (solid) and the log taken after multiplying in float64 (dashed). The two go down together until the product drops below the smallest number that keeps full precision (the smallest normal number; shaded, below −708.4 in log), and then at n=414n = 414 the product becomes 0 and the dashed line breaks off. The log sum is still −1080.7 at n=600n = 600. figures/log_underflow.py

Multiplied in float64, from n=395n = 395 on the product is below the smallest normal number, about 2.2×10−3082.2 \times 10^{-308}, and loses precision, and at n=414n = 414 it becomes 0.11 A normal number is one that float64 stores with full precision (about 16 decimal digits), and the smallest of them is about 2.2×10−3082.2 \times 10^{-308}. Below that, it holds on while losing digits down to about 4.9×10−3244.9 \times 10^{-324} (−744.4 in log), and then becomes 0. You cannot take the log of a product that has become 0.

The numbers come from Eq. (4). The log sum falls by 5.4037 every three factors, so 395, which is 131 groups plus two more (131×(−5.4037)−0.105−0.693≈−708.7131 \times (-5.4037) - 0.105 - 0.693 \approx -708.7), is where it first drops below −708.4. 414 is 138 groups (138×(−5.4037)≈−745.7138 \times (-5.4037) \approx -745.7), where it first drops below −744.4 and the product becomes 0. The log sum is still fine at n=600n = 600 (200 groups), at 200×(−5.4037)≈−1080.7200 \times (-5.4037) \approx -1080.7.22 PyTorch's default data type, float32, has a much narrower range: the same product leaves the normal range from n=51n = 51 and becomes 0 at n=60n = 60.

The loss used in training is this log sum divided by the number of examples, with the sign flipped. For classification with a single correct answer, this is the cross-entropy loss.

L=−1n∑i=1nlog⁡pi(5)L = -\frac{1}{n} \sum_{i=1}^{n} \log p_i \tag{5}

For the three probabilities, L=5.404/3≈1.801L = 5.404 / 3 \approx 1.801. It is the mean of the −log⁡pi-\log p_i of each example, as in the right panel of Figure 1.

Key idea

AI takes the log of probabilities to turn products into sums, so that even very small probabilities can be computed without vanishing to 0.

Why is the -log p loss so large when the probability of the correct answer is small?

Because −log⁡p-\log p grows without bound as the correct-answer probability pp approaches 0. Giving the correct answer 0.9 costs 0.105 and 0.5 costs 0.693, but 0.01 costs 4.605, about 44 times the cost at 0.9. The slope −1/p-1/p, the derivative with respect to pp, is also −100 at 0.01, much steeper than the roughly −1.11 at 0.9 (Eq. (6)). But the gradient the network actually receives, the derivative with respect to the logit (score) of the correct class, is p−1p - 1, so it stays between −1 and 0 (−0.99 and −0.1; the derivation will be on the Cross-Entropy page, planned).

Why do we maximize the log-likelihood but minimize the loss?

Because the loss is the log-likelihood multiplied by −1/n-1/n (Eq. (5)), the point that makes one of them as large as possible is the point that makes the other as small as possible. For the three probabilities the log-likelihood is −5.404 and the loss is 1.801; they differ only in sign and size. Optimizers are built to 'reduce', so we flip the sign.

Taking the log leaves the location of the maximum unchanged

The logarithm always grows when its input grows (it is monotonically increasing). So for a positive quantity such as a likelihood, the point that makes it largest stays the same after you take its log. The slope makes this clear.

ddxlog⁡x=1x,ddxex=ex(6)\frac{d}{dx} \log x = \frac{1}{x}, \qquad \frac{d}{dx} e^x = e^x \tag{6}

For x>0x > 0, 1/x>01/x > 0, so the slope of the log is positive. But the slope 1/x1/x is greater than 1 for 0<x<10 < x < 1 and less than 1 for x>1x > 1. So the log stretches the differences between values below 1 and shrinks the differences between larger values.

Say you toss a coin 10 times independently and get 7 heads. If the probability of heads is θ\theta, the likelihood of this result, with the order of the tosses fixed, is f(θ)=θ7(1−θ)3f(\theta) = \theta^7 (1-\theta)^3, and by Eqs. (2) and (3), log⁡f(θ)=7log⁡θ+3log⁡(1−θ)\log f(\theta) = 7\log\theta + 3\log(1-\theta). Both are largest at θ=0.7\theta = 0.7; the maxima differ, 0.00222 and −6.109, but the location is the same.33 The probability of '7 heads' without regard to order is this times (107)=120\binom{10}{7} = 120, which is 0.267 at θ=0.7\theta = 0.7. A constant multiple does not move the location of the maximum.

A sum is easier to differentiate than a product, so maximum likelihood estimation (planned) differentiates the log. Solving 7/θ−3/(1−θ)=07/\theta - 3/(1-\theta) = 0 with Eq. (6) gives θ=0.7\theta = 0.7.44 By the chain rule, the derivative of log⁡(1−θ)\log(1-\theta) is −1/(1−θ)-1/(1-\theta).

The natural log ln and logs in base 2 or 10 differ only in units

A logarithm in any base is the natural logarithm divided by a constant (the change-of-base formula, Eq. (7)). So as long as the base is greater than 1, changing the base keeps the shape of the curve and only stretches or shrinks the vertical scale. For the common logarithm, with base 10, Eq. (3) and Eq. (7) below give log⁡100=log⁡102=2log⁡10\log 100 = \log 10^2 = 2\log 10, so log⁡10100=2\log_{10} 100 = 2; in natural logarithms, log⁡100≈4.605\log 100 \approx 4.605 and log⁡10≈2.303\log 10 \approx 2.303.

log⁡bx=log⁡xlog⁡b(7)\log_b x = \frac{\log x}{\log b} \tag{7}

What's the difference between ln and log?

ln⁡x\ln x is always the natural logarithm, with base ee, while log⁡x\log x with no base changes its base from place to place. In many school textbooks log⁡x\log x is the common logarithm, with base 10, and in AI papers and in NumPy and PyTorch log is the natural logarithm. So log⁡100\log 100 is 2 in such a textbook and about 4.605 in np.log(100). The two values differ by a factor of log⁡10≈2.303\log 10 \approx 2.303, as Eq. (7) says.

When you measure information, the base becomes the unit. Measured with the natural logarithm it is in nats, and with log⁡2\log_2 in bits. −log⁡p-\log p for p=0.5p = 0.5 is 0.693 nat, and dividing by log⁡2≈0.693\log 2 \approx 0.693 with Eq. (7) gives exactly 1 bit. 1 nat is 1/log⁡2≈1.4431/\log 2 \approx 1.443 bits, and the Entropy & KL page (planned) uses these units.

log⁡10\log_{10} is used to put values that differ by factors of ten on one axis. The learning rates 0.0001, 0.001, 0.01 are −4, −3, −2 in log⁡10\log_{10}, so they are evenly spaced. Learning rates are searched at this spacing, and loss curves are drawn on such a log scale too (the loss figure on the Momentum page is an example).

Before taking the log of a sum of exponentials, subtract the maximum

Compute log⁡∑jezj\log \sum_j e^{z_j} (log-sum-exp) by first subtracting the largest logit m=max⁡jzjm = \max_j z_j. The value is the same, and nothing overflows or vanishes even when the logits are very large or very small.

This sum comes from the softmax. A classification model produces a score (logit) zjz_j for each class and turns it into the probability ezj/∑kezke^{z_j} / \sum_k e^{z_k}. Taking the log of that gives zj−log⁡∑kezkz_j - \log \sum_k e^{z_k} by Eq. (3).

log⁡∑j=1Kezj=m+log⁡∑j=1Kezj−m(8)\log \sum_{j=1}^{K} e^{z_j} = m + \log \sum_{j=1}^{K} e^{z_j - m} \tag{8}
Why Eq. (8) gives the same value

Since ezj=emezj−me^{z_j} = e^{m} e^{z_j - m}, you can factor eme^m out of the sum. Then by Eq. (2), log⁡(em∑jezj−m)=m+log⁡∑jezj−m\log\big(e^m \sum_j e^{z_j - m}\big) = m + \log \sum_j e^{z_j - m}. Because zj−m≤0z_j - m \le 0, each term is at most 1, and the term with zj=mz_j = m is e0=1e^0 = 1. So if K≥2K \ge 2 and the logits are finite, the sum is greater than 1 and does not exceed KK.

The slope of exe^x is the function itself (Eq. (6)), so the larger xx gets, the faster it grows. The largest number float64 can represent is about 1.8×103081.8 \times 10^{308}, and its log is 709.78, so when xx is above that, exe^x is inf.55 In PyTorch's default float32 it is about 88.72, and in half-precision float16 about 11.09. So if even one logit is above this value, computing directly overflows.66 Even if all logits are below 709.78, it overflows when the sum exceeds about 1.8×103081.8 \times 10^{308}. Two logits of 709.5 give inf.

With logits z=(1000,999,998)\mathbf{z} = (1000, 999, 998), e1000e^{1000} is inf, so computing directly gives inf. With Eq. (8) it is 1000+log⁡(1+e−1+e−2)≈1000.4081000 + \log(1 + e^{-1} + e^{-2}) \approx 1000.408. Subtracting this value from each logit gives the log probabilities −0.408, −1.408, −2.408, which as probabilities are 0.665, 0.245, 0.090. Conversely, if the logits are all below about −745, as in (−800,−801)(-800, -801), every ezje^{z_j} becomes 0 and computing directly gives −∞-\infty, but Eq. (8) gives about −799.687.

This calculation is the log-softmax on the Softmax page.

Common misconceptions about the logarithm

  • log⁡(s+t)=log⁡s+log⁡t\log(s+t) = \log s + \log t — Eq. (2) is about products. log⁡(0.9+0.5)=log⁡1.4≈0.336\log(0.9 + 0.5) = \log 1.4 \approx 0.336, but log⁡0.9+log⁡0.5≈−0.7985\log 0.9 + \log 0.5 \approx -0.7985. When you hold only the log probabilities log⁡s\log s and log⁡t\log t, compute the log of the sum by viewing s=elog⁡ss = e^{\log s} and using np.logaddexp(log_s, log_t), that is, Eq. (8).
  • log⁡0\log 0 is a very large negative number — It is outside the x>0x > 0 of Eq. (1), so it is undefined. NumPy's np.log(0) gives -inf with a warning and np.log(-1) gives nan, and Python's math.log(0) raises ValueError. That is why the loss becomes inf when a probability is rounded to 0.
  • log in code has base 10 — np.log, math.log, and torch.log are the natural logarithm (ln⁡\ln). np.log(100) is 4.605, not 2. Base 10 is called separately as np.log10, and base 2 as np.log2.
  • Taking the log also moves the location of the maximum — The value changes, but the location stays. In the coin example, the likelihood f(θ)f(\theta) and the log-likelihood are both largest at θ=0.7\theta = 0.7.
  • You feed nn.CrossEntropyLoss probabilities that have been through softmax — It applies log-softmax again inside, so the softmax is applied twice. Feed the probabilities 0.665, 0.245, 0.090 instead of the logits (1000,999,998)(1000, 999, 998), and the loss for the correct class 0 becomes −log⁡(e0.665/(e0.665+e0.245+e0.090))≈0.797-\log\big(e^{0.665} / (e^{0.665} + e^{0.245} + e^{0.090})\big) \approx 0.797 instead of 0.408.

Where the logarithm shows up in AI

When a deep-learning classification model is trained, the log sits in the stage that computes the loss. The model outputs logits z\mathbf{z}, and log-softmax makes the log probabilities zj−log⁡∑kezkz_j - \log\sum_k e^{z_k} with Eq. (8). Flip the sign of the correct class's value and average over the examples, and you have the loss of Eq. (5). Then backpropagation gets the gradient of this loss, and gradient descent takes one step.

This pageNumPy / PyTorch
log⁡x\log x (natural logarithm)np.log(x), torch.log(x), math.log(x)
log⁡2x\log_2 x, log⁡10x\log_{10} xnp.log2(x), np.log10(x)
log⁡(1+x)\log(1 + x) when xx is very smallnp.log1p(x), torch.log1p(x). 1 + 1e-16 rounds to 1.0 in float64, so np.log(1 + 1e-16) is 0.0, but np.log1p(1e-16) is 1e-16. log⁡(1−p)\log(1 - p) is also written np.log1p(-p)
log⁡(ez1+ez2)\log(e^{z_1} + e^{z_2})np.logaddexp(z1, z2), torch.logaddexp(z1, z2)
Eq. (8), log-sum-exptorch.logsumexp(z, dim=-1)
Log probability zj−log⁡∑kezkz_j - \log\sum_k e^{z_k}F.log_softmax(z, dim=-1) (torch.nn.functional; the same as torch.log_softmax)
The loss of Eq. (5)nn.CrossEntropyLoss()(logits, target). It takes logits, computes log-softmax and the negative log-likelihood inside, and averages with the default reduction='mean'
exe^xnp.exp(x), torch.exp(x)

Why does nn.CrossEntropyLoss take logits instead of probabilities?

Because if you convert to probabilities first, a very small probability is rounded to 0 and its log becomes -inf. That happens when the correct logit is smaller than the largest logit by more than about 745 in float64, or about 104 in float32. With logits (0,−800)(0, -800), the probability of the second class is 0 in float64, but the log probability computed directly with Eq. (8) is −800. So PyTorch takes logits and computes with log-softmax inside.

Check it in code

Check that the log of the product of the three probabilities and the log sum are the same value, and that at 600 factors only the product vanishes. The last two lines are the right end of Figure 2.

Math
log⁡∏i=1npi=∑i=1nlog⁡pi(4)\log \prod_{i=1}^{n} \sym{probs}{p_i} = \sum_{i=1}^{n} \sym{log_probs}{\log p_i} \tag{4}
Code figures/log_prob.py
import numpy as np

probs = np.array([0.9, 0.5, 0.01])   # 정답 확률 p_i
log_probs = np.log(probs)            # 자연로그
print("-log p      :", np.round(-log_probs, 3))
print("곱          :", round(np.prod(probs), 6))
# 식 (4): 곱의 로그 = 로그의 합
print("곱의 로그   :", round(np.log(np.prod(probs)), 3))
print("로그의 합   :", round(np.sum(log_probs), 3))
# 식 (5): 손실 = 평균 -log p
print("평균 -log p :", round(-np.mean(log_probs), 3))

many = np.resize(probs, 600)         # 600개로 되풀이
print("600개의 곱  :", np.prod(many))
print("600개 로그합:", round(np.sum(np.log(many)), 1))
$ python log_prob.py
-log p      : [0.105 0.693 4.605]
곱          : 0.0045
곱의 로그   : -5.404
로그의 합   : -5.404
평균 -log p : 1.801
600개의 곱  : 0.0
600개 로그합: -1080.7

Try it yourself

  1. If the probability of the correct answer is 0.25, how many nats and how many bits is −log⁡p-\log p?
Answer

−log⁡0.25=log⁡4≈1.386-\log 0.25 = \log 4 \approx 1.386 nats. Dividing by log⁡2\log 2 with Eq. (7) gives 2 bits. 0.25 is 1/21/2 multiplied by itself, so in bits it is exactly 2.

  1. Find the log-sum-exp of the logits (3,1)(3, 1) with Eq. (8).
Answer

m=3m = 3, so it is 3+log⁡(1+e−2)≈3.1273 + \log(1 + e^{-2}) \approx 3.127. That equals log⁡(e3+e1)≈log⁡22.80\log(e^3 + e^1) \approx \log 22.80 computed directly.

Further reading

  • Deep Learning, Chapter 3 Probability and Information Theory — Goodfellow, Bengio, Courville, 2016. Section 3.13 defines the self-information of an event as −log⁡P(x)-\log P(x), and notes that it is in nats when measured with the natural logarithm and in bits with base 2. The section 'The natural log ln and logs in base 2 or 10 differ only in units' relies on it.
  • Deep Learning, Chapter 4 Numerical Computation — Goodfellow, Bengio, Courville, 2016. Section 4.1 covers overflow and underflow, and stabilizes the softmax by subtracting the largest value before computing it. The section 'Before taking the log of a sum of exponentials, subtract the maximum' relies on it.
  • Deep Learning, Chapter 5 Machine Learning Basics — Goodfellow, Bengio, Courville, 2016. Section 5.5, on maximum likelihood estimation, takes the log of the product-form likelihood to turn it into a sum, and notes that the location of the maximum stays the same. The section 'Taking the log leaves the location of the maximum unchanged' relies on it.
  • numpy.log — NumPy official documentation. It is the natural logarithm, and gives nan for inputs that cannot be represented as a real number. The second and third items of the section 'Common misconceptions about the logarithm' rely on it.
  • torch.nn.CrossEntropyLoss — PyTorch official documentation. The input is unnormalized logits, and the default reduction is 'mean'. The mapping table and the question in the section 'Where the logarithm shows up in AI' rely on it.

Video notes

Video notes
  • 00:00 Opening question — “Why does AI take the log of probabilities?”
  • 00:30 Intuition — computing 100 × 1,000 by counting 0s
  • 01:20 Write Eq. (1) on the board — undoing the exponent, the mirror in the left panel of Figure 1
  • 02:30 Eqs. (2) and (4) — 0.9×0.5×0.01=0.00450.9 \times 0.5 \times 0.01 = 0.0045, log sum −5.404-5.404
  • 03:40 Figure 2 — the product that vanishes at n=414n = 414
  • 04:40 Eq. (5) and the right panel of Figure 1 — −log⁡p-\log p = 0.105, 0.693, 4.605
  • 05:40 Eq. (8) — subtracting the maximum so a logit of 1000 does not overflow
  • 06:40 PyTorch mapping table · wrap-up

This is the filming plan. After upload, the timestamps will be replaced with the real ones.

Where this sits on the map

Real ExponentsCalculusLaws of ExponentsCalculusExponential FunctionCalculusLogarithmYou are hereCross-EntropyPlannedEntropy & KLPlannedMaximum LikelihoodPlannedSoftmaxAI
Prerequisites
This page
Next
DonePlannedThe path to this pageSee the full map →