Intuition — turning multiplication into adding counts of 0s
For 100 × 1,000 you only need to count the 0s. Multiply a number with two 0s by one with three and you get 100,000, with five 0s. Multiplication has turned into adding up counts of 0s.
Here the number of 0s is the logarithm, and 10 is the base of that logarithm. The logarithm extends this 'number of 0s' to every positive number. For 2 the count of 0s is about 0.301 (), and 0.01, which is 1 divided by 100, has had two 0s erased, so its count is −2. In AI the base is instead of 10, and this is called the natural logarithm.
The logarithm undoes the exponential function
asks to what power the base must be raised to give ( is called the argument). So the logarithm is the function that undoes the exponential function (its inverse function).
At first, the top five rows of the table below are enough. The remaining rows are used in later sections.
| Symbol | Name | Meaning |
|---|---|---|
| logarithm with base | The power to which must be raised to give . Defined only for . Viewed as a function of , it is the logarithmic function | |
| base | Usually , 2, or 10 | |
| natural logarithm | The logarithm with base . Also written . This is the log of this site and of NumPy and PyTorch; for base 2 or 10 we write or | |
| Euler's number | ||
| , | exponential function | raised to the power . The inverse function of |
| , | probability | The probability the model gave to the correct answer, and that probability for the th example. |
| number of examples | How many probabilities are multiplied or added | |
| loss | The mean of of the correct-answer probabilities. Smaller is better | |
| , , | logit, number of classes | The per-class scores that go into the softmax |
| , , | positive number, real number | Any positive numbers , and any real number put into Eqs. (2) and (3) |
| maximum logit | . Subtracted first in Eq. (8) | |
| , | parameter, likelihood | In the coin example, the probability of heads, and the probability that the observed result, with the order of the tosses fixed, comes out at that |
In many school textbooks, with no base is the common logarithm (base 10), and the natural logarithm is written . On this page, is the natural logarithm.
With , and undo each other: and . So the two curves in the left panel of Figure 1 face each other like mirror images across the diagonal . The points and on become and on .
figures/log_curves.pyThree things are visible right away in the figure. The logarithm is defined only for , , and as approaches 0 it falls to . So a probability between 0 and 1 has a negative log, and , with the sign flipped, is 0 or more.
The logarithm turns products into sums
The logarithm turns products into sums, division into subtraction, and powers into multiplication (the 'laws of logarithms' in textbooks). Each of these is the exponent law (multiplying by times and then more times is multiplying by it times) read backward. In the equations below, and are positive numbers and is a real number.
Why Eq. (2) holds
Set and ; then by Eq. (1), and . So , and by Eq. (1) again, . Division works the same way with , and powers with .
Let's check with three probabilities, 0.9, 0.5, 0.01. The product is , and . Adding the individual logs , , , using the values before rounding, also gives , that is, about . With terms, applying Eq. (2) again and again turns the product into the sum (the symbols from the Sigma Notation page).
Why add logs instead of multiplying probabilities
Because when you multiply many probabilities, the computer rounds the result to 0 (underflow). Multiply 0.9, 0.5, 0.01 over and over in the computer's 64-bit floating-point numbers (float64), and the product becomes 0 at the 414th factor, while the sum of logs remains, at −745.7. The absolute value of the sum of logs only grows as (the mean ), so it does not vanish.
Say the probabilities the model gave to the correct answers of examples are . If the examples are taken to be independent, the probability that the model gets all of them right (the likelihood) is the product . Figure 2 measures this product and the log sum over examples in which the three probabilities 0.9, 0.5, 0.01 repeat in turn.
figures/log_underflow.pyMultiplied in float64, from on the product is below the smallest normal number, about , and loses precision, and at it becomes 0.11 A normal number is one that float64 stores with full precision (about 16 decimal digits), and the smallest of them is about . Below that, it holds on while losing digits down to about (−744.4 in log), and then becomes 0. You cannot take the log of a product that has become 0.
The numbers come from Eq. (4). The log sum falls by 5.4037 every three factors, so 395, which is 131 groups plus two more (), is where it first drops below −708.4. 414 is 138 groups (), where it first drops below −744.4 and the product becomes 0. The log sum is still fine at (200 groups), at .22 PyTorch's default data type, float32, has a much narrower range: the same product leaves the normal range from and becomes 0 at .
The loss used in training is this log sum divided by the number of examples, with the sign flipped. For classification with a single correct answer, this is the cross-entropy loss.
For the three probabilities, . It is the mean of the of each example, as in the right panel of Figure 1.
AI takes the log of probabilities to turn products into sums, so that even very small probabilities can be computed without vanishing to 0.
Why is the -log p loss so large when the probability of the correct answer is small?
Because grows without bound as the correct-answer probability approaches 0. Giving the correct answer 0.9 costs 0.105 and 0.5 costs 0.693, but 0.01 costs 4.605, about 44 times the cost at 0.9. The slope , the derivative with respect to , is also −100 at 0.01, much steeper than the roughly −1.11 at 0.9 (Eq. (6)). But the gradient the network actually receives, the derivative with respect to the logit (score) of the correct class, is , so it stays between −1 and 0 (−0.99 and −0.1; the derivation will be on the Cross-Entropy page, planned).
Why do we maximize the log-likelihood but minimize the loss?
Because the loss is the log-likelihood multiplied by (Eq. (5)), the point that makes one of them as large as possible is the point that makes the other as small as possible. For the three probabilities the log-likelihood is −5.404 and the loss is 1.801; they differ only in sign and size. Optimizers are built to 'reduce', so we flip the sign.
Taking the log leaves the location of the maximum unchanged
The logarithm always grows when its input grows (it is monotonically increasing). So for a positive quantity such as a likelihood, the point that makes it largest stays the same after you take its log. The slope makes this clear.
For , , so the slope of the log is positive. But the slope is greater than 1 for and less than 1 for . So the log stretches the differences between values below 1 and shrinks the differences between larger values.
Say you toss a coin 10 times independently and get 7 heads. If the probability of heads is , the likelihood of this result, with the order of the tosses fixed, is , and by Eqs. (2) and (3), . Both are largest at ; the maxima differ, 0.00222 and −6.109, but the location is the same.33 The probability of '7 heads' without regard to order is this times , which is 0.267 at . A constant multiple does not move the location of the maximum.
A sum is easier to differentiate than a product, so maximum likelihood estimation (planned) differentiates the log. Solving with Eq. (6) gives .44 By the chain rule, the derivative of is .
The natural log ln and logs in base 2 or 10 differ only in units
A logarithm in any base is the natural logarithm divided by a constant (the change-of-base formula, Eq. (7)). So as long as the base is greater than 1, changing the base keeps the shape of the curve and only stretches or shrinks the vertical scale. For the common logarithm, with base 10, Eq. (3) and Eq. (7) below give , so ; in natural logarithms, and .
What's the difference between ln and log?
is always the natural logarithm, with base , while with no base changes its base from place to place. In many school textbooks is the common logarithm, with base 10, and in AI papers and in NumPy and PyTorch log is the natural logarithm. So is 2 in such a textbook and about 4.605 in np.log(100). The two values differ by a factor of , as Eq. (7) says.
When you measure information, the base becomes the unit. Measured with the natural logarithm it is in nats, and with in bits. for is 0.693 nat, and dividing by with Eq. (7) gives exactly 1 bit. 1 nat is bits, and the Entropy & KL page (planned) uses these units.
is used to put values that differ by factors of ten on one axis. The learning rates 0.0001, 0.001, 0.01 are −4, −3, −2 in , so they are evenly spaced. Learning rates are searched at this spacing, and loss curves are drawn on such a log scale too (the loss figure on the Momentum page is an example).
Before taking the log of a sum of exponentials, subtract the maximum
Compute (log-sum-exp) by first subtracting the largest logit . The value is the same, and nothing overflows or vanishes even when the logits are very large or very small.
This sum comes from the softmax. A classification model produces a score (logit) for each class and turns it into the probability . Taking the log of that gives by Eq. (3).
Why Eq. (8) gives the same value
Since , you can factor out of the sum. Then by Eq. (2), . Because , each term is at most 1, and the term with is . So if and the logits are finite, the sum is greater than 1 and does not exceed .
The slope of is the function itself (Eq. (6)), so the larger gets, the faster it grows. The largest number float64 can represent is about , and its log is 709.78, so when is above that, is inf.55 In PyTorch's default float32 it is about 88.72, and in half-precision float16 about 11.09. So if even one logit is above this value, computing directly overflows.66 Even if all logits are below 709.78, it overflows when the sum exceeds about . Two logits of 709.5 give inf.
With logits , is inf, so computing directly gives inf. With Eq. (8) it is . Subtracting this value from each logit gives the log probabilities −0.408, −1.408, −2.408, which as probabilities are 0.665, 0.245, 0.090. Conversely, if the logits are all below about −745, as in , every becomes 0 and computing directly gives , but Eq. (8) gives about −799.687.
This calculation is the log-softmax on the Softmax page.
Common misconceptions about the logarithm
- — Eq. (2) is about products. , but . When you hold only the log probabilities and , compute the log of the sum by viewing and using
np.logaddexp(log_s, log_t), that is, Eq. (8). - is a very large negative number — It is outside the of Eq. (1), so it is undefined. NumPy's
np.log(0)gives-infwith a warning andnp.log(-1)givesnan, and Python'smath.log(0)raises ValueError. That is why the loss becomes inf when a probability is rounded to 0. - log in code has base 10 —
np.log,math.log, andtorch.logare the natural logarithm ().np.log(100)is 4.605, not 2. Base 10 is called separately asnp.log10, and base 2 asnp.log2. - Taking the log also moves the location of the maximum — The value changes, but the location stays. In the coin example, the likelihood and the log-likelihood are both largest at .
- You feed nn.CrossEntropyLoss probabilities that have been through softmax — It applies log-softmax again inside, so the softmax is applied twice. Feed the probabilities 0.665, 0.245, 0.090 instead of the logits , and the loss for the correct class 0 becomes instead of 0.408.
Where the logarithm shows up in AI
When a deep-learning classification model is trained, the log sits in the stage that computes the loss. The model outputs logits , and log-softmax makes the log probabilities with Eq. (8). Flip the sign of the correct class's value and average over the examples, and you have the loss of Eq. (5). Then backpropagation gets the gradient of this loss, and gradient descent takes one step.
| This page | NumPy / PyTorch |
|---|---|
| (natural logarithm) | np.log(x), torch.log(x), math.log(x) |
| , | np.log2(x), np.log10(x) |
| when is very small | np.log1p(x), torch.log1p(x). 1 + 1e-16 rounds to 1.0 in float64, so np.log(1 + 1e-16) is 0.0, but np.log1p(1e-16) is 1e-16. is also written np.log1p(-p) |
np.logaddexp(z1, z2), torch.logaddexp(z1, z2) | |
| Eq. (8), log-sum-exp | torch.logsumexp(z, dim=-1) |
| Log probability | F.log_softmax(z, dim=-1) (torch.nn.functional; the same as torch.log_softmax) |
| The loss of Eq. (5) | nn.CrossEntropyLoss()(logits, target). It takes logits, computes log-softmax and the negative log-likelihood inside, and averages with the default reduction='mean' |
np.exp(x), torch.exp(x) |
Why does nn.CrossEntropyLoss take logits instead of probabilities?
Because if you convert to probabilities first, a very small probability is rounded to 0 and its log becomes -inf. That happens when the correct logit is smaller than the largest logit by more than about 745 in float64, or about 104 in float32. With logits , the probability of the second class is 0 in float64, but the log probability computed directly with Eq. (8) is −800. So PyTorch takes logits and computes with log-softmax inside.
Check it in code
Check that the log of the product of the three probabilities and the log sum are the same value, and that at 600 factors only the product vanishes. The last two lines are the right end of Figure 2.
import numpy as np
probs = np.array([0.9, 0.5, 0.01]) # 정답 확률 p_i
log_probs = np.log(probs) # 자연로그
print("-log p :", np.round(-log_probs, 3))
print("곱 :", round(np.prod(probs), 6))
# 식 (4): 곱의 로그 = 로그의 합
print("곱의 로그 :", round(np.log(np.prod(probs)), 3))
print("로그의 합 :", round(np.sum(log_probs), 3))
# 식 (5): 손실 = 평균 -log p
print("평균 -log p :", round(-np.mean(log_probs), 3))
many = np.resize(probs, 600) # 600개로 되풀이
print("600개의 곱 :", np.prod(many))
print("600개 로그합:", round(np.sum(np.log(many)), 1))$ python log_prob.py
-log p : [0.105 0.693 4.605]
곱 : 0.0045
곱의 로그 : -5.404
로그의 합 : -5.404
평균 -log p : 1.801
600개의 곱 : 0.0
600개 로그합: -1080.7Try it yourself
- If the probability of the correct answer is 0.25, how many nats and how many bits is ?
Answer
nats. Dividing by with Eq. (7) gives 2 bits. 0.25 is multiplied by itself, so in bits it is exactly 2.
- Find the log-sum-exp of the logits with Eq. (8).
Answer
, so it is . That equals computed directly.
Further reading
- Deep Learning, Chapter 3 Probability and Information Theory — Goodfellow, Bengio, Courville, 2016. Section 3.13 defines the self-information of an event as , and notes that it is in nats when measured with the natural logarithm and in bits with base 2. The section 'The natural log ln and logs in base 2 or 10 differ only in units' relies on it.
- Deep Learning, Chapter 4 Numerical Computation — Goodfellow, Bengio, Courville, 2016. Section 4.1 covers overflow and underflow, and stabilizes the softmax by subtracting the largest value before computing it. The section 'Before taking the log of a sum of exponentials, subtract the maximum' relies on it.
- Deep Learning, Chapter 5 Machine Learning Basics — Goodfellow, Bengio, Courville, 2016. Section 5.5, on maximum likelihood estimation, takes the log of the product-form likelihood to turn it into a sum, and notes that the location of the maximum stays the same. The section 'Taking the log leaves the location of the maximum unchanged' relies on it.
- numpy.log — NumPy official documentation. It is the natural logarithm, and gives nan for inputs that cannot be represented as a real number. The second and third items of the section 'Common misconceptions about the logarithm' rely on it.
- torch.nn.CrossEntropyLoss — PyTorch official documentation. The input is unnormalized logits, and the default
reductionis'mean'. The mapping table and the question in the section 'Where the logarithm shows up in AI' rely on it.
Video notes
- 00:00 Opening question — “Why does AI take the log of probabilities?”
- 00:30 Intuition — computing 100 × 1,000 by counting 0s
- 01:20 Write Eq. (1) on the board — undoing the exponent, the mirror in the left panel of Figure 1
- 02:30 Eqs. (2) and (4) — , log sum
- 03:40 Figure 2 — the product that vanishes at
- 04:40 Eq. (5) and the right panel of Figure 1 — = 0.105, 0.693, 4.605
- 05:40 Eq. (8) — subtracting the maximum so a logit of 1000 does not overflow
- 06:40 PyTorch mapping table · wrap-up
This is the filming plan. After upload, the timestamps will be replaced with the real ones.