Intuition — a skill rating looks only at the difference
A skill rating in a game (such as the Elo rating in chess) sets the chance of who wins by looking only at the difference between the two players' ratings. Give both players the same extra 100 points and that chance stays the same. The wider the difference, the more it tilts to one side, but it never reaches 0 or 1.
Here a player's rating is the logit, and the chance of winning is the probability that softmax outputs. The 100 points given to everyone is the 'constant added to the logits' that comes up later. Softmax extends this computation from two players to candidates.
Say a classification model that decides whether a photo shows a dog, a cat, or a bird gives the three candidates the scores 2.0, 1.0, 0.1. Softmax turns these scores into the probabilities of about 66%, 24%, 10%. A language model does the same computation over its vocabulary each time it picks the next word.
The softmax function turns scores (logits) into probabilities that sum to 1
The softmax function turns the scores (logits) a model outputs into probabilities that are all greater than 0 and sum to 1. It takes the exponential of each logit and divides by their sum.
A logit is the score a classification model outputs for each class; it may be negative, and the logits need not sum to 1. The logits lined up in a row form the vector , and each number in it is a component. The result is a probability distribution (Korean for now): a probability of 0 or more for each outcome, with a total of 1.
| Symbol | Name | Meaning |
|---|---|---|
| , | logit | The vector of the scores. The protagonist of this page |
| , | probability | The output of softmax. All greater than 0, summing to 1 |
| , , | number of classes, class index | How many answers there are to choose from, and the index from 1 to . in this page's example |
| sum | Add in order from to (the Sigma Notation page) | |
| exponential function | to the power . Always greater than 0, and multiplied by when grows by 1 | |
| natural logarithm | The logarithm with base . It undoes the exponential function, . The textbook , not the base-10 logarithm (the Logarithm page) | |
| temperature | A positive number that divides the logits (Eq. (4)). The default is . Depending on the text it is written |
Say the logits of the three classes (dog, cat, bird) are . The exponentials are , , , and their sum is about 11.21. Dividing each by the sum gives , and the three values sum to 1.000. The middle bars of Figure 1 are these values.
figures/temperature_bars.pyThe ratio of probabilities is set only by the difference of logits
The ratio of two probabilities is the exponential of the difference of their logits. In the example , so .
That is because the denominator of Eq. (1) cancels between the top and the bottom. Read the other way, the difference of logits is the log of the probability ratio, . Since , .
Three properties follow from Eq. (1) and Eq. (2). The exponential is greater than 0, so every . If , then , so the order is kept. Adding the same number to every logit leaves the differences, and so the ratios, unchanged, and since the sum is 1, the probabilities are unchanged too.
Here is the vector with added to each component. Conversely, raising just one logit by 1 does not multiply its probability by . Raising from 2 to 3 takes from 0.659 to 0.840, about 1.275 times, because the denominator grows too.
Softmax looks only at the differences of the logits. The ratio of two probabilities, , is the exponential of the logit difference, .
Why does softmax use the exponential (exp)?
Because the exponential turns any real number into a number greater than 0, and turns differences into ratios (). That is why Eq. (2) and Eq. (3) hold. If you divide by the sum without the exponential, gives , but , with 1 subtracted from each, sums to 0.1 and gives .
Why lowering the softmax temperature lets the largest logit take all the probability
Because dividing the logits by the temperature multiplies their differences by , and the ratio in Eq. (2) becomes . Shrinking toward 0 sends the probability of the largest logit (when there is only one) to 1, and raising sends every probability to . In terms of the skill rating in the intuition, the temperature is the unit, in points, in which the rating difference is measured.
The left panel of Figure 1 () puts into Eq. (1) and tilts to . The right panel () puts in and spreads out to .
Figure 2 shows the bars while rises from 0.1 to 10. The low end is close to the one-hot vector , which is 1 only at the largest logit, and the high end is close to an even 1/3.
figures/temperature_sweep.pyIn AI the temperature shows up in two places. The temperature in a language model's generation settings is the of Eq. (4) (the Sampling from a Distribution page, Korean for now). Attention in a transformer divides its dot products by ( is the dimension of the key vectors), which, seen through Eq. (4), also sits in the place of the temperature (Vaswani et al. 2017).
Does raising the softmax temperature change the ranking?
No. Raising or lowering the softmax temperature leaves the ranking of the probabilities as it is. Dividing by a positive only scales all the logit differences by the same factor, so anywhere for between 0.1 and 10 (Figure 2). The probability of the lowest class grows ( goes from 0.099 to 0.194 at ), but it does not overtake the others.
What happens if the softmax temperature is set to 0?
Setting the softmax temperature to exactly 0 divides the logits by 0, so the value is undefined. Instead, shrinking close to 0 approaches the one-hot in which the single largest logit takes 1; in the example, gives . Hugging Face Transformers raises an error for temperature=0.0 and tells you to use greedy decoding (do_sample=False), which picks only the largest logit.
With two classes, softmax is the sigmoid
With , the softmax is the sigmoid of the logit difference . It follows directly from dividing the numerator and the denominator of Eq. (1) by .
| Symbol | Name | Meaning |
|---|---|---|
| sigmoid | . An S-shaped curve between 0 and 1. Also called the logistic function |
Figure 3 draws this curve for each temperature (with a temperature it is ). The smaller , the steeper.
At the example's difference of 1.0 with , . That differs from the 0.659 with three classes, but in both cases is exactly (Eq. (2)).
figures/sigmoid_family.pyWhat's the difference between softmax and the sigmoid?
With two classes they are the same function. The softmax is the sigmoid of the logit difference, (Eq. (5)), which is 0.731 for a difference of 1.0. That is why binary classification outputs just one logit and uses the sigmoid. When one input can have several correct answers (multi-label), a separate sigmoid is used for each class, and the probabilities need not sum to 1.
Softmax is a smoothed argmax, not a smoothed max
What softmax smooths is not the largest value (max) but the position of that value, the argmax. The argmax of the example is position 1 (0 in code, which counts from 0), and written as a one-hot that puts 1 only at that position it is . The softmax is a smoothed-out version of this one-hot. 'Soft' means smooth enough to differentiate, and the name is said to come from Bridle (1990).11 Goodfellow et al. (Section 6.2.2.3) write that it would be better to call this function softargmax.
The function that smoothly approximates max itself is log-sum-exp (LSE). The log-softmax is exactly the logits minus LSE.
In the example, . As Eq. (7) says, it is larger than the max 2.0 and smaller than . The log-softmax is , each logit minus 2.417. How to compute LSE without overflow is covered on the Logarithm page.
Taking the partial derivative of LSE with respect to gives the softmax. The partial derivative is the derivative when only moves and the other logits stay fixed (the Gradient page, Korean for now).
Differentiating max (when there is a single largest logit) gives the one-hot that is 1 only at that position, that is, the argmax written as a one-hot. In the same way, differentiating LSE, the smoothed max, gives softmax, the smoothed argmax. How all the probabilities move when one logit moves, that is, the derivative of softmax, is in the folded Eq. (9) below.
The derivative of softmax (the Jacobian) — how the probabilities move when one logit moves
is 1 if and 0 otherwise (the Kronecker delta).
Differentiating Eq. (6) with respect to and using Eq. (8) gives . The derivative of the log is , so multiplying both sides by gives the Jacobian (the table of partial derivatives).
In the example, , , and . So raising by 0.01 raises by about 0.0022, and and together go down by the same amount. That each column (fixed ) sums to 0 is the derivative of the condition that the sum is 1, and that each row (fixed ) sums to 0 is the derivative of Eq. (3).
Common misconceptions about softmax
- If softmax outputs 0.9, the prediction is right with 90% probability — Multiplying the logits by 2 with Eq. (4) () keeps the prediction at class 1, but rises from 0.659 to 0.864. The size of the probabilities depends on the scale of the logits, so matching them to the actual accuracy takes a separate calibration such as temperature scaling.
- You should apply softmax before passing values to
CrossEntropyLoss—nn.CrossEntropyLossapplies log-softmax inside, so feeding it probabilities applies softmax twice. The probability of the correct answer shrinks to , and the example's loss changes from 0.417 to 0.802. With three classes it can never go below 0.551 however long you train (Try it yourself 1). Why it takes logits is answered by a question on the Logarithm page.
Where softmax shows up in AI
The last layer (output layer) of a classification network outputs one logit per class. That layer is the linear layer of the Matrix Operations page, and that page's output is the here. Softmax turns the logits into probabilities, and training reduces the cross-entropy loss (Cross-Entropy page — planned).
On this page is the one-hot vector that is 1 only at the correct answer, so , and putting in Eq. (6) gives . Differentiating softmax together with the cross-entropy then takes a single line, thanks to Eq. (8).
If the correct answer is class 1 (dog), . The loss is , and the gradient is . Backpropagation (Korean for now) carries this value back to the weights, and gradient descent moves the weights in the opposite direction. Seen from the logits, that raises the correct logit and lowers the others.
Why does the output layer of a multi-class classifier use the softmax function?
Because the softmax function turns the per-class scores into positive probabilities that sum to 1. In multi-class classification, which picks one out of three or more, using it as the activation function of the output layer lets you read the outputs directly as per-class probabilities. The example's scores become 0.659 for dog, 0.242 for cat, and 0.099 for bird.
Attention in a transformer also applies softmax over each row of the table of query–key dot products, , to make the weights that mix the value vectors (the Matrix Operations page; the bold is the key matrix, different from the number of classes ). A language model outputs one logit for every token in its vocabulary each time it picks the next token, and softmax turns them into per-token probabilities.
| This page | Code |
|---|---|
| Eq. (1) | torch.softmax(z, dim=-1) (= torch.nn.functional.softmax), scipy.special.softmax(z, axis=-1) (the whole array if axis is omitted). dim and axis set the axis along which the sum is made 1 |
| Eq. (6) | torch.nn.functional.log_softmax(z, dim=-1) — it does not compute log(softmax(z)) separately, but computes Eq. (6) with LSE after subtracting the maximum |
torch.logsumexp(z, dim=-1), scipy.special.logsumexp(z, axis=-1) (the whole array if axis is omitted) | |
The first argument of nn.CrossEntropyLoss()(z, target) (in functional form, F.cross_entropy(z, target)). The logits before softmax; target is the index of the correct class (from 0) | |
| Eq. (10) | The value that ends up in z.grad after loss.backward() on the loss above. The z the model outputs is an intermediate tensor, so you must call z.retain_grad() first to keep it. With the default reduction="mean", it is divided by the batch size |
torch.softmax(z / tau, dim=-1) — the temperature of language-model generation | |
torch.softmax(Q @ K.transpose(-2, -1) / d_k ** 0.5, dim=-1) in attention |
Why subtract the maximum when softmax gives nan
When the logits are large, the exponentials overflow to inf (infinity), and inf ÷ inf = nan (not a number). Thanks to Eq. (3), subtracting the largest logit from all of them gives the same result, and then the exponentials do not overflow (where the overflow starts is on the Logarithm page).
Compute , the example plus 998, directly with Eq. (1), and the numerator and the denominator are both inf, so the result is nan. Eq. (11) subtracts and uses the exponentials of , so the result is , the same as for .
After subtracting the maximum, every exponential is at most , and the denominator contains one , so there is no division by 0 either. Conversely, if you compute directly, all the exponentials drop to 0 (underflow), and 0 ÷ 0 = nan.22 Library functions such as torch.softmax and scipy.special.softmax do this inside.
When using softmax in PyTorch, watch the axis and the log
The common mistakes in practice are the axis along which the sum is made 1, and the order of computing the probabilities first and then taking the log.
What does dim=-1 mean in torch.softmax?
dim=-1 in torch.softmax means that the axis along which the sum is made 1 is the last axis (the classes). Give dim=-1 to a 2×3 array that stacks the example logits twice, and each row comes out as . Give dim=0, and the sum is made 1 across the two samples for each class, so every cell becomes the meaningless value 0.5.
Why use log_softmax (log-softmax) separately?
Because if you compute the probabilities first and then take the log, a very small probability drops to 0 and becomes . With the logits the softmax becomes 0, but log_softmax gives as it is (the computation is on the Logarithm page). The PyTorch documentation also says to put log_softmax, not softmax, in front of NLLLoss.
Check it in code
Change tau in the code to make the three bar charts of Figure 1 yourself. The '빼지 않음' (without subtracting) line of the output is nan.
import numpy as np
def softmax(logits, tau=1.0):
s = logits / tau # 식 (4) 온도
ex = np.exp(s - s.max()) # 식 (11) 최댓값 빼기
return ex / ex.sum() # 식 (1)
logits = np.array([2.0, 1.0, 0.1]) # z
probs = softmax(logits) # p
print("p ", probs.round(3), probs.sum())
print("p1 / p2 ", round(probs[0] / probs[1], 3)) # 식 (2)
print("z + 998 ", softmax(logits + 998).round(3)) # 식 (3)
with np.errstate(over="ignore", invalid="ignore"):
raw = np.exp(logits + 998) # e^1000 = inf
print("빼지 않음", raw / raw.sum())
for tau in (0.5, 1.0, 2.0):
print("tau", tau, " ", softmax(logits, tau).round(3))$ python softmax.py
p [0.659 0.242 0.099] 1.0
p1 / p2 2.718
z + 998 [0.659 0.242 0.099]
빼지 않음 [nan nan nan]
tau 0.5 [0.864 0.117 0.019]
tau 1.0 [0.659 0.242 0.099]
tau 2.0 [0.502 0.304 0.194]Try it yourself
- Compute the softmax of the logits by hand. What about the logits ?
- With two classes, what must be for ? And if the temperature is ?
Answers
- The denominator is , so it is . Subtracting 1 from every entry of gives , so by Eq. (3) it is the same value. The 0.551 in the common misconceptions is the loss when the one-hot that is 1 only at the correct answer is fed to
CrossEntropyLossas probabilities, . - From Eq. (2), , so . With , Eq. (4) gives , so it is twice that, about 4.394.
Further reading
- Deep Learning, Chapter 6 Deep Feedforward Networks — Goodfellow, Bengio, Courville, 2016. Section 6.2.2.3 covers softmax in the output layer for classification. It reads logits as unnormalized log probabilities, writes the loss with log-softmax, and shows that adding the same number to every input leaves the output unchanged. Section 4.1 of Chapter 4 Numerical Computation in the same book gives stabilizing softmax by subtracting the maximum as an example. The sections 'Why subtract the maximum when softmax gives nan' and 'Softmax is a smoothed argmax, not a smoothed max' rely on it.
- Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition — J. S. Bridle, 1990. Neurocomputing (NATO ASI Series F68), Springer. Known as the paper in which the name 'softmax' appeared. The story of the name in the section 'Softmax is a smoothed argmax, not a smoothed max' relies on it.
- Attention Is All You Need — Vaswani et al., 2017. NeurIPS. The scaled dot-product attention of Section 3.2.1 applies softmax to dot products divided by , and offers a guess at why it divides. The temperature section and the attention sentence of the section 'Where softmax shows up in AI' rely on it.
- On Calibration of Modern Neural Networks — Guo, Pleiss, Sun, Weinberger, 2017. ICML. It measures the tendency of the softmax probabilities of modern neural networks to come out higher than the actual accuracy, and proposes temperature scaling, which calibrates them by dividing the logits by a single temperature. The first item of the section 'Common misconceptions about softmax' relies on it.
- torch.nn.CrossEntropyLoss — PyTorch official documentation. It states that the input should be unnormalized logits (checked against version 2.15), and that this loss is equivalent to applying
LogSoftmaxfollowed byNLLLoss. The softmax and log_softmax documentation explain thedimargument and why log_softmax is used separately. The mapping table and the section 'Common misconceptions about softmax' rely on it. - Transformers — generation/logits_process.py — Hugging Face, source checked in November 2026.
TemperatureLogitsWarperraises an error for atemperaturethat is not positive, and for 0.0 it tells you to usedo_sample=False. The question 'What happens if the softmax temperature is set to 0?' relies on it.
Video notes
- 00:00 Opening question — “How do you turn the logits (2.0, 1.0, 0.1) into probabilities?”
- 00:40 Write Eq. (1) on the board — grow with the exponential, divide by the sum, by hand up to 0.659
- 02:00 Eqs. (2) and (3) — the ratio depends only on the difference; a difference of 1 gives · the skill rating analogy
- 03:20 Eq. (4) and Figures 1 and 2 — lower the temperature and you get one-hot, raise it and you get 1/3
- 04:40 Figure 3 — with two classes, it is the sigmoid
- 05:40 Eqs. (6)–(8) — log-sum-exp; 'soft' is on the argmax side
- 06:50 Give
CrossEntropyLosslogits · the of attention - 08:00 Eq. (11) — is inf; subtracting the maximum changes nothing · wrap-up
This is the filming plan. After upload, the timestamps will be replaced with the real ones.