AISoftmax#008

Softmax

소프트맥스

The softmax function turns scores (logits) into probabilities that sum to 1 by dividing each exp(logit) by the sum. Scores (2.0, 1.0, 0.1) become (0.659, 0.242, 0.099). Covers why exp, temperature, the sigmoid, nan, and the derivative.

Key symbol
z
Reading time
14 min
Prerequisites
Vectors and Dot ProductSigma NotationLogarithmConditional ProbabilityExponential FunctionMax and ArgmaxSampling from a Distribution

Translated by an agent and not reviewed yet. Original: 한국어

Intuition — a skill rating looks only at the difference

Intuition

A skill rating in a game (such as the Elo rating in chess) sets the chance of who wins by looking only at the difference between the two players' ratings. Give both players the same extra 100 points and that chance stays the same. The wider the difference, the more it tilts to one side, but it never reaches 0 or 1.

Here a player's rating is the logit, and the chance of winning is the probability that softmax outputs. The 100 points given to everyone is the 'constant added to the logits' that comes up later. Softmax extends this computation from two players to KK candidates.

Say a classification model that decides whether a photo shows a dog, a cat, or a bird gives the three candidates the scores 2.0, 1.0, 0.1. Softmax turns these scores into the probabilities of about 66%, 24%, 10%. A language model does the same computation over its vocabulary each time it picks the next word.

The softmax function turns scores (logits) into probabilities that sum to 1

The softmax function turns the KK scores (logits) a model outputs into probabilities that are all greater than 0 and sum to 1. It takes the exponential ezie^{z_i} of each logit and divides by their sum.

A logit is the score a classification model outputs for each class; it may be negative, and the logits need not sum to 1. The KK logits lined up in a row form the vector z\mathbf{z}, and each number ziz_i in it is a component. The result p\mathbf{p} is a probability distribution (Korean for now): a probability of 0 or more for each outcome, with a total of 1.

pi=softmax⁡(z)i=ezi∑j=1Kezj,i=1,…,K(1)p_i = \operatorname{softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}, \qquad i = 1, \dots, K \tag{1}
Definition
SymbolNameMeaning
z\mathbf{z}, ziz_ilogitThe vector (z1,…,zK)∈RK(z_1, \dots, z_K) \in \mathbb{R}^K of the KK scores. The protagonist of this page
p\mathbf{p}, pip_iprobabilityThe output of softmax. All greater than 0, summing to 1
KK, ii, jjnumber of classes, class indexHow many answers there are to choose from, and the index from 1 to KK. K=3K = 3 in this page's example
∑j=1K\sum_{j=1}^{K}sumAdd in order from j=1j = 1 to KK (the Sigma Notation page)
ex=exp⁡(x)e^{x} = \exp(x)exponential functione≈2.718e \approx 2.718 to the power xx. Always greater than 0, and multiplied by ee when xx grows by 1
log⁡\lognatural logarithmThe logarithm with base ee. It undoes the exponential function, log⁡ex=x\log e^{x} = x. The textbook ln⁡\ln, not the base-10 logarithm (the Logarithm page)
τ\tautemperatureA positive number that divides the logits (Eq. (4)). The default is τ=1\tau = 1. Depending on the text it is written TT

Say the logits of the three classes (dog, cat, bird) are z=(2.0, 1.0, 0.1)\mathbf{z} = (2.0,\ 1.0,\ 0.1). The exponentials are e2≈7.389e^{2} \approx 7.389, e1≈2.718e^{1} \approx 2.718, e0.1≈1.105e^{0.1} \approx 1.105, and their sum is about 11.21. Dividing each by the sum gives p≈(0.659, 0.242, 0.099)\mathbf{p} \approx (0.659,\ 0.242,\ 0.099), and the three values sum to 1.000. The middle bars of Figure 1 are these values.

Probabilities p_i from the softmax of the same logits z = (2.0, 1.0, 0.1) at temperatures τ = 0.5, 1, 2. The middle one (τ = 1) is Eq. (1) as it is, (0.659, 0.242, 0.099). τ = 0.5 tilts toward class 1 and τ = 2 spreads out evenly, but in all three panels p_1 > p_2 > p_3. The p_1/p_2 written on top is e^1/τ, that is, 7.389, 2.718, 1.649. The dashed line is the uniform probability 1/3.
Figure 1 Probabilities pip_i from the softmax of the same logits z=(2.0, 1.0, 0.1)\mathbf{z} = (2.0,\ 1.0,\ 0.1) at temperatures τ=0.5, 1, 2\tau = 0.5,\ 1,\ 2. The middle one (τ=1\tau = 1) is Eq. (1) as it is, (0.659, 0.242, 0.099)(0.659,\ 0.242,\ 0.099). τ=0.5\tau = 0.5 tilts toward class 1 and τ=2\tau = 2 spreads out evenly, but in all three panels p1>p2>p3p_1 > p_2 > p_3. The p1/p2p_1/p_2 written on top is e1/τe^{1/\tau}, that is, 7.389, 2.718, 1.649. The dashed line is the uniform probability 1/3. figures/temperature_bars.py

The ratio of probabilities is set only by the difference of logits

The ratio of two probabilities is the exponential of the difference of their logits. In the example z1−z2=1z_1 - z_2 = 1, so p1/p2=e≈2.718p_1/p_2 = e \approx 2.718.

pipj=eziezj=e zi−zj(2)\frac{p_i}{p_j} = \frac{e^{z_i}}{e^{z_j}} = e^{\,z_i - z_j} \tag{2}

That is because the denominator of Eq. (1) cancels between the top and the bottom. Read the other way, the difference of logits is the log of the probability ratio, log⁡(pi/pj)\log(p_i/p_j). Since z2−z3=0.9z_2 - z_3 = 0.9, p2/p3=e0.9≈2.46p_2/p_3 = e^{0.9} \approx 2.46.

Three properties follow from Eq. (1) and Eq. (2). The exponential is greater than 0, so every pi>0p_i > 0. If zi>zjz_i > z_j, then pi/pj>1p_i/p_j > 1, so the order is kept. Adding the same number cc to every logit leaves the differences, and so the ratios, unchanged, and since the sum is 1, the probabilities are unchanged too.

softmax⁡(z+c)=softmax⁡(z)(3)\operatorname{softmax}(\mathbf{z} + c) = \operatorname{softmax}(\mathbf{z}) \tag{3}

Here z+c\mathbf{z} + c is the vector with cc added to each component. Conversely, raising just one logit by 1 does not multiply its probability by ee. Raising z1z_1 from 2 to 3 takes p1p_1 from 0.659 to 0.840, about 1.275 times, because the denominator grows too.

Key idea

Softmax looks only at the differences of the logits. The ratio of two probabilities, pi/pjp_i/p_j, is the exponential of the logit difference, ezi−zje^{z_i - z_j}.

Why does softmax use the exponential (exp)?

Because the exponential turns any real number into a number greater than 0, and turns differences into ratios (ea/eb=ea−be^{a}/e^{b} = e^{a-b}). That is why Eq. (2) and Eq. (3) hold. If you divide by the sum without the exponential, (2.0, 1.0, 0.1)(2.0,\ 1.0,\ 0.1) gives (0.645, 0.323, 0.032)(0.645,\ 0.323,\ 0.032), but (1.0, 0.0, −0.9)(1.0,\ 0.0,\ -0.9), with 1 subtracted from each, sums to 0.1 and gives (10, 0, −9)(10,\ 0,\ -9).

Why lowering the softmax temperature lets the largest logit take all the probability

Because dividing the logits by the temperature τ\tau multiplies their differences by 1/τ1/\tau, and the ratio in Eq. (2) becomes e(zi−zj)/τe^{(z_i - z_j)/\tau}. Shrinking τ\tau toward 0 sends the probability of the largest logit (when there is only one) to 1, and raising τ\tau sends every probability to 1/K1/K. In terms of the skill rating in the intuition, the temperature is the unit, in points, in which the rating difference is measured.

pi=ezi/τ∑j=1Kezj/τ,τ>0(4)p_i = \frac{e^{z_i/\tau}}{\sum_{j=1}^{K} e^{z_j/\tau}}, \qquad \tau > 0 \tag{4}

The left panel of Figure 1 (τ=0.5\tau = 0.5) puts z/τ=(4, 2, 0.2)\mathbf{z}/\tau = (4,\ 2,\ 0.2) into Eq. (1) and tilts to (0.864, 0.117, 0.019)(0.864,\ 0.117,\ 0.019). The right panel (τ=2\tau = 2) puts in (1, 0.5, 0.05)(1,\ 0.5,\ 0.05) and spreads out to (0.502, 0.304, 0.194)(0.502,\ 0.304,\ 0.194).

Figure 2 shows the bars while τ\tau rises from 0.1 to 10. The low end is close to the one-hot vector (1, 0, 0)(1,\ 0,\ 0), which is 1 only at the largest logit, and the high end is close to an even 1/3.

Figure 2 Softmax bars of the same logits, drawn while the temperature τ\tau rises from 0.1 to 10 in log-spaced steps (5 seconds). At τ=0.1\tau = 0.1, p1≈0.99995p_1 \approx 0.99995, close to one-hot, and at τ=10\tau = 10 the bars approach the dashed 1/3, at (0.366, 0.331, 0.303)(0.366,\ 0.331,\ 0.303). In every frame p1>p2>p3p_1 > p_2 > p_3. figures/temperature_sweep.py

In AI the temperature shows up in two places. The temperature in a language model's generation settings is the τ\tau of Eq. (4) (the Sampling from a Distribution page, Korean for now). Attention in a transformer divides its dot products by dk\sqrt{d_k} (dkd_k is the dimension of the key vectors), which, seen through Eq. (4), also sits in the place of the temperature (Vaswani et al. 2017).

Does raising the softmax temperature change the ranking?

No. Raising or lowering the softmax temperature leaves the ranking of the probabilities as it is. Dividing by a positive τ\tau only scales all the logit differences by the same factor, so p1>p2>p3p_1 > p_2 > p_3 anywhere for τ\tau between 0.1 and 10 (Figure 2). The probability of the lowest class grows (p3p_3 goes from 0.099 to 0.194 at τ=2\tau = 2), but it does not overtake the others.

What happens if the softmax temperature is set to 0?

Setting the softmax temperature τ\tau to exactly 0 divides the logits by 0, so the value is undefined. Instead, shrinking τ\tau close to 0 approaches the one-hot in which the single largest logit takes 1; in the example, τ=0.05\tau = 0.05 gives p1≈0.999999998p_1 \approx 0.999999998. Hugging Face Transformers raises an error for temperature=0.0 and tells you to use greedy decoding (do_sample=False), which picks only the largest logit.

With two classes, softmax is the sigmoid

With K=2K = 2, the softmax p1p_1 is the sigmoid of the logit difference z1−z2z_1 - z_2. It follows directly from dividing the numerator and the denominator of Eq. (1) by ez1e^{z_1}.

p1=ez1ez1+ez2=11+e−(z1−z2)=σ(z1−z2)(5)p_1 = \frac{e^{z_1}}{e^{z_1} + e^{z_2}} = \frac{1}{1 + e^{-(z_1 - z_2)}} = \sigma(z_1 - z_2) \tag{5}
Definition
SymbolNameMeaning
σ\sigmasigmoidσ(x)=1/(1+e−x)\sigma(x) = 1/(1 + e^{-x}). An S-shaped curve between 0 and 1. Also called the logistic function

Figure 3 draws this curve for each temperature (with a temperature it is σ((z1−z2)/τ)\sigma((z_1 - z_2)/\tau)). The smaller τ\tau, the steeper.

At the example's difference of 1.0 with τ=1\tau = 1, p1=0.731p_1 = 0.731. That differs from the 0.659 with three classes, but in both cases p1/p2p_1/p_2 is exactly ee (Eq. (2)).

With two classes, p_1 as a function of the logit difference z_1 - z_2, at temperatures τ = 0.5, 1, 2. All three curves pass through 0.5 at difference 0, and the smaller τ, the steeper. At the example's difference of 1.0, p_1 is 0.881, 0.731, 0.622 in that order.
Figure 3 With two classes, p1p_1 as a function of the logit difference z1−z2z_1 - z_2, at temperatures τ=0.5, 1, 2\tau = 0.5,\ 1,\ 2. All three curves pass through 0.5 at difference 0, and the smaller τ\tau, the steeper. At the example's difference of 1.0, p1p_1 is 0.881, 0.731, 0.622 in that order. figures/sigmoid_family.py

What's the difference between softmax and the sigmoid?

With two classes they are the same function. The softmax p1p_1 is the sigmoid of the logit difference, σ(z1−z2)\sigma(z_1 - z_2) (Eq. (5)), which is 0.731 for a difference of 1.0. That is why binary classification outputs just one logit and uses the sigmoid. When one input can have several correct answers (multi-label), a separate sigmoid is used for each class, and the probabilities need not sum to 1.

Softmax is a smoothed argmax, not a smoothed max

What softmax smooths is not the largest value (max) but the position of that value, the argmax. The argmax of the example is position 1 (0 in code, which counts from 0), and written as a one-hot that puts 1 only at that position it is (1, 0, 0)(1,\ 0,\ 0). The softmax (0.659, 0.242, 0.099)(0.659,\ 0.242,\ 0.099) is a smoothed-out version of this one-hot. 'Soft' means smooth enough to differentiate, and the name is said to come from Bridle (1990).11 Goodfellow et al. (Section 6.2.2.3) write that it would be better to call this function softargmax.

The function that smoothly approximates max itself is log-sum-exp (LSE). The log-softmax is exactly the logits minus LSE.

log⁡pi=zi−LSE⁡(z),LSE⁡(z)=log⁡∑j=1Kezj(6)\log p_i = z_i - \operatorname{LSE}(\mathbf{z}), \qquad \operatorname{LSE}(\mathbf{z}) = \log \sum_{j=1}^{K} e^{z_j} \tag{6} max⁡jzj  ≤  LSE⁡(z)  ≤  max⁡jzj+log⁡K(7)\max_j z_j \;\le\; \operatorname{LSE}(\mathbf{z}) \;\le\; \max_j z_j + \log K \tag{7}

In the example, LSE⁡(z)=log⁡11.21≈2.417\operatorname{LSE}(\mathbf{z}) = \log 11.21 \approx 2.417. As Eq. (7) says, it is larger than the max 2.0 and smaller than 2.0+log⁡3≈3.0992.0 + \log 3 \approx 3.099. The log-softmax is (−0.417, −1.417, −2.317)(-0.417,\ -1.417,\ -2.317), each logit minus 2.417. How to compute LSE without overflow is covered on the Logarithm page.

Taking the partial derivative of LSE with respect to ziz_i gives the softmax. The partial derivative ∂\partial is the derivative when only ziz_i moves and the other logits stay fixed (the Gradient page, Korean for now).

∂LSE⁡(z)∂zi=pi(8)\frac{\partial \operatorname{LSE}(\mathbf{z})}{\partial z_i} = p_i \tag{8}

Differentiating max (when there is a single largest logit) gives the one-hot that is 1 only at that position, that is, the argmax written as a one-hot. In the same way, differentiating LSE, the smoothed max, gives softmax, the smoothed argmax. How all the probabilities move when one logit moves, that is, the derivative of softmax, is in the folded Eq. (9) below.

The derivative of softmax (the Jacobian) — how the probabilities move when one logit moves

δij\delta_{ij} is 1 if i=ji = j and 0 otherwise (the Kronecker delta).

Differentiating Eq. (6) with respect to zjz_j and using Eq. (8) gives ∂log⁡pi/∂zj=δij−pj\partial \log p_i / \partial z_j = \delta_{ij} - p_j. The derivative of the log is ∂log⁡pi=∂pi/pi\partial \log p_i = \partial p_i / p_i, so multiplying both sides by pip_i gives the Jacobian (the table of partial derivatives).

∂pi∂zj=pi (δij−pj)(9)\frac{\partial p_i}{\partial z_j} = p_i\,(\delta_{ij} - p_j) \tag{9}

In the example, ∂p1/∂z1=p1(1−p1)≈0.225\partial p_1/\partial z_1 = p_1(1 - p_1) \approx 0.225, ∂p2/∂z1=−p2p1≈−0.160\partial p_2/\partial z_1 = -p_2 p_1 \approx -0.160, and ∂p3/∂z1=−p3p1≈−0.065\partial p_3/\partial z_1 = -p_3 p_1 \approx -0.065. So raising z1z_1 by 0.01 raises p1p_1 by about 0.0022, and p2p_2 and p3p_3 together go down by the same amount. That each column (fixed jj) sums to 0 is the derivative of the condition that the sum is 1, and that each row (fixed ii) sums to 0 is the derivative of Eq. (3).

Common misconceptions about softmax

  • If softmax outputs 0.9, the prediction is right with 90% probability — Multiplying the logits by 2 with Eq. (4) (τ=0.5\tau = 0.5) keeps the prediction at class 1, but p1p_1 rises from 0.659 to 0.864. The size of the probabilities depends on the scale of the logits, so matching them to the actual accuracy takes a separate calibration such as temperature scaling.
  • You should apply softmax before passing values to CrossEntropyLoss — nn.CrossEntropyLoss applies log-softmax inside, so feeding it probabilities applies softmax twice. The probability of the correct answer shrinks to e0.659/(e0.659+e0.242+e0.099)≈0.4484e^{0.659}/(e^{0.659} + e^{0.242} + e^{0.099}) \approx 0.4484, and the example's loss changes from 0.417 to 0.802. With three classes it can never go below 0.551 however long you train (Try it yourself 1). Why it takes logits is answered by a question on the Logarithm page.

Where softmax shows up in AI

The last layer (output layer) of a classification network outputs one logit per class. That layer is the linear layer of the Matrix Operations page, and that page's output y\mathbf{y} is the z\mathbf{z} here. Softmax turns the logits into probabilities, and training reduces the cross-entropy loss L=−∑iyilog⁡piL = -\sum_i y_i \log p_i (Cross-Entropy page — planned).

On this page y\mathbf{y} is the one-hot vector that is 1 only at the correct answer, so ∑iyi=1\sum_i y_i = 1, and putting in Eq. (6) gives L=LSE⁡(z)−∑iyiziL = \operatorname{LSE}(\mathbf{z}) - \sum_i y_i z_i. Differentiating softmax together with the cross-entropy then takes a single line, thanks to Eq. (8).

∂L∂z=p−y(10)\frac{\partial L}{\partial \mathbf{z}} = \mathbf{p} - \mathbf{y} \tag{10}

If the correct answer is class 1 (dog), y=(1, 0, 0)\mathbf{y} = (1,\ 0,\ 0). The loss is −log⁡0.659≈0.417-\log 0.659 \approx 0.417, and the gradient is (−0.341, 0.242, 0.099)(-0.341,\ 0.242,\ 0.099). Backpropagation (Korean for now) carries this value back to the weights, and gradient descent moves the weights in the opposite direction. Seen from the logits, that raises the correct logit and lowers the others.

Why does the output layer of a multi-class classifier use the softmax function?

Because the softmax function turns the per-class scores into positive probabilities that sum to 1. In multi-class classification, which picks one out of three or more, using it as the activation function of the output layer lets you read the outputs directly as per-class probabilities. The example's scores (2.0, 1.0, 0.1)(2.0,\ 1.0,\ 0.1) become 0.659 for dog, 0.242 for cat, and 0.099 for bird.

Attention in a transformer also applies softmax over each row of the table of query–key dot products, QK⊤\mathbf{Q}\mathbf{K}^\top, to make the weights that mix the value vectors (the Matrix Operations page; the bold K\mathbf{K} is the key matrix, different from the number of classes KK). A language model outputs one logit for every token in its vocabulary each time it picks the next token, and softmax turns them into per-token probabilities.

This pageCode
Eq. (1)torch.softmax(z, dim=-1) (= torch.nn.functional.softmax), scipy.special.softmax(z, axis=-1) (the whole array if axis is omitted). dim and axis set the axis along which the sum is made 1
Eq. (6)torch.nn.functional.log_softmax(z, dim=-1) — it does not compute log(softmax(z)) separately, but computes Eq. (6) with LSE after subtracting the maximum
LSE⁡\operatorname{LSE}torch.logsumexp(z, dim=-1), scipy.special.logsumexp(z, axis=-1) (the whole array if axis is omitted)
z\mathbf{z}The first argument of nn.CrossEntropyLoss()(z, target) (in functional form, F.cross_entropy(z, target)). The logits before softmax; target is the index of the correct class (from 0)
Eq. (10)The value that ends up in z.grad after loss.backward() on the loss above. The z the model outputs is an intermediate tensor, so you must call z.retain_grad() first to keep it. With the default reduction="mean", it is divided by the batch size
τ\tautorch.softmax(z / tau, dim=-1) — the temperature of language-model generation
dk\sqrt{d_k}torch.softmax(Q @ K.transpose(-2, -1) / d_k ** 0.5, dim=-1) in attention

Why subtract the maximum when softmax gives nan

When the logits are large, the exponentials overflow to inf (infinity), and inf ÷ inf = nan (not a number). Thanks to Eq. (3), subtracting the largest logit mm from all of them gives the same result, and then the exponentials do not overflow (where the overflow starts is on the Logarithm page).

pi=ezi−m∑j=1Kezj−m,m=max⁡jzj(11)p_i = \frac{e^{z_i - m}}{\sum_{j=1}^{K} e^{z_j - m}}, \qquad m = \max_j z_j \tag{11}

Compute z+998=(1000, 999, 998.1)\mathbf{z} + 998 = (1000,\ 999,\ 998.1), the example plus 998, directly with Eq. (1), and the numerator and the denominator are both inf, so the result is nan. Eq. (11) subtracts m=1000m = 1000 and uses the exponentials of (0, −1, −1.9)(0,\ -1,\ -1.9), so the result is (0.659, 0.242, 0.099)(0.659,\ 0.242,\ 0.099), the same as for z\mathbf{z}.

After subtracting the maximum, every exponential is at most e0=1e^{0} = 1, and the denominator contains one e0=1e^{0} = 1, so there is no division by 0 either. Conversely, if you compute z−1000\mathbf{z} - 1000 directly, all the exponentials drop to 0 (underflow), and 0 ÷ 0 = nan.22 Library functions such as torch.softmax and scipy.special.softmax do this inside.

When using softmax in PyTorch, watch the axis and the log

The common mistakes in practice are the axis along which the sum is made 1, and the order of computing the probabilities first and then taking the log.

What does dim=-1 mean in torch.softmax?

dim=-1 in torch.softmax means that the axis along which the sum is made 1 is the last axis (the classes). Give dim=-1 to a 2×3 array that stacks the example logits twice, and each row comes out as (0.659, 0.242, 0.099)(0.659,\ 0.242,\ 0.099). Give dim=0, and the sum is made 1 across the two samples for each class, so every cell becomes the meaningless value 0.5.

Why use log_softmax (log-softmax) separately?

Because if you compute the probabilities first and then take the log, a very small probability drops to 0 and becomes log⁡0=−∞\log 0 = -\infty. With the logits (2000, 1000, 100)(2000,\ 1000,\ 100) the softmax p2p_2 becomes 0, but log_softmax gives log⁡p2≈−1000\log p_2 \approx -1000 as it is (the computation is on the Logarithm page). The PyTorch documentation also says to put log_softmax, not softmax, in front of NLLLoss.

Check it in code

Change tau in the code to make the three bar charts of Figure 1 yourself. The '빼지 않음' (without subtracting) line of the output is nan.

Math
pi=ezi/τ∑j=1Kezj/τ(4)\sym{probs}{p}_i = \frac{e^{\sym{logits}{z}_i/\sym{tau}{\tau}}}{\sum_{j=1}^{K} e^{\sym{logits}{z}_j/\sym{tau}{\tau}}} \tag{4}pi=ezi−m∑j=1Kezj−m,m=max⁡jzj(11)\sym{probs}{p}_i = \frac{e^{\sym{logits}{z}_i - m}}{\sum_{j=1}^{K} e^{\sym{logits}{z}_j - m}}, \qquad m = \max_j \sym{logits}{z}_j \tag{11}
Code figures/softmax.py
import numpy as np

def softmax(logits, tau=1.0):
    s = logits / tau              # 식 (4) 온도
    ex = np.exp(s - s.max())      # 식 (11) 최댓값 빼기
    return ex / ex.sum()          # 식 (1)

logits = np.array([2.0, 1.0, 0.1])     # z
probs = softmax(logits)                # p
print("p        ", probs.round(3), probs.sum())
print("p1 / p2  ", round(probs[0] / probs[1], 3))  # 식 (2)
print("z + 998  ", softmax(logits + 998).round(3))  # 식 (3)
with np.errstate(over="ignore", invalid="ignore"):
    raw = np.exp(logits + 998)         # e^1000 = inf
    print("빼지 않음", raw / raw.sum())
for tau in (0.5, 1.0, 2.0):
    print("tau", tau, " ", softmax(logits, tau).round(3))
$ python softmax.py
p         [0.659 0.242 0.099] 1.0
p1 / p2   2.718
z + 998   [0.659 0.242 0.099]
빼지 않음 [nan nan nan]
tau 0.5   [0.864 0.117 0.019]
tau 1.0   [0.659 0.242 0.099]
tau 2.0   [0.502 0.304 0.194]

Try it yourself

  1. Compute the softmax of the logits (1, 0, 0)(1,\ 0,\ 0) by hand. What about the logits (2.0, 1.0, 1.0)(2.0,\ 1.0,\ 1.0)?
  2. With two classes, what must z1−z2z_1 - z_2 be for p1=0.9p_1 = 0.9? And if the temperature is τ=2\tau = 2?
Answers
  1. The denominator is e+1+1≈4.718e + 1 + 1 \approx 4.718, so it is (e/(e+2), 1/(e+2), 1/(e+2))≈(0.576, 0.212, 0.212)(e/(e+2),\ 1/(e+2),\ 1/(e+2)) \approx (0.576,\ 0.212,\ 0.212). Subtracting 1 from every entry of (2.0, 1.0, 1.0)(2.0,\ 1.0,\ 1.0) gives (1, 0, 0)(1,\ 0,\ 0), so by Eq. (3) it is the same value. The 0.551 in the common misconceptions is the loss when the one-hot that is 1 only at the correct answer is fed to CrossEntropyLoss as probabilities, −log⁡ee+2=log⁡(1+2/e)≈0.551-\log\frac{e}{e+2} = \log(1 + 2/e) \approx 0.551.
  2. From Eq. (2), p1/p2=0.9/0.1=9=ez1−z2p_1/p_2 = 0.9/0.1 = 9 = e^{z_1 - z_2}, so z1−z2=log⁡9≈2.197z_1 - z_2 = \log 9 \approx 2.197. With τ=2\tau = 2, Eq. (4) gives (z1−z2)/2=log⁡9(z_1 - z_2)/2 = \log 9, so it is twice that, about 4.394.

Further reading

  • Deep Learning, Chapter 6 Deep Feedforward Networks — Goodfellow, Bengio, Courville, 2016. Section 6.2.2.3 covers softmax in the output layer for classification. It reads logits as unnormalized log probabilities, writes the loss with log-softmax, and shows that adding the same number to every input leaves the output unchanged. Section 4.1 of Chapter 4 Numerical Computation in the same book gives stabilizing softmax by subtracting the maximum as an example. The sections 'Why subtract the maximum when softmax gives nan' and 'Softmax is a smoothed argmax, not a smoothed max' rely on it.
  • Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition — J. S. Bridle, 1990. Neurocomputing (NATO ASI Series F68), Springer. Known as the paper in which the name 'softmax' appeared. The story of the name in the section 'Softmax is a smoothed argmax, not a smoothed max' relies on it.
  • Attention Is All You Need — Vaswani et al., 2017. NeurIPS. The scaled dot-product attention of Section 3.2.1 applies softmax to dot products divided by dk\sqrt{d_k}, and offers a guess at why it divides. The temperature section and the attention sentence of the section 'Where softmax shows up in AI' rely on it.
  • On Calibration of Modern Neural Networks — Guo, Pleiss, Sun, Weinberger, 2017. ICML. It measures the tendency of the softmax probabilities of modern neural networks to come out higher than the actual accuracy, and proposes temperature scaling, which calibrates them by dividing the logits by a single temperature. The first item of the section 'Common misconceptions about softmax' relies on it.
  • torch.nn.CrossEntropyLoss — PyTorch official documentation. It states that the input should be unnormalized logits (checked against version 2.15), and that this loss is equivalent to applying LogSoftmax followed by NLLLoss. The softmax and log_softmax documentation explain the dim argument and why log_softmax is used separately. The mapping table and the section 'Common misconceptions about softmax' rely on it.
  • Transformers — generation/logits_process.py — Hugging Face, source checked in November 2026. TemperatureLogitsWarper raises an error for a temperature that is not positive, and for 0.0 it tells you to use do_sample=False. The question 'What happens if the softmax temperature is set to 0?' relies on it.

Video notes

Video notes
  • 00:00 Opening question — “How do you turn the logits (2.0, 1.0, 0.1) into probabilities?”
  • 00:40 Write Eq. (1) on the board — grow with the exponential, divide by the sum, by hand up to 0.659
  • 02:00 Eqs. (2) and (3) — the ratio depends only on the difference; a difference of 1 gives ee · the skill rating analogy
  • 03:20 Eq. (4) and Figures 1 and 2 — lower the temperature and you get one-hot, raise it and you get 1/3
  • 04:40 Figure 3 — with two classes, it is the sigmoid
  • 05:40 Eqs. (6)–(8) — log-sum-exp; 'soft' is on the argmax side
  • 06:50 Give CrossEntropyLoss logits · the dk\sqrt{d_k} of attention
  • 08:00 Eq. (11) — e1000e^{1000} is inf; subtracting the maximum changes nothing · wrap-up

This is the filming plan. After upload, the timestamps will be replaced with the real ones.

Where this sits on the map