Sigma says to add the terms from the first index to the last
is the value you get by adding every term of a sequence, from the first term to the th term . The below is the start, and the above is the end. Read aloud, it is 'the sum over from 1 to of '. In AI formulas, the mean of the loss, the dot product, and the softmax denominator are all written with this sigma symbol.
| Symbol | Name | Meaning |
|---|---|---|
| summation, sigma notation | The symbol that says to add up all the terms while changing the index | |
| , | dummy index | The number that points to the term being added. It has meaning only inside the sigma |
| , | lower limit, upper limit | The start and end numbers (both included). is the number of terms |
| , | th term | The th value of the sequence being added |
| partial sum (cumulative sum) | The sum from the first term to the th term | |
| constant | A number that stays the same as the index changes | |
| entry | The number in row , column of a grid ( runs from 1 to , from 1 to ) |
Let the sequence 3, 1, 4, 1, 5 be to (). Adding them one at a time from the front gives 3, 4, 8, 9, 14, and these running values are the partial sums . The last one, , is the value of Eq. (1) (Figure 1).
figures/cumsum_bars.pyOne bar in Figure 1 is one pass of a for loop. You start the sum at 0 and add a term each time you raise the index by one. In NumPy it is just np.sum(a) (the section 'Check it in code').11 This page numbers the sequence from , while Python counts from a[0]. The a[0] in the code is the in the formula (Eq. (4)). The Series page also counts its terms from .
Can I write k instead of i inside the sigma?
Yes, you can, and many textbooks write . For the sequence 3, 1, 4, 1, 5, and are both 14. is only a name that points to a term, and once everything is added it is gone. But if you switch to a letter that is already in use (, or the other index of a double sum), the meanings get mixed up.
Five properties of sigma are enough to compute a sum
The five properties are pulling out constant multiples and sums, adding a constant times, shifting the index, swapping the order of a double sum, and expanding a product of sums. The numerical examples also use a second sequence, (sum 7).
Constant multiples and sums come out of the sigma
Multiplying every term by gives the same result as adding everything first and multiplying once. With , the left side is 35, the sum of 8, 2, 9, 5, 11, and the right side is . This property is called linearity.
Adding the same number n times gives n times that number
Even if the term has no in it, the index still runs times. So .
The four 'properties of sigma' in school textbooks (sum, difference, constant multiple, sum of a constant) are Eq. (2) and Eq. (3). The difference rule is Eq. (2) with in place of .
Why is the sum from 1 to n equal to n(n+1)/2?
Because writing the sum from 1 to once more in reverse and adding gives for every pair. (Eqs. (2) and (3)), and the two sums are the same value added in a different order, so you halve it. For , . This is the first of the textbook summation formulas; the others, such as the sum of squares, are not covered on this page.
Shifting the index leaves the sum the same
Switching to lowers the start and the end by one each and raises the subscript by one. The a[j] for , which Python's range(5) runs through, is exactly .
A double sum is the same whichever index you add first
The cells of a grid give the same sum whether you add by rows first or by columns first.22 This holds when there are finitely many cells. A sum that goes on forever (a series) can change its value when you change the order. Figure 2 is a grid of 5 rows and 5 columns with . Adding the row sums 21, 7, 28, 7, 35 gives 98, and so does adding the column sums 28, 0, 14, 42, 14.
figures/double_sum_grid.pyA sum of products is usually not the product of sums
Multiplying two sums as in Eq. (6) produces every pair , which is the whole grid of Figure 2. The sum of products takes only the pairs with the same index, that is, only the diagonal ().
So and 18 differ, and the difference 80 is the sum of the 20 cells off the diagonal. The two are equal only when the off-diagonal cells sum to 0.
Check it in code
Compute Figure 1 and Figure 2 with NumPy.
import numpy as np
a = np.array([3, 1, 4, 1, 5]) # a[0] 이 수식의 a₁
total = 0
for j in range(len(a)): # j = 0 … n-1
total += a[j] # 식 (4): a[j] 가 a_{j+1}
print("for 문 합 :", total)
print("np.sum(a) :", np.sum(a))
print("누적합 :", np.cumsum(a))
print("평균 :", np.sum(a) / len(a))
b = np.array([2, 0, 1, 3, 1])
grid = np.outer(a, b) # 5 × 5 격자, 칸은 a_i × b_j
print("axis=1 (행 합):", np.sum(grid, axis=1))
print("axis=0 (열 합):", np.sum(grid, axis=0))
print("전체 합 :", np.sum(grid)) # 식 (5)
print("곱의 합 :", np.sum(a * b)) # 식 (6) 대각선$ python sigma_sum.py
for 문 합 : 14
np.sum(a) : 14
누적합 : [ 3 4 8 9 14]
평균 : 2.8
axis=1 (행 합): [21 7 28 7 35]
axis=0 (열 합): [28 0 14 42 14]
전체 합 : 98
곱의 합 : 18What's the difference between axis=0 and axis=1 in np.sum?
axis sets which index you add along and remove. On the grid of (Figure 2), axis=1 adds over the column index and leaves 21, 7, 28, 7, 35, one per row; axis=0 adds over the row index and leaves 28, 0, 14, 42, 14. Adding again gives 98 either way (Eq. (5)). In PyTorch this argument is called dim.
Common misconceptions about sigma
- The answer of a sigma still has in it — is a single number, 14. If you write it as , it is 15 for and 25 for , so the value is not fixed.
- A sum of products equals the product of sums — As in Figure 2, they differ: 18 and 98. The product of sums also contains the pairs off the diagonal.
- is 1 — You add 1 times, so it is . It is the case of Eq. (3), and with it is 5.
- Code counts from 1, just like the formula — Looping over
range(1, 5)adds onlya[1]toa[4], giving 11 and leaving out the first term, 3. Count from 0, as in Eq. (4).
Where sigma shows up — starting with the mean and the loss
The mean is (for this sequence, ). Most Σ in AI formulas, too, say to add over each data point, component, or class.
Optional extension: The following table and discussion apply summation notation to AI formulas. They are not required for the school-level notation and sum rules.
| Where | Formula | What is added |
|---|---|---|
| Loss | The mean of the per-example losses . The default reduction='mean' of PyTorch losses | |
| Dot product | Multiply the components of two vectors pairwise and add (a sum of products). For and it is 18 | |
| Softmax denominator | The denominator that makes the probabilities sum to 1. is Euler's number | |
| Expectation | Multiply each value by its probability and add |
Differentiation also passes through sums and constant multiples, as in Eq. (2), so the gradient of the loss is the mean of the per-example gradients. Gradient descent walks against it, and a mini-batch approximates this sum with only some of the . The velocity of momentum is also a sum, of past gradients shrunk step by step.
The symbols in the table are defined on the pages Vectors and Dot Product, Softmax, and Expected Value (Korean).
The symbol that says to multiply is (product notation, capital pi). . If every term is positive, taking the log turns the product into a sum: . The Logarithm page covers it.
Further reading
- Concrete Mathematics, Chapter 2 Sums — Graham, Knuth, Patashnik, 2nd edition 1994. A textbook chapter that sets out sigma notation, manipulating sums (Section 2.3), and double sums and changing their order (Section 2.4). The section 'Five properties of sigma are enough to compute a sum' relies on it. The link is the author's (Knuth's) page introducing the book.
- numpy.sum — NumPy official documentation. It explains how the
axisargument sets which axis to add along, with examples. The question 'What's the difference between axis=0 and axis=1 in np.sum' and the code lab rely on it. - Deep Learning, Chapter 2 Linear Algebra — Goodfellow, Bengio, Courville, 2016. Section 2.2 writes the entries of a matrix product with sigma, and defines the dot product as that matrix product. The dot-product row in the section 'Where sigma shows up' relies on it.
- Deep Learning, Chapter 3 Probability and Information Theory — Goodfellow, Bengio, Courville, 2016. Section 3.8 defines the expectation of a discrete random variable with sigma. The expectation row in the section 'Where sigma shows up' relies on it.