CalculusSigma Notation#006

Sigma Notation

시그마 (수열의 합)

Sigma (Σ) adds a sequence's terms from the first index to the last. 3, 1, 4, 1, 5 sums to 14. Five properties of sigma and the sum n(n+1)/2, checked with the same numbers in code.

Reading time
7 min

Translated by an agent and not reviewed yet. Original: 한국어

Sigma says to add the terms from the first index to the last

∑i=1nai\sum_{i=1}^{n} a_i is the value you get by adding every term of a sequence, from the first term a1a_1 to the nnth term ana_n. The i=1i = 1 below is the start, and the nn above is the end. Read aloud, it is 'the sum over ii from 1 to nn of aia_i'. In AI formulas, the mean of the loss, the dot product, and the softmax denominator are all written with this sigma symbol.

∑i=1nai=a1+a2+⋯+an(1)\sum_{i=1}^{n} a_i = a_1 + a_2 + \cdots + a_n \tag{1}
Definition
SymbolNameMeaning
∑\sumsummation, sigma notationThe symbol that says to add up all the terms while changing the index
ii, jjdummy indexThe number that points to the term being added. It has meaning only inside the sigma
11, nnlower limit, upper limitThe start and end numbers (both included). nn is the number of terms
aia_i, bib_iiith termThe iith value of the sequence being added
SkS_kpartial sum (cumulative sum)The sum from the first term to the kkth term
ccconstantA number that stays the same as the index changes
AijA_{ij}entryThe number in row ii, column jj of a grid (ii runs from 1 to mm, jj from 1 to nn)

Let the sequence 3, 1, 4, 1, 5 be a1a_1 to a5a_5 (n=5n = 5). Adding them one at a time from the front gives 3, 4, 8, 9, 14, and these running values are the partial sums SkS_k. The last one, S5=14S_5 = 14, is the value of Eq. (1) (Figure 1).

Partial sums S_k of the sequence a_1, …, a_5 = 3, 1, 4, 1, 5. The blue block is the term a_k added at that turn, and the gray one is the sum before it, S_k-1.
Figure 1 Partial sums SkS_k of the sequence a1,…,a5=3,1,4,1,5a_1, \dots, a_5 = 3, 1, 4, 1, 5. The blue block is the term aka_k added at that turn, and the gray one is the sum before it, Sk−1S_{k-1}. figures/cumsum_bars.py

One bar in Figure 1 is one pass of a for loop. You start the sum at 0 and add a term each time you raise the index by one. In NumPy it is just np.sum(a) (the section 'Check it in code').11 This page numbers the sequence from a1a_1, while Python counts from a[0]. The a[0] in the code is the a1a_1 in the formula (Eq. (4)). The Series page also counts its terms from a1a_1.

Can I write k instead of i inside the sigma?

Yes, you can, and many textbooks write kk. For the sequence 3, 1, 4, 1, 5, ∑i=15ai\sum_{i=1}^{5} a_i and ∑k=15ak\sum_{k=1}^{5} a_k are both 14. ii is only a name that points to a term, and once everything is added it is gone. But if you switch to a letter that is already in use (nn, or the other index of a double sum), the meanings get mixed up.

Five properties of sigma are enough to compute a sum

The five properties are pulling out constant multiples and sums, adding a constant nn times, shifting the index, swapping the order of a double sum, and expanding a product of sums. The numerical examples also use a second sequence, b1,…,b5=2,0,1,3,1b_1, \dots, b_5 = 2, 0, 1, 3, 1 (sum 7).

Constant multiples and sums come out of the sigma

∑i=1n(c ai+bi)=c∑i=1nai+∑i=1nbi(2)\sum_{i=1}^{n} (c\,a_i + b_i) = c \sum_{i=1}^{n} a_i + \sum_{i=1}^{n} b_i \tag{2}

Multiplying every term by cc gives the same result as adding everything first and multiplying once. With c=2c = 2, the left side is 35, the sum of 8, 2, 9, 5, 11, and the right side is 2×14+7=352 \times 14 + 7 = 35. This property is called linearity.

Adding the same number n times gives n times that number

∑i=1nc=nc(3)\sum_{i=1}^{n} c = nc \tag{3}

Even if the term has no ii in it, the index still runs nn times. So ∑i=152=10\sum_{i=1}^{5} 2 = 10.

The four 'properties of sigma' in school textbooks (sum, difference, constant multiple, sum of a constant) are Eq. (2) and Eq. (3). The difference rule is Eq. (2) with −bi-b_i in place of bib_i.

Why is the sum from 1 to n equal to n(n+1)/2?

Because writing the sum from 1 to nn once more in reverse and adding gives n+1n + 1 for every pair. ∑k=1nk+∑k=1n(n+1−k)=∑k=1n(n+1)=n(n+1)\sum_{k=1}^{n} k + \sum_{k=1}^{n} (n+1-k) = \sum_{k=1}^{n} (n+1) = n(n+1) (Eqs. (2) and (3)), and the two sums are the same value added in a different order, so you halve it. For n=5n = 5, 1+2+3+4+5=15=5×6/21 + 2 + 3 + 4 + 5 = 15 = 5 \times 6 / 2. This is the first of the textbook summation formulas; the others, such as the sum of squares, are not covered on this page.

Shifting the index leaves the sum the same

∑i=1nai=∑j=0n−1aj+1(4)\sum_{i=1}^{n} a_i = \sum_{j=0}^{n-1} a_{j+1} \tag{4}

Switching to j=i−1j = i - 1 lowers the start and the end by one each and raises the subscript by one. The a[j] for j=0,…,4j = 0, \dots, 4, which Python's range(5) runs through, is exactly aj+1a_{j+1}.

A double sum is the same whichever index you add first

∑i=1m∑j=1nAij=∑j=1n∑i=1mAij(5)\sum_{i=1}^{m} \sum_{j=1}^{n} A_{ij} = \sum_{j=1}^{n} \sum_{i=1}^{m} A_{ij} \tag{5}

The cells of a grid give the same sum whether you add by rows first or by columns first.22 This holds when there are finitely many cells. A sum that goes on forever (a series) can change its value when you change the order. Figure 2 is a grid of 5 rows and 5 columns with Aij=aibjA_{ij} = a_i b_j. Adding the row sums 21, 7, 28, 7, 35 gives 98, and so does adding the column sums 28, 0, 14, 42, 14.

A grid built from a_1, …, a_5 = 3, 1, 4, 1, 5 (left column) and b_1, …, b_5 = 2, 0, 1, 3, 1 (top row). Each cell is a_i b_j, the right column holds the row sums, and the bottom row holds the column sums. Either way the total is 98, and adding only the highlighted diagonal gives 18.
Figure 2 A grid built from a1,…,a5=3,1,4,1,5a_1, \dots, a_5 = 3, 1, 4, 1, 5 (left column) and b1,…,b5=2,0,1,3,1b_1, \dots, b_5 = 2, 0, 1, 3, 1 (top row). Each cell is aibja_i b_j, the right column holds the row sums, and the bottom row holds the column sums. Either way the total is 98, and adding only the highlighted diagonal gives 18. figures/double_sum_grid.py

A sum of products is usually not the product of sums

(∑i=1nai)(∑j=1nbj)=∑i=1n∑j=1naibj(6)\Big(\sum_{i=1}^{n} a_i\Big)\Big(\sum_{j=1}^{n} b_j\Big) = \sum_{i=1}^{n} \sum_{j=1}^{n} a_i b_j \tag{6}

Multiplying two sums as in Eq. (6) produces every pair aibja_i b_j, which is the whole grid of Figure 2. The sum of products ∑i=1naibi\sum_{i=1}^{n} a_i b_i takes only the pairs with the same index, that is, only the diagonal (6+0+4+3+5=186 + 0 + 4 + 3 + 5 = 18).

So 14×7=9814 \times 7 = 98 and 18 differ, and the difference 80 is the sum of the 20 cells off the diagonal. The two are equal only when the off-diagonal cells sum to 0.

Check it in code

Compute Figure 1 and Figure 2 with NumPy.

Math
Sn=∑j=0n−1aj+1(4)\sym{total}{S_n} = \sum_{\sym{j}{j}=0}^{n-1} \sym{a}{a_{j+1}} \tag{4}
Code figures/sigma_sum.py
import numpy as np

a = np.array([3, 1, 4, 1, 5])   # a[0] 이 수식의 a₁
total = 0
for j in range(len(a)):         # j = 0 … n-1
    total += a[j]               # 식 (4): a[j] 가 a_{j+1}
print("for 문 합 :", total)
print("np.sum(a) :", np.sum(a))
print("누적합    :", np.cumsum(a))
print("평균      :", np.sum(a) / len(a))

b = np.array([2, 0, 1, 3, 1])
grid = np.outer(a, b)           # 5 × 5 격자, 칸은 a_i × b_j
print("axis=1 (행 합):", np.sum(grid, axis=1))
print("axis=0 (열 합):", np.sum(grid, axis=0))
print("전체 합 :", np.sum(grid))     # 식 (5)
print("곱의 합 :", np.sum(a * b))    # 식 (6) 대각선
$ python sigma_sum.py
for 문 합 : 14
np.sum(a) : 14
누적합    : [ 3  4  8  9 14]
평균      : 2.8
axis=1 (행 합): [21  7 28  7 35]
axis=0 (열 합): [28  0 14 42 14]
전체 합 : 98
곱의 합 : 18

What's the difference between axis=0 and axis=1 in np.sum?

axis sets which index you add along and remove. On the grid of aibja_i b_j (Figure 2), axis=1 adds over the column index jj and leaves 21, 7, 28, 7, 35, one per row; axis=0 adds over the row index ii and leaves 28, 0, 14, 42, 14. Adding again gives 98 either way (Eq. (5)). In PyTorch this argument is called dim.

Common misconceptions about sigma

  • The answer of a sigma still has ii in it — ∑i=15ai\sum_{i=1}^{5} a_i is a single number, 14. If you write it as 5ai5a_i, it is 15 for a1a_1 and 25 for a5a_5, so the value is not fixed.
  • A sum of products equals the product of sums — As in Figure 2, they differ: 18 and 98. The product of sums also contains the pairs off the diagonal.
  • ∑i=1n1\sum_{i=1}^{n} 1 is 1 — You add 1 nn times, so it is nn. It is the case c=1c = 1 of Eq. (3), and with n=5n = 5 it is 5.
  • Code counts from 1, just like the formula — Looping over range(1, 5) adds only a[1] to a[4], giving 11 and leaving out the first term, 3. Count from 0, as in Eq. (4).

Where sigma shows up — starting with the mean and the loss

The mean is 1n∑i=1nai\frac{1}{n}\sum_{i=1}^{n} a_i (for this sequence, 14/5=2.814/5 = 2.8). Most Σ in AI formulas, too, say to add over each data point, component, or class.

Optional extension: The following table and discussion apply summation notation to AI formulas. They are not required for the school-level notation and sum rules.

WhereFormulaWhat is added
LossL=1n∑i=1nℓiL = \frac{1}{n}\sum_{i=1}^{n} \ell_iThe mean of the per-example losses ℓi\ell_i. The default reduction='mean' of PyTorch losses
Dot product∑i=1nxiyi\sum_{i=1}^{n} x_i y_iMultiply the components of two vectors pairwise and add (a sum of products). For aa and bb it is 18
Softmax denominator∑j=1Kezj\sum_{j=1}^{K} e^{z_j}The denominator that makes the probabilities sum to 1. e≈2.718e \approx 2.718 is Euler's number
Expectation∑ipixi\sum_{i} p_i x_iMultiply each value xix_i by its probability pip_i and add

Differentiation also passes through sums and constant multiples, as in Eq. (2), so the gradient of the loss is the mean of the per-example gradients. Gradient descent walks against it, and a mini-batch approximates this sum with only some of the ii. The velocity of momentum is also a sum, of past gradients shrunk step by step.

The symbols in the table are defined on the pages Vectors and Dot Product, Softmax, and Expected Value (Korean).

The symbol that says to multiply is ∏\prod (product notation, capital pi). ∏i=15ai=3×1×4×1×5=60\prod_{i=1}^{5} a_i = 3 \times 1 \times 4 \times 1 \times 5 = 60. If every term is positive, taking the log turns the product into a sum: log⁡∏iai=∑ilog⁡ai\log \prod_i a_i = \sum_i \log a_i. The Logarithm page covers it.

Further reading

  • Concrete Mathematics, Chapter 2 Sums — Graham, Knuth, Patashnik, 2nd edition 1994. A textbook chapter that sets out sigma notation, manipulating sums (Section 2.3), and double sums and changing their order (Section 2.4). The section 'Five properties of sigma are enough to compute a sum' relies on it. The link is the author's (Knuth's) page introducing the book.
  • numpy.sum — NumPy official documentation. It explains how the axis argument sets which axis to add along, with examples. The question 'What's the difference between axis=0 and axis=1 in np.sum' and the code lab rely on it.
  • Deep Learning, Chapter 2 Linear Algebra — Goodfellow, Bengio, Courville, 2016. Section 2.2 writes the entries of a matrix product with sigma, and defines the dot product as that matrix product. The dot-product row in the section 'Where sigma shows up' relies on it.
  • Deep Learning, Chapter 3 Probability and Information Theory — Goodfellow, Bengio, Courville, 2016. Section 3.8 defines the expectation of a discrete random variable with sigma. The expectation row in the section 'Where sigma shows up' relies on it.

Where this sits on the map