Intuition — stretching a rubber sheet with a grid on it
Draw a grid on a transparent rubber sheet and pin down the point in the middle. Grab the tips of the two arrows that point one square to the right and one square up, and drag them to new places, stretching and slanting the sheet while keeping the grid lines straight and evenly spaced. Once you note down where the two arrows went, you know where every point on the sheet goes.
Here the rubber sheet is the plane, and the pinned point is the origin. The two arrows are the standard basis vectors , . The table that writes where the two went as columns is the matrix, and dragging the sheet to move a point is the matrix–vector product.
The table that records the result of dragging the sheet twice, in one go, is the matrix product (matrix multiplication). Each cell of that table is the dot product of one horizontal line of the left matrix and one vertical line of the right matrix, and it is the protagonist of this page. When you drag twice, the result depends on which drag comes first (Figure 3).
One linear layer of a neural network multiplies an input vector by a weight matrix. Counting how many dot products that product takes is the job of this page.
This page reuses the example vectors from Vectors and Dot Product: the matrix that stacks as rows, and the matrix that stands up as columns.
A matrix is a table of numbers in m rows and n columns
A matrix is a table of numbers laid out in horizontal lines (rows) and vertical lines (columns). It is called an matrix, and the number in row , column is written as the entry .
In this page's numeric examples, refers to one matrix, whose row 1 is and whose row 2 is .
is the entry in row 1, column 2, and is the entry in row 2, column 2. Addition works position by position, as with vectors, so only matrices of the same shape can be added. A scalar multiple multiplies every entry by a single number. For example, .
The right side of Eq. (2) is the transpose (transposed matrix) of . It swaps rows and columns, , so becomes . Row 1 of , , has become column 1 of .
When a vector is multiplied with a matrix, it is viewed as an matrix, that is, as a column vector standing upright. Then the dot product can also be written as , the product of a and an .11 That is why textbooks and papers write the dot product as . The result is a matrix, that is, a single number.
| Symbol | Name | Meaning |
|---|---|---|
| matrix | A table of numbers in rows and columns. Written as bold capital letters. The example is Eq. (2) | |
| entry | The number in row , column | |
| , | shape | rows and columns. The shape in NumPy and PyTorch is (m, n). is the set of all matrices with real entries |
| transpose | The matrix with rows and columns swapped. | |
| , | column vector, component | When multiplied with a matrix, a vector is viewed as an matrix |
| matrix product | Eq. (5). Also called matrix multiplication. The protagonist of this page | |
| index | is the row number and the column number. The in Eq. (3) and the in Eq. (5) are indices that are summed over | |
| number of columns | The number of columns when is | |
| identity matrix (unit matrix) | A square matrix with 1 on the diagonal and 0 elsewhere. | |
| standard basis vectors | and . A bold letter with a subscript is the number of a vector, not a component | |
| example square matrices | has column 1 and column 2 . is the matrix that stretches only the horizontal length by 2 | |
| Hadamard product | The matrix of products of the entries in the same position. Not the matrix product | |
| inverse matrix | The matrix that gives when multiplied. Only some square matrices have one |
A matrix–vector product is one dot product per row
The th component of is the dot product of row of and . Multiplying an matrix by an -dimensional vector takes dot products and gives an -dimensional vector.
Component 1 of is the dot product with row 1, , which is . Component 2 is the dot product with row 2, , which is . So , and a 3-D vector has become a 2-D vector.
The product exists only if the number of columns of , 3, equals the dimension of , because a row and must pair up component by component to form a dot product.
Where does a matrix move the points of the plane — linear transformations
Read by columns, the same product is the vector you get by multiplying the columns of by and adding them. So goes to column 1, and to column 2.
Column 1 of is , and column 2 is . In Figure 1, goes to , goes to , and goes to . Reading by rows gives the same .22 is a symmetric matrix, unchanged when rows and columns are swapped, so its row 1 is also . That the first basis vector goes to column 1 and not to row 1 shows up in (Eq. (7) below, Figure 3), which first multiplies by , the matrix that stretches only the horizontal length by 2. is column 1, which differs from row 1, .
figures/grid_transform.pyThe grid lines stay straight and evenly spaced because Eq. (4) only multiplies components by numbers and adds them. A rule that moves points by multiplying with a matrix like this is called a linear transformation. The matrix that moves nothing is the identity matrix (unit matrix) , with 1 on the diagonal and 0 elsewhere. Whatever you put in, .
A matrix that is 0 everywhere off the diagonal (a diagonal matrix) multiplies the component along each axis by its own diagonal number. The below multiplies the horizontal component by 2 and the vertical component by 1.33 The matrix of second derivatives of the loss on the gradient descent page (the Hessian matrix) is also a diagonal matrix, with 1 and 10 on the diagonal.
How to compute a matrix multiplication — m×p dot products
The entry of the matrix product is the dot product of row of and column of . Multiplying an by an takes dot products and gives an matrix. For and , that is 4 dot products.
The two inner in Eq. (6) must match so that the entries of row and column pair up one to one. The outer and are the shape of the result.
Rows are paired with columns because that is what makes moving by and then by fit into a single product, . For that, column of has to be times column of , and by Eq. (3) its th component is the dot product of row and column .
Put () in place of and the shape is . The entry in row 1, column 2 is the dot product of row 1 of , , and column 2 of , , which is (Figure 2).
figures/matmul_grid.pyThe four entries are , , , . Indeed, column 1, , is : times column 1 of , which is . is the table of the dot products of every pair of rows of . The 4 dot products take 3 multiplications each, 12 multiplications in all; computed as in Eq. (5), it takes multiplications in general.
The matrix product of an matrix and an matrix is a table of the dot products that pair the rows of with the columns of .
When can't two matrices be multiplied?
Two matrices cannot be multiplied when the number of columns of the first differs from the number of rows of the second (the inner of Eq. (6)). For (), is undefined because 3 and 2 differ, while gives a because 3 and 3 match. A row and a column must have the same number of entries to form a dot product.
How do I read a matmul shape error message?
Read a matmul shape error by comparing the last size of the first matrix with the second-to-last size of the second matrix (its first size if it is 2-D; the inner of Eq. (6)). In NumPy, A @ A raises ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 2 is different from 3). It means that the first size of the second matrix (operand 1), 2, differs from the number of columns of the first matrix, 3. In the same case PyTorch shows both shapes, as in mat1 and mat2 shapes cannot be multiplied (2x3 and 2x3), so make the two middle numbers (3 and 2) match.
Why swapping the order changes a matrix product
and pair different rows and columns, so they are usually different matrices. Even their shapes can differ: is , but is .
is the table of dot products between the columns of , so it is . The 6 in row 1, column 3 is the dot product of column 1, , and column 3, , which is .
Order matters even between square matrices of the same shape. Multiplying with , which stretches only the horizontal length by 2, gives the following.
The entry in row 1, column 1 of is the dot product of row 1 of , , and column 1 of , , which is . Since , first stretches horizontally and then moves with . moves with and then stretches horizontally. In Figure 3, goes to under and to under .
figures/order_matters.pyIs matrix multiplication commutative?
No, matrix multiplication is not commutative in general. With and , the entry in row 1, column 2 is 1 for and 2 for (Eq. (7)). The identity matrix can swap places with any square matrix of the same shape, but such pairs are the exception. What you can change instead, when the shapes fit, is the grouping (the associative law): .
Why does the transpose of a product reverse the order, (AB)ᵀ = BᵀAᵀ?
Transposing the product of two matrices reverses the order of multiplication: . The transpose swaps rows and columns, so the dot product of a row of and a column of becomes the dot product of a row of and a column of . The and of Eq. (7) are unchanged by transposing, so the transpose of , , is exactly .
Common misconceptions about matrix multiplication
- Matrix products can be taken in either order (AB = BA) — Eq. (7) is a counterexample. and multiply the same two matrices and differ (Figure 3, and the question on commutativity above).
- A matrix product multiplies the entries in the same position — Multiplying the entries in the same position is the Hadamard product , which in mathematics requires the same shape. works, but the matrix product is undefined because 3 and 2 differ. In NumPy,
A * Ais the former, andA @ Ais aValueError. - The transpose is the inverse — The inverse is the matrix that gives when multiplied, but multiplying by gives , which is not . The one that gives is (row 1, column 1 of is ), and the two are equal only for a square matrix whose columns are orthogonal to each other and each of length 1 (an orthogonal matrix). swaps the 3 and 2 on the diagonal, flips the sign of the other two entries, and divides by (the formula for the inverse of a matrix).
Where matrix multiplication shows up in AI
In the forward pass (Korean for now) of a neural network, the computations of linear layers and of attention are matrix products. A linear layer (fully connected layer) multiplies the input by a weight matrix and adds a bias . Each row of is the weight vector that makes one output, so each output is the dot product of that row and .
| Symbol | Name | Meaning |
|---|---|---|
| weight matrix, bias | The linear layer (Eq. (8)). Texts often write the bias as , but on this site is an example vector, so we write | |
| , | batch | Matrices that stack the inputs and outputs of samples as rows. The site's convention for the mini-batch size, , is easily confused with the bold matrix , so this page writes |
| 1-vector (ones vector) | The column vector whose components are all 1. is the matrix whose every row is | |
| , | query matrix, key matrix, key dimension | Attention. Their rows are the query vectors and the key vectors (the subscript of a bold letter is a vector number). is the dimension of the key vectors (a conventional notation) |
With and , . The first component is 0 because is orthogonal to (Try it yourself 2 on Vectors and Dot Product). Adding gives .
To feed samples at once, use a batch () that stacks the samples as rows. Then, as on the right of Eq. (8), is transposed and multiplied on the right, and row of is the of the th sample. That is because laying a sample down as a row gives (transposing a product reverses the order).
is the matrix whose every row is , so adds to every row. In code, simply adding c of shape (2,) adds it to each row (broadcasting).
If the two rows of are and , then , and , with added, is . Row 2, , is the same as .
If this is the last linear layer of a classification model, the output is the logits that go into the softmax (that page writes the logits as and the correct answer as ).
| This page | NumPy · PyTorch |
|---|---|
A.shape — (2, 3) for the example | |
| (2-D) | A.T in both NumPy and PyTorch. NumPy's A.transpose(0, 1) on a 2-D array means 'keep the axis order as it is', so it is not a transpose |
| Transpose of a batch (3-D or more) | Swap only the last two axes. A.mT (NumPy 2.0 or later, and PyTorch), NumPy A.swapaxes(-2, -1), PyTorch A.transpose(-2, -1). NumPy's .T reverses all axes, so (5, 6, 3) becomes (3, 6, 5). PyTorch's .T is also deprecated on tensors that are not 2-D |
| (Eq. (5)) | A @ B, np.matmul(A, B), torch.matmul(A, B) |
| (Eq. (3)) | A @ x — if x is 1-D, (3,), the result is also 1-D, (2,) |
A * B — not the matrix product | |
np.eye(2), torch.eye(2) | |
The weight of torch.nn.Linear(in_features=3, out_features=2). Its shape is (2, 3), that is, (output, input) | |
The bias of the same layer. Shape (2,) | |
| Right side of Eq. (8) | layer(X) — if X is (N, 3), the output is (N, 2). The docs write the same formula as (the docs' is this page's , and is ) |
Q @ K.mT — in PyTorch also written Q @ K.transpose(-2, -1) |
Why is the nn.Linear weight shape (out, in)?
Because the weight of nn.Linear stores the weight matrix of the linear layer as it is (Eq. (8)). In , each output is the dot product of one row of and the input, so the number of rows is the number of outputs and the number of columns is the number of inputs. The weight of nn.Linear(3, 2) is (2, 3), and a batch X () is multiplied by its transpose weight.T (), so X @ weight.T is . Keras's Dense does the opposite: it keeps kernel as (input, output) and computes x @ kernel, so you transpose when moving weights between the two frameworks.
Why attention's QKᵀ transposes K
Because and both stack their query and key vectors as rows, has to be transposed to stand the keys up as columns, so that Eq. (5) pairs each query with each key in a dot product. So a single gives the table of scores for every query and key, with as the entry.
If , the scores are . In this example the untransposed is and cannot even be multiplied.44 If the number of queries equals , can also be multiplied, and it gives wrong scores without any error. With and , , but . Dividing these scores by and putting them through the softmax is covered on the Softmax page.
Check it in code
Eq. (5) is turned directly into loops and checked against the result of @. Click a symbol to highlight the same index in both the formula and the code. The inner index k runs over the columns of and the rows of at the same time.
import numpy as np
A = np.array([[1.0, 2.0, 2.0], # 1행 = u
[2.0, 1.0, 2.0]]) # 2행 = w
B = A.T # 식 (2) 의 전치: 3×2
m, n = A.shape
p = B.shape[1]
prod = np.zeros((m, p)) # 식 (6): (m×n)(n×p) = m×p
for i in range(m):
for j in range(p):
prod[i, j] = sum(A[i, k] * B[k, j] for k in range(n)) # 식 (5)
print("A", A.shape, "@ B", B.shape, "->", prod.shape)
print("반복문으로:\n", prod)
print("A @ B 로:\n", A @ B)
print("B @ A 의 shape:", (B @ A).shape)
print("A @ u =", A @ np.array([1.0, 2.0, 2.0])) # 식 (3)$ python matmul.py
A (2, 3) @ B (3, 2) -> (2, 2)
반복문으로:
[[9. 8.]
[8. 9.]]
A @ B 로:
[[9. 8.]
[8. 9.]]
B @ A 의 shape: (3, 3)
A @ u = [9. 8.]Try it yourself
- Find the entry in row 3, column 3 of with Eq. (5). How is this value related to the length of which column of ?
- Where does go when multiplied by ? Express it in terms of and .
- If you feed a batch
Xof shape intonn.Linear(3, 2), what is the shape of the output? Why doesX @ layer.weightgive an error?
Answers
- It is the dot product of column 3 of , , with itself, so . That is the square of the length of column 3, . The diagonal of holds the squared lengths of the columns.
- By Eq. (4), , that is, . Reading by rows also gives .
- The of Eq. (8) is , so the output is
(4, 2).layer.weightis(2, 3), soX @ layer.weightis , and since the inner 3 and 2 differ, it cannot be multiplied.
Further reading
- Deep Learning, Chapter 2 Linear Algebra — Goodfellow, Bengio, Courville, 2016. Section 2.2 defines the matrix product as in Eq. (5), and notes separately that it is not commutative and that there is the Hadamard product A ⊙ B. Section 2.3 covers the identity matrix and the inverse matrix. The section 'How to compute a matrix multiplication — m×p dot products' and the section 'Common misconceptions about matrix multiplication' rely on it.
- MIT 18.06 Linear Algebra, Spring 2010 — Gilbert Strang, MIT OpenCourseWare, 2010. Lecture 3 (Multiplication and inverse matrices) shows how to read a matrix product by rows, by columns, and as combinations of columns. It pairs with Section 2.4 (Rules for Matrix Operations) of the textbook Introduction to Linear Algebra. The sections 'A matrix–vector product is one dot product per row' and 'Where does a matrix move the points of the plane — linear transformations' rely on it.
- numpy.matmul — NumPy official documentation. It gives the signature
(n,k),(k,m)->(n,m), and states that it raisesValueErrorwhen the last size of the first array differs from the second-to-last size of the second array. The 'matmul shape error' question relies on it. - torch.nn.Linear — PyTorch official documentation. It gives the layer's formula , the shape of
weight, (out_features, in_features), and the shape ofbias, (out_features). The mapping table and the question in the section 'Where matrix multiplication shows up in AI' rely on it. - Attention Is All You Need — Vaswani et al., 2017. Eq. (1) in Section 3.2.1 computes the attention scores as the product of the query matrix and the key matrix, , divided by . The section 'Why attention's QKᵀ transposes K' relies on it.
Video notes
- 00:00 Opening question — "How many dot products is a matrix product?"
- 00:40 Eqs. (1) and (2) — the matrix as a table, the entry , the transpose
- 02:00 Eq. (3) — , one dot product per row
- 03:20 Figure 1 and Eq. (4) — the columns are where the standard basis vectors go, the rubber sheet with a grid
- 05:00 Figure 2 and Eqs. (5) and (6) — the 8 in row 1, column 2, why the inner sizes must match
- 06:40 Figure 3 and Eq. (7) — , the one applied first is on the right
- 08:00 Eq. (8) — why the
nn.Linearweight is (2, 3), · wrap-up
This is the filming plan. After upload, the timestamps will be replaced with the real ones.