Linear AlgebraMatrix Operations#005

Matrix Operations

행렬 연산

Entry (i, j) of the matrix product AB is the dot product of row i of A and column j of B. A 2×3 times a 3×2 matrix takes 4 dot products and gives a 2×2 matrix; in the other order it is 3×3. Covers when a product is undefined, the transpose, and the linear layer Wx + c.

Key symbol
AB
Reading time
16 min
Prerequisites
Vectors and Dot Product

Translated by an agent and not reviewed yet. Original: 한국어

Intuition — stretching a rubber sheet with a grid on it

Intuition

Draw a grid on a transparent rubber sheet and pin down the point in the middle. Grab the tips of the two arrows that point one square to the right and one square up, and drag them to new places, stretching and slanting the sheet while keeping the grid lines straight and evenly spaced. Once you note down where the two arrows went, you know where every point on the sheet goes.

Here the rubber sheet is the plane, and the pinned point is the origin. The two arrows are the standard basis vectors e1=(1,0)\mathbf{e}_1 = (1, 0), e2=(0,1)\mathbf{e}_2 = (0, 1). The table that writes where the two went as columns is the matrix, and dragging the sheet to move a point is the matrix–vector product.

The table that records the result of dragging the sheet twice, in one go, is the matrix product (matrix multiplication). Each cell of that table is the dot product of one horizontal line of the left matrix and one vertical line of the right matrix, and it is the protagonist of this page. When you drag twice, the result depends on which drag comes first (Figure 3).

One linear layer of a neural network multiplies an input vector by a weight matrix. Counting how many dot products that product takes is the job of this page.

This page reuses the example vectors from Vectors and Dot Product: the 2×32 \times 3 matrix A\mathbf{A} that stacks u,w\mathbf{u}, \mathbf{w} as rows, and the 2×22 \times 2 matrix M\mathbf{M} that stands a,b\mathbf{a}, \mathbf{b} up as columns.

A matrix is a table of numbers in m rows and n columns

A matrix is a table of numbers laid out in mm horizontal lines (rows) and nn vertical lines (columns). It is called an m×nm \times n matrix, and the number in row ii, column jj is written as the entry AijA_{ij}.

A=(A11A12⋯A1n⋮⋮⋮Am1Am2⋯Amn)∈Rm×n(1)\mathbf{A} = \begin{pmatrix} A_{11} & A_{12} & \cdots & A_{1n} \\ \vdots & \vdots & & \vdots \\ A_{m1} & A_{m2} & \cdots & A_{mn} \end{pmatrix} \in \mathbb{R}^{m \times n} \tag{1}

In this page's numeric examples, A\mathbf{A} refers to one 2×32 \times 3 matrix, whose row 1 is u=(1,2,2)\mathbf{u} = (1, 2, 2) and whose row 2 is w=(2,1,2)\mathbf{w} = (2, 1, 2).

A=(122212),A⊤=(122122)(2)\mathbf{A} = \begin{pmatrix} 1 & 2 & 2 \\ 2 & 1 & 2 \end{pmatrix}, \qquad \mathbf{A}^\top = \begin{pmatrix} 1 & 2 \\ 2 & 1 \\ 2 & 2 \end{pmatrix} \tag{2}

A12=2A_{12} = 2 is the entry in row 1, column 2, and A22=1A_{22} = 1 is the entry in row 2, column 2. Addition works position by position, as with vectors, so only matrices of the same shape can be added. A scalar multiple multiplies every entry by a single number. For example, A+A=2A=(244424)\mathbf{A} + \mathbf{A} = 2\mathbf{A} = \left(\begin{smallmatrix} 2 & 4 & 4 \\ 4 & 2 & 4 \end{smallmatrix}\right).

The right side of Eq. (2) is the transpose (transposed matrix) A⊤\mathbf{A}^\top of A\mathbf{A}. It swaps rows and columns, (A⊤)ij=Aji(\mathbf{A}^\top)_{ij} = A_{ji}, so 2×32 \times 3 becomes 3×23 \times 2. Row 1 of A\mathbf{A}, (1,2,2)(1, 2, 2), has become column 1 of A⊤\mathbf{A}^\top.

When a vector is multiplied with a matrix, it is viewed as an n×1n \times 1 matrix, that is, as a column vector standing upright. Then the dot product x⋅y\mathbf{x} \cdot \mathbf{y} can also be written as x⊤y\mathbf{x}^\top \mathbf{y}, the product of a 1×n1 \times n and an n×1n \times 1.11 That is why textbooks and papers write the dot product as x⊤y\mathbf{x}^\top \mathbf{y}. The result is a 1×11 \times 1 matrix, that is, a single number.

Definition
SymbolNameMeaning
A,B,C\mathbf{A}, \mathbf{B}, \mathbf{C}matrixA table of numbers in mm rows and nn columns. Written as bold capital letters. The example A\mathbf{A} is Eq. (2)
AijA_{ij}entryThe number in row ii, column jj
m×nm \times n, Rm×n\mathbb{R}^{m \times n}shapemm rows and nn columns. The shape in NumPy and PyTorch is (m, n). Rm×n\mathbb{R}^{m \times n} is the set of all m×nm \times n matrices with real entries
A⊤\mathbf{A}^\toptransposeThe n×mn \times m matrix with rows and columns swapped. (A⊤)ij=Aji(\mathbf{A}^\top)_{ij} = A_{ji}
x\mathbf{x}, xjx_jcolumn vector, componentWhen multiplied with a matrix, a vector is viewed as an n×1n \times 1 matrix
AB\mathbf{A}\mathbf{B}matrix productEq. (5). Also called matrix multiplication. The protagonist of this page
i,j,ki, j, kindexii is the row number and jj the column number. The jj in Eq. (3) and the kk in Eq. (5) are indices that are summed over
ppnumber of columnsThe number of columns when B\mathbf{B} is n×pn \times p
I\mathbf{I}identity matrix (unit matrix)A square matrix with 1 on the diagonal and 0 elsewhere. Ix=x\mathbf{I}\mathbf{x} = \mathbf{x}
e1,e2\mathbf{e}_1, \mathbf{e}_2standard basis vectors(1,0)(1, 0) and (0,1)(0, 1). A bold letter with a subscript is the number of a vector, not a component
M,S\mathbf{M}, \mathbf{S}example square matricesM\mathbf{M} has column 1 a=(3,1)\mathbf{a} = (3, 1) and column 2 b=(1,2)\mathbf{b} = (1, 2). S\mathbf{S} is the matrix that stretches only the horizontal length by 2
A⊙B\mathbf{A} \odot \mathbf{B}Hadamard productThe matrix of products of the entries in the same position. Not the matrix product
M−1\mathbf{M}^{-1}inverse matrixThe matrix that gives I\mathbf{I} when multiplied. Only some square matrices have one

A matrix–vector product is one dot product per row

The iith component of Ax\mathbf{A}\mathbf{x} is the dot product of row ii of A\mathbf{A} and x\mathbf{x}. Multiplying an m×nm \times n matrix by an nn-dimensional vector takes mm dot products and gives an mm-dimensional vector.

(Ax)i=∑j=1nAij xj(3)(\mathbf{A}\mathbf{x})_i = \sum_{j=1}^{n} A_{ij}\, x_j \tag{3}

Component 1 of Au\mathbf{A}\mathbf{u} is the dot product with row 1, u\mathbf{u}, which is u⋅u=9\mathbf{u} \cdot \mathbf{u} = 9. Component 2 is the dot product with row 2, w\mathbf{w}, which is w⋅u=8\mathbf{w} \cdot \mathbf{u} = 8. So Au=(9,8)\mathbf{A}\mathbf{u} = (9, 8), and a 3-D vector has become a 2-D vector.

The product exists only if the number of columns of A\mathbf{A}, 3, equals the dimension of x\mathbf{x}, because a row and x\mathbf{x} must pair up component by component to form a dot product.

Where does a matrix move the points of the plane — linear transformations

Read by columns, the same product Ax\mathbf{A}\mathbf{x} is the vector you get by multiplying the columns of A\mathbf{A} by x1,x2,…x_1, x_2, \dots and adding them. So e1\mathbf{e}_1 goes to column 1, and e2\mathbf{e}_2 to column 2.

Mx=x1(31)+x2(12),M=(3112)(4)\mathbf{M}\mathbf{x} = x_1 \begin{pmatrix} 3 \\ 1 \end{pmatrix} + x_2 \begin{pmatrix} 1 \\ 2 \end{pmatrix}, \qquad \mathbf{M} = \begin{pmatrix} 3 & 1 \\ 1 & 2 \end{pmatrix} \tag{4}

Column 1 of M\mathbf{M} is a=(3,1)\mathbf{a} = (3, 1), and column 2 is b=(1,2)\mathbf{b} = (1, 2). In Figure 1, e1\mathbf{e}_1 goes to a\mathbf{a}, e2\mathbf{e}_2 goes to b\mathbf{b}, and (1,1)(1, 1) goes to a+b=(4,3)\mathbf{a} + \mathbf{b} = (4, 3). Reading by rows gives the same (3+1, 1+2)=(4,3)(3 + 1,\ 1 + 2) = (4, 3).22 M\mathbf{M} is a symmetric matrix, unchanged when rows and columns are swapped, so its row 1 is also (3,1)(3, 1). That the first basis vector goes to column 1 and not to row 1 shows up in MS\mathbf{M}\mathbf{S} (Eq. (7) below, Figure 3), which first multiplies by S\mathbf{S}, the matrix that stretches only the horizontal length by 2. MSe1=(6,2)\mathbf{M}\mathbf{S}\mathbf{e}_1 = (6, 2) is column 1, which differs from row 1, (6,1)(6, 1).

How the matrix M moves the plane. Left: the grid and the unit square before the move; right: after moving them with M. e_1 = (1, 0) goes to column 1, a = (3, 1); e_2 = (0, 1) goes to column 2, b = (1, 2); and (1, 1) goes to (4, 3). The square becomes the parallelogram spanned by a and b.
Figure 1 How the matrix M\mathbf{M} moves the plane. Left: the grid and the unit square before the move; right: after moving them with M\mathbf{M}. e1=(1, 0)\mathbf{e}_1 = (1,\ 0) goes to column 1, a=(3, 1)\mathbf{a} = (3,\ 1); e2=(0, 1)\mathbf{e}_2 = (0,\ 1) goes to column 2, b=(1, 2)\mathbf{b} = (1,\ 2); and (1, 1)(1,\ 1) goes to (4, 3)(4,\ 3). The square becomes the parallelogram spanned by a\mathbf{a} and b\mathbf{b}. figures/grid_transform.py

The grid lines stay straight and evenly spaced because Eq. (4) only multiplies components by numbers and adds them. A rule that moves points by multiplying with a matrix like this is called a linear transformation. The matrix that moves nothing is the identity matrix (unit matrix) I\mathbf{I}, with 1 on the diagonal and 0 elsewhere. Whatever x\mathbf{x} you put in, Ix=x\mathbf{I}\mathbf{x} = \mathbf{x}.

A matrix that is 0 everywhere off the diagonal (a diagonal matrix) multiplies the component along each axis by its own diagonal number. The S\mathbf{S} below multiplies the horizontal component by 2 and the vertical component by 1.33 The matrix of second derivatives of the loss 12(x2+10y2)\tfrac{1}{2}(x^2 + 10y^2) on the gradient descent page (the Hessian matrix) is also a diagonal matrix, with 1 and 10 on the diagonal.

How to compute a matrix multiplication — m×p dot products

The (i,j)(i, j) entry of the matrix product AB\mathbf{A}\mathbf{B} is the dot product of row ii of A\mathbf{A} and column jj of B\mathbf{B}. Multiplying an m×nm \times n by an n×pn \times p takes m×pm \times p dot products and gives an m×pm \times p matrix. For 2×32 \times 3 and 3×23 \times 2, that is 4 dot products.

(AB)ij=∑k=1nAik Bkj(5)(\mathbf{A}\mathbf{B})_{ij} = \sum_{k=1}^{n} A_{ik}\, B_{kj} \tag{5} (m×n) (n×p)=m×p(6)(m \times n)\,(n \times p) = m \times p \tag{6}

The two inner nn in Eq. (6) must match so that the entries of row ii and column jj pair up one to one. The outer mm and pp are the shape of the result.

Rows are paired with columns because that is what makes moving by B\mathbf{B} and then by A\mathbf{A} fit into a single product, (AB)x=A(Bx)(\mathbf{A}\mathbf{B})\mathbf{x} = \mathbf{A}(\mathbf{B}\mathbf{x}). For that, column jj of AB\mathbf{A}\mathbf{B} has to be A\mathbf{A} times column jj of B\mathbf{B}, and by Eq. (3) its iith component is the dot product of row ii and column jj.

Put A⊤\mathbf{A}^\top (3×23 \times 2) in place of B\mathbf{B} and the shape is (2×3)(3×2)=2×2(2 \times 3)(3 \times 2) = 2 \times 2. The entry in row 1, column 2 is the dot product of row 1 of A\mathbf{A}, (1,2,2)(1, 2, 2), and column 2 of A⊤\mathbf{A}^\top, (2,1,2)(2, 1, 2), which is 2+2+4=82 + 2 + 4 = 8 (Figure 2).

Computing AA^ᵀ with Eq. (5). Pairing row 1 of A (2 × 3) with column 2 of A^ᵀ (3 × 2) and taking the dot product gives 8, the entry in row 1, column 2 of the result (2 × 2). The other three entries are also the dot product of one row and one column.
Figure 2 Computing AA⊤\mathbf{A}\mathbf{A}^\top with Eq. (5). Pairing row 1 of A\mathbf{A} (2×32 \times 3) with column 2 of A⊤\mathbf{A}^\top (3×23 \times 2) and taking the dot product gives 8, the entry in row 1, column 2 of the result (2×22 \times 2). The other three entries are also the dot product of one row and one column. figures/matmul_grid.py

The four entries are u⋅u=9\mathbf{u} \cdot \mathbf{u} = 9, u⋅w=8\mathbf{u} \cdot \mathbf{w} = 8, w⋅u=8\mathbf{w} \cdot \mathbf{u} = 8, w⋅w=9\mathbf{w} \cdot \mathbf{w} = 9. Indeed, column 1, (9,8)(9, 8), is Au\mathbf{A}\mathbf{u}: A\mathbf{A} times column 1 of A⊤\mathbf{A}^\top, which is u\mathbf{u}. AA⊤\mathbf{A}\mathbf{A}^\top is the table of the dot products of every pair of rows of A\mathbf{A}. The 4 dot products take 3 multiplications each, 12 multiplications in all; computed as in Eq. (5), it takes m×n×pm \times n \times p multiplications in general.

Key idea

The matrix product AB\mathbf{A}\mathbf{B} of an m×nm \times n matrix A\mathbf{A} and an n×pn \times p matrix B\mathbf{B} is a table of the m×pm \times p dot products that pair the rows of A\mathbf{A} with the columns of B\mathbf{B}.

When can't two matrices be multiplied?

Two matrices cannot be multiplied when the number of columns of the first differs from the number of rows of the second (the inner nn of Eq. (6)). For A\mathbf{A} (2×32 \times 3), AA\mathbf{A}\mathbf{A} is undefined because 3 and 2 differ, while AA⊤\mathbf{A}\mathbf{A}^\top gives a 2×22 \times 2 because 3 and 3 match. A row and a column must have the same number of entries to form a dot product.

How do I read a matmul shape error message?

Read a matmul shape error by comparing the last size of the first matrix with the second-to-last size of the second matrix (its first size if it is 2-D; the inner nn of Eq. (6)). In NumPy, A @ A raises ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 2 is different from 3). It means that the first size of the second matrix (operand 1), 2, differs from the number of columns of the first matrix, 3. In the same case PyTorch shows both shapes, as in mat1 and mat2 shapes cannot be multiplied (2x3 and 2x3), so make the two middle numbers (3 and 2) match.

Why swapping the order changes a matrix product

AB\mathbf{A}\mathbf{B} and BA\mathbf{B}\mathbf{A} pair different rows and columns, so they are usually different matrices. Even their shapes can differ: AA⊤\mathbf{A}\mathbf{A}^\top is 2×22 \times 2, but A⊤A\mathbf{A}^\top\mathbf{A} is 3×33 \times 3.

A⊤A\mathbf{A}^\top\mathbf{A} is the table of dot products between the columns of A\mathbf{A}, so it is (546456668)\left(\begin{smallmatrix} 5 & 4 & 6 \\ 4 & 5 & 6 \\ 6 & 6 & 8 \end{smallmatrix}\right). The 6 in row 1, column 3 is the dot product of column 1, (1,2)(1, 2), and column 3, (2,2)(2, 2), which is 2+42 + 4.

Order matters even between square matrices of the same shape. Multiplying M\mathbf{M} with S=(2001)\mathbf{S} = \left(\begin{smallmatrix} 2 & 0 \\ 0 & 1 \end{smallmatrix}\right), which stretches only the horizontal length by 2, gives the following.

MS=(6122)≠SM=(6212)(7)\mathbf{M}\mathbf{S} = \begin{pmatrix} 6 & 1 \\ 2 & 2 \end{pmatrix} \ne \mathbf{S}\mathbf{M} = \begin{pmatrix} 6 & 2 \\ 1 & 2 \end{pmatrix} \tag{7}

The entry in row 1, column 1 of MS\mathbf{M}\mathbf{S} is the dot product of row 1 of M\mathbf{M}, (3,1)(3, 1), and column 1 of S\mathbf{S}, (2,0)(2, 0), which is 6+0=66 + 0 = 6. Since MSx=M(Sx)\mathbf{M}\mathbf{S}\mathbf{x} = \mathbf{M}(\mathbf{S}\mathbf{x}), MS\mathbf{M}\mathbf{S} first stretches horizontally and then moves with M\mathbf{M}. SM\mathbf{S}\mathbf{M} moves with M\mathbf{M} and then stretches horizontally. In Figure 3, e1\mathbf{e}_1 goes to (6,2)(6, 2) under MS\mathbf{M}\mathbf{S} and to (6,1)(6, 1) under SM\mathbf{S}\mathbf{M}.

The same unit square (dashed) moved by MS (left) and by SM (right). e_1 goes to (6, 2) and (6, 1), and e_2 to (1, 2) and (2, 2), giving two different parallelograms.
Figure 3 The same unit square (dashed) moved by MS\mathbf{M}\mathbf{S} (left) and by SM\mathbf{S}\mathbf{M} (right). e1\mathbf{e}_1 goes to (6, 2)(6,\ 2) and (6, 1)(6,\ 1), and e2\mathbf{e}_2 to (1, 2)(1,\ 2) and (2, 2)(2,\ 2), giving two different parallelograms. figures/order_matters.py

Is matrix multiplication commutative?

No, matrix multiplication is not commutative in general. With M=(3112)\mathbf{M} = \left(\begin{smallmatrix} 3 & 1 \\ 1 & 2 \end{smallmatrix}\right) and S=(2001)\mathbf{S} = \left(\begin{smallmatrix} 2 & 0 \\ 0 & 1 \end{smallmatrix}\right), the entry in row 1, column 2 is 1 for MS\mathbf{M}\mathbf{S} and 2 for SM\mathbf{S}\mathbf{M} (Eq. (7)). The identity matrix can swap places with any square matrix of the same shape, but such pairs are the exception. What you can change instead, when the shapes fit, is the grouping (the associative law): (AB)C=A(BC)(\mathbf{A}\mathbf{B})\mathbf{C} = \mathbf{A}(\mathbf{B}\mathbf{C}).

Why does the transpose of a product reverse the order, (AB)ᵀ = BᵀAᵀ?

Transposing the product of two matrices reverses the order of multiplication: (AB)⊤=B⊤A⊤(\mathbf{A}\mathbf{B})^\top = \mathbf{B}^\top\mathbf{A}^\top. The transpose swaps rows and columns, so the dot product of a row of A\mathbf{A} and a column of B\mathbf{B} becomes the dot product of a row of B⊤\mathbf{B}^\top and a column of A⊤\mathbf{A}^\top. The M\mathbf{M} and S\mathbf{S} of Eq. (7) are unchanged by transposing, so the transpose of MS\mathbf{M}\mathbf{S}, (6212)\left(\begin{smallmatrix} 6 & 2 \\ 1 & 2 \end{smallmatrix}\right), is exactly SM\mathbf{S}\mathbf{M}.

Common misconceptions about matrix multiplication

  • Matrix products can be taken in either order (AB = BA) — Eq. (7) is a counterexample. MS\mathbf{M}\mathbf{S} and SM\mathbf{S}\mathbf{M} multiply the same two matrices and differ (Figure 3, and the question on commutativity above).
  • A matrix product multiplies the entries in the same position — Multiplying the entries in the same position is the Hadamard product A⊙B\mathbf{A} \odot \mathbf{B}, which in mathematics requires the same shape. A⊙A=(144414)\mathbf{A} \odot \mathbf{A} = \left(\begin{smallmatrix} 1 & 4 & 4 \\ 4 & 1 & 4 \end{smallmatrix}\right) works, but the matrix product AA\mathbf{A}\mathbf{A} is undefined because 3 and 2 differ. In NumPy, A * A is the former, and A @ A is a ValueError.
  • The transpose is the inverse — The inverse is the matrix that gives I\mathbf{I} when multiplied, but multiplying by M⊤=M\mathbf{M}^\top = \mathbf{M} gives MM⊤=(10555)\mathbf{M}\mathbf{M}^\top = \left(\begin{smallmatrix} 10 & 5 \\ 5 & 5 \end{smallmatrix}\right), which is not I\mathbf{I}. The one that gives I\mathbf{I} is M−1=15(2−1−13)\mathbf{M}^{-1} = \tfrac{1}{5}\left(\begin{smallmatrix} 2 & -1 \\ -1 & 3 \end{smallmatrix}\right) (row 1, column 1 of MM−1\mathbf{M}\mathbf{M}^{-1} is (3×2−1×1)/5=1(3 \times 2 - 1 \times 1)/5 = 1), and the two are equal only for a square matrix whose columns are orthogonal to each other and each of length 1 (an orthogonal matrix). M−1\mathbf{M}^{-1} swaps the 3 and 2 on the diagonal, flips the sign of the other two entries, and divides by 3×2−1×1=53 \times 2 - 1 \times 1 = 5 (the formula for the inverse of a 2×22 \times 2 matrix).

Where matrix multiplication shows up in AI

In the forward pass (Korean for now) of a neural network, the computations of linear layers and of attention are matrix products. A linear layer (fully connected layer) multiplies the input x\mathbf{x} by a weight matrix W\mathbf{W} and adds a bias c\mathbf{c}. Each row of W\mathbf{W} is the weight vector that makes one output, so each output is the dot product of that row and x\mathbf{x}.

Definition
SymbolNameMeaning
W,c\mathbf{W}, \mathbf{c}weight matrix, biasThe linear layer (Eq. (8)). Texts often write the bias as b\mathbf{b}, but on this site b\mathbf{b} is an example vector, so we write c\mathbf{c}
X,Y\mathbf{X}, \mathbf{Y}, NNbatchMatrices that stack the inputs and outputs of NN samples as rows. The site's convention for the mini-batch size, BB, is easily confused with the bold matrix B\mathbf{B}, so this page writes NN
1\mathbf{1}1-vector (ones vector)The N×1N \times 1 column vector whose components are all 1. 1c⊤\mathbf{1}\mathbf{c}^\top is the matrix whose every row is c⊤\mathbf{c}^\top
Q,K\mathbf{Q}, \mathbf{K}, dkd_kquery matrix, key matrix, key dimensionAttention. Their rows are the query vectors qi\mathbf{q}_i and the key vectors kj\mathbf{k}_j (the subscript of a bold letter is a vector number). dkd_k is the dimension of the key vectors (a conventional notation)
y=Wx+c,Y=XW⊤+1c⊤(8)\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{c}, \qquad \mathbf{Y} = \mathbf{X}\mathbf{W}^\top + \mathbf{1}\mathbf{c}^\top \tag{8}

With W=A\mathbf{W} = \mathbf{A} and x=(2,−2,1)\mathbf{x} = (2, -2, 1), Wx=(0,4)\mathbf{W}\mathbf{x} = (0, 4). The first component is 0 because (2,−2,1)(2, -2, 1) is orthogonal to u\mathbf{u} (Try it yourself 2 on Vectors and Dot Product). Adding c=(1,−1)\mathbf{c} = (1, -1) gives y=(1,3)\mathbf{y} = (1, 3).

To feed NN samples at once, use a batch X\mathbf{X} (N×3N \times 3) that stacks the samples as rows. Then, as on the right of Eq. (8), W\mathbf{W} is transposed and multiplied on the right, and row ii of XW⊤\mathbf{X}\mathbf{W}^\top is the Wx\mathbf{W}\mathbf{x} of the iith sample. That is because laying a sample down as a row gives (Wx)⊤=x⊤W⊤(\mathbf{W}\mathbf{x})^\top = \mathbf{x}^\top\mathbf{W}^\top (transposing a product reverses the order).

1c⊤\mathbf{1}\mathbf{c}^\top is the matrix whose every row is c⊤\mathbf{c}^\top, so + 1c⊤+\,\mathbf{1}\mathbf{c}^\top adds c⊤\mathbf{c}^\top to every row. In code, simply adding c of shape (2,) adds it to each row (broadcasting).

If the two rows of X\mathbf{X} are (2,−2,1)(2, -2, 1) and u=(1,2,2)\mathbf{u} = (1, 2, 2), then XW⊤=(0498)\mathbf{X}\mathbf{W}^\top = \left(\begin{smallmatrix} 0 & 4 \\ 9 & 8 \end{smallmatrix}\right), and Y\mathbf{Y}, with c\mathbf{c} added, is (13107)\left(\begin{smallmatrix} 1 & 3 \\ 10 & 7 \end{smallmatrix}\right). Row 2, (9,8)(9, 8), is the same as Au\mathbf{A}\mathbf{u}.

If this is the last linear layer of a classification model, the output y\mathbf{y} is the logits that go into the softmax (that page writes the logits as z\mathbf{z} and the correct answer as y\mathbf{y}).

This pageNumPy · PyTorch
m×nm \times nA.shape — (2, 3) for the example A\mathbf{A}
A⊤\mathbf{A}^\top (2-D)A.T in both NumPy and PyTorch. NumPy's A.transpose(0, 1) on a 2-D array means 'keep the axis order as it is', so it is not a transpose
Transpose of a batch (3-D or more)Swap only the last two axes. A.mT (NumPy 2.0 or later, and PyTorch), NumPy A.swapaxes(-2, -1), PyTorch A.transpose(-2, -1). NumPy's .T reverses all axes, so (5, 6, 3) becomes (3, 6, 5). PyTorch's .T is also deprecated on tensors that are not 2-D
AB\mathbf{A}\mathbf{B} (Eq. (5))A @ B, np.matmul(A, B), torch.matmul(A, B)
Ax\mathbf{A}\mathbf{x} (Eq. (3))A @ x — if x is 1-D, (3,), the result is also 1-D, (2,)
A⊙B\mathbf{A} \odot \mathbf{B}A * B — not the matrix product
I\mathbf{I}np.eye(2), torch.eye(2)
W\mathbf{W}The weight of torch.nn.Linear(in_features=3, out_features=2). Its shape is (2, 3), that is, (output, input)
c\mathbf{c}The bias of the same layer. Shape (2,)
Right side of Eq. (8)layer(X) — if X is (N, 3), the output is (N, 2). The docs write the same formula as y=xA⊤+by = xA^\top + b (the docs' AA is this page's W\mathbf{W}, and bb is c\mathbf{c})
QK⊤\mathbf{Q}\mathbf{K}^\topQ @ K.mT — in PyTorch also written Q @ K.transpose(-2, -1)

Why is the nn.Linear weight shape (out, in)?

Because the weight of nn.Linear stores the weight matrix W\mathbf{W} of the linear layer y=Wx+c\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{c} as it is (Eq. (8)). In Wx\mathbf{W}\mathbf{x}, each output is the dot product of one row of W\mathbf{W} and the input, so the number of rows is the number of outputs and the number of columns is the number of inputs. The weight of nn.Linear(3, 2) is (2, 3), and a batch X (N×3N \times 3) is multiplied by its transpose weight.T (3×23 \times 2), so X @ weight.T is N×2N \times 2. Keras's Dense does the opposite: it keeps kernel as (input, output) and computes x @ kernel, so you transpose when moving weights between the two frameworks.

Why attention's QKᵀ transposes K

Because Q\mathbf{Q} and K\mathbf{K} both stack their query and key vectors as rows, K\mathbf{K} has to be transposed to stand the keys up as columns, so that Eq. (5) pairs each query qi\mathbf{q}_i with each key kj\mathbf{k}_j in a dot product. So a single QK⊤\mathbf{Q}\mathbf{K}^\top gives the table of scores for every query and key, with qi⋅kj\mathbf{q}_i \cdot \mathbf{k}_j as the (i,j)(i, j) entry.

If Q=K=A\mathbf{Q} = \mathbf{K} = \mathbf{A}, the scores are AA⊤=(9889)\mathbf{A}\mathbf{A}^\top = \left(\begin{smallmatrix} 9 & 8 \\ 8 & 9 \end{smallmatrix}\right). In this example the untransposed QK\mathbf{Q}\mathbf{K} is (2×3)(2×3)(2 \times 3)(2 \times 3) and cannot even be multiplied.44 If the number of queries equals dkd_k, QK\mathbf{Q}\mathbf{K} can also be multiplied, and it gives wrong scores without any error. With Q=(1011)\mathbf{Q} = \left(\begin{smallmatrix} 1 & 0 \\ 1 & 1 \end{smallmatrix}\right) and K=(1234)\mathbf{K} = \left(\begin{smallmatrix} 1 & 2 \\ 3 & 4 \end{smallmatrix}\right), QK⊤=(1337)\mathbf{Q}\mathbf{K}^\top = \left(\begin{smallmatrix} 1 & 3 \\ 3 & 7 \end{smallmatrix}\right), but QK=(1246)\mathbf{Q}\mathbf{K} = \left(\begin{smallmatrix} 1 & 2 \\ 4 & 6 \end{smallmatrix}\right). Dividing these scores by dk\sqrt{d_k} and putting them through the softmax is covered on the Softmax page.

Check it in code

Eq. (5) is turned directly into loops and checked against the result of @. Click a symbol to highlight the same index in both the formula and the code. The inner index k runs over the columns of A\mathbf{A} and the rows of B\mathbf{B} at the same time.

Math
(AB)ij=∑k=1nAik Bkj(5)\sym{prod}{(\mathbf{A}\mathbf{B})}_{\sym{i}{i}\sym{j}{j}} = \sum_{\sym{k}{k}=1}^{n} A_{\sym{i}{i}\sym{k}{k}}\, B_{\sym{k}{k}\sym{j}{j}} \tag{5}
Code figures/matmul.py
import numpy as np

A = np.array([[1.0, 2.0, 2.0],      # 1행 = u
              [2.0, 1.0, 2.0]])     # 2행 = w
B = A.T                             # 식 (2) 의 전치: 3×2
m, n = A.shape
p = B.shape[1]

prod = np.zeros((m, p))             # 식 (6): (m×n)(n×p) = m×p
for i in range(m):
    for j in range(p):
        prod[i, j] = sum(A[i, k] * B[k, j] for k in range(n))   # 식 (5)

print("A", A.shape, "@ B", B.shape, "->", prod.shape)
print("반복문으로:\n", prod)
print("A @ B 로:\n", A @ B)
print("B @ A 의 shape:", (B @ A).shape)
print("A @ u =", A @ np.array([1.0, 2.0, 2.0]))   # 식 (3)
$ python matmul.py
A (2, 3) @ B (3, 2) -> (2, 2)
반복문으로:
 [[9. 8.]
 [8. 9.]]
A @ B 로:
 [[9. 8.]
 [8. 9.]]
B @ A 의 shape: (3, 3)
A @ u = [9. 8.]

Try it yourself

  1. Find the entry in row 3, column 3 of A⊤A\mathbf{A}^\top\mathbf{A} with Eq. (5). How is this value related to the length of which column of A\mathbf{A}?
  2. Where does (1,−1)(1, -1) go when multiplied by M\mathbf{M}? Express it in terms of a\mathbf{a} and b\mathbf{b}.
  3. If you feed a batch X of shape 4×34 \times 3 into nn.Linear(3, 2), what is the shape of the output? Why does X @ layer.weight give an error?
Answers
  1. It is the dot product of column 3 of A\mathbf{A}, (2,2)(2, 2), with itself, so 2×2+2×2=82 \times 2 + 2 \times 2 = 8. That is the square of the length of column 3, 8\sqrt{8}. The diagonal of A⊤A\mathbf{A}^\top\mathbf{A} holds the squared lengths of the columns.
  2. By Eq. (4), 1×(3,1)+(−1)×(1,2)=(2,−1)1 \times (3, 1) + (-1) \times (1, 2) = (2, -1), that is, a−b\mathbf{a} - \mathbf{b}. Reading by rows also gives (3−1, 1−2)=(2,−1)(3 - 1,\ 1 - 2) = (2, -1).
  3. The XW⊤\mathbf{X}\mathbf{W}^\top of Eq. (8) is (4×3)(3×2)(4 \times 3)(3 \times 2), so the output is (4, 2). layer.weight is (2, 3), so X @ layer.weight is (4×3)(2×3)(4 \times 3)(2 \times 3), and since the inner 3 and 2 differ, it cannot be multiplied.

Further reading

  • Deep Learning, Chapter 2 Linear Algebra — Goodfellow, Bengio, Courville, 2016. Section 2.2 defines the matrix product as in Eq. (5), and notes separately that it is not commutative and that there is the Hadamard product A ⊙ B. Section 2.3 covers the identity matrix and the inverse matrix. The section 'How to compute a matrix multiplication — m×p dot products' and the section 'Common misconceptions about matrix multiplication' rely on it.
  • MIT 18.06 Linear Algebra, Spring 2010 — Gilbert Strang, MIT OpenCourseWare, 2010. Lecture 3 (Multiplication and inverse matrices) shows how to read a matrix product by rows, by columns, and as combinations of columns. It pairs with Section 2.4 (Rules for Matrix Operations) of the textbook Introduction to Linear Algebra. The sections 'A matrix–vector product is one dot product per row' and 'Where does a matrix move the points of the plane — linear transformations' rely on it.
  • numpy.matmul — NumPy official documentation. It gives the signature (n,k),(k,m)->(n,m), and states that it raises ValueError when the last size of the first array differs from the second-to-last size of the second array. The 'matmul shape error' question relies on it.
  • torch.nn.Linear — PyTorch official documentation. It gives the layer's formula y=xA⊤+by = xA^\top + b, the shape of weight, (out_features, in_features), and the shape of bias, (out_features). The mapping table and the question in the section 'Where matrix multiplication shows up in AI' rely on it.
  • Attention Is All You Need — Vaswani et al., 2017. Eq. (1) in Section 3.2.1 computes the attention scores as the product of the query matrix and the key matrix, QK⊤QK^\top, divided by dk\sqrt{d_k}. The section 'Why attention's QKᵀ transposes K' relies on it.

Video notes

Video notes
  • 00:00 Opening question — "How many dot products is a matrix product?"
  • 00:40 Eqs. (1) and (2) — the matrix as a table, the entry AijA_{ij}, the transpose
  • 02:00 Eq. (3) — Au=(9,8)\mathbf{A}\mathbf{u} = (9, 8), one dot product per row
  • 03:20 Figure 1 and Eq. (4) — the columns are where the standard basis vectors go, the rubber sheet with a grid
  • 05:00 Figure 2 and Eqs. (5) and (6) — the 8 in row 1, column 2, why the inner sizes must match
  • 06:40 Figure 3 and Eq. (7) — MS≠SM\mathbf{M}\mathbf{S} \ne \mathbf{S}\mathbf{M}, the one applied first is on the right
  • 08:00 Eq. (8) — why the nn.Linear weight is (2, 3), QK⊤\mathbf{Q}\mathbf{K}^\top · wrap-up

This is the filming plan. After upload, the timestamps will be replaced with the real ones.

Where this sits on the map

Sigma NotationCalculusVectors and Dot ProductLinear AlgebraMatrix OperationsYou are hereAttentionPlannedEigenvaluesLinear AlgebraLinear TransformationLinear AlgebraSoftmax RegressionPlanned
Prerequisites
This page
Next
DonePlannedThe path to this pageSee the full map →