Linear AlgebraVectors and Dot Product#004

Vectors and Dot Product

벡터와 내적

The dot product of two vectors multiplies matching components and adds them into one number. (3, 1) and (1, 2) give 5, and dividing by the two lengths gives a cosine similarity of 0.707 (45°). See where the two disagree, with projection and NumPy code.

Key symbol
x·y
Reading time
16 min
Prerequisites
Sigma Notation

Translated by an agent and not reviewed yet. Original: 한국어

Intuition — the shadow of a stick at noon

Intuition

At noon, with sunlight falling straight down on the ground, lay one stick on the ground, stand another at its base, and tilt it only within the upright wall that stands right over the lying stick. The standing stick's shadow then falls along the lying stick: it gets longer the more you tilt the stick toward the ground, it shrinks to a point when the stick stands straight up, and it falls on the opposite side when you tip the stick backward. Give the shadow's length a negative sign when it falls in the opposite direction; multiplying this signed value by the lying stick's length gives the dot product.

Here the lying stick is the vector a\mathbf{a} and the standing stick is b\mathbf{b}. The wall is the plane in which the two vectors lie together (the plane of Figure 1). The angle between the two sticks is the angle φ\varphi. The shadow is b\mathbf{b} dropped onto a\mathbf{a}, and it comes back later under the name projection.

AI models turn words and sentences into vectors of hundreds of numbers. How well two vectors match is measured with the dot product. Search for similar documents and attention in a transformer both use this computation.

This page measures this shadow with numbers, using a 2-D pair a=(3,1)\mathbf{a} = (3, 1), b=(1,2)\mathbf{b} = (1, 2) and a 3-D pair u\mathbf{u}, w\mathbf{w}. Then it asks whether a large dot product means 'similar'.

A vector is numbers lined up in order

A vector is nn numbers lined up in a fixed order. Each of those numbers is a component, and the count nn is the dimension.

x=(x1, x2, …, xn)∈Rn(1)\mathbf{x} = (x_1,\ x_2,\ \dots,\ x_n) \in \mathbb{R}^n \tag{1}

Rn\mathbb{R}^n is the set of all vectors of nn real numbers, and ∈\in means 'belongs to it'. a=(3,1)\mathbf{a} = (3, 1) is 2-D, so it can be drawn as an arrow in the plane (Figure 1). An embedding, which turns a word or a sentence into 768 numbers, cannot be drawn, but it is just as much a 768-dimensional vector.11 In physics you learn a vector as 'an arrow with a magnitude and a direction'. The arrow is only one way to draw 2-D and 3-D vectors; the definition is the list of numbers in Eq. (1).

Two vectors of the same dimension are added component by component. Multiplying by a single number (a scalar) multiplies every component (a scalar multiple).

x+y=(x1+y1, …, xn+yn),s x=(sx1, …, sxn)(2)\mathbf{x} + \mathbf{y} = (x_1 + y_1,\ \dots,\ x_n + y_n), \qquad s\,\mathbf{x} = (s x_1,\ \dots,\ s x_n) \tag{2}

a+b=(3+1, 1+2)=(4,3)\mathbf{a} + \mathbf{b} = (3 + 1,\ 1 + 2) = (4, 3), and 2a=(6,2)2\mathbf{a} = (6, 2). In the upper panel of Figure 1, a+b\mathbf{a} + \mathbf{b} is where you land when you attach b\mathbf{b} to the tip of a\mathbf{a}, and 2a2\mathbf{a} is twice as long in the same direction. Subtraction also works component by component: a−b=(2,−1)\mathbf{a} - \mathbf{b} = (2, -1).

The length of a vector (its magnitude, or norm) is the Pythagorean theorem extended to nn components.

∥x∥=x12+x22+⋯+xn2(3)\lVert \mathbf{x} \rVert = \sqrt{x_1^2 + x_2^2 + \cdots + x_n^2} \tag{3}

∥a∥=9+1=10≈3.16\lVert \mathbf{a} \rVert = \sqrt{9 + 1} = \sqrt{10} \approx 3.16 and ∥b∥=5≈2.24\lVert \mathbf{b} \rVert = \sqrt{5} \approx 2.24. ∥a+b∥=16+9=5\lVert \mathbf{a} + \mathbf{b} \rVert = \sqrt{16 + 9} = 5, and the length of 2a2\mathbf{a} is twice that of a\mathbf{a}. The distance between two vectors is measured by the length of their difference, ∥x−y∥\lVert \mathbf{x} - \mathbf{y} \rVert.

Definition
SymbolNameMeaning
x,y\mathbf{x}, \mathbf{y}vectorNumbers lined up in order. Written as bold lowercase letters
xix_icomponentThe iith number of x\mathbf{x}. A plain letter with a subscript is used only for components
nndimensionThe number of components
Rn\mathbb{R}^nnn-dimensional real coordinate spaceThe set of all vectors of nn real numbers
ssscalarA single number. Multiplying a vector by it multiplies every component
∥x∥\lVert \mathbf{x} \rVertnorm (length, magnitude)Eq. (3). The distance from the origin to the tip of the vector. Also called the L2 norm or the Euclidean norm. The distance between two vectors is ∥x−y∥\lVert \mathbf{x} - \mathbf{y} \rVert
x⋅y\mathbf{x} \cdot \mathbf{y}dot productEq. (4). The result is a single number. Also called the inner product or the scalar product (a different operation from the scalar multiple). Texts also write it x⊤y\mathbf{x}^\top \mathbf{y} (⊤\top is the transpose; see the Matrix Operations page) or ⟨x,y⟩\langle \mathbf{x}, \mathbf{y} \rangle. The protagonist of this page
φ\varphiangleThe angle between two vectors, between 0° and 180°. Textbooks often write it θ\theta, but on this site θ\theta is the parameter, so we write φ\varphi
proj⁡xy\operatorname{proj}_{\mathbf{x}} \mathbf{y}projectionEq. (6). The shadow of y\mathbf{y} dropped perpendicularly onto the line in the direction of x\mathbf{x}
cos⁡φ\cos\varphi, cos⁡−1\cos^{-1}cosine similarity, inverse cosine (arccos)Eq. (7). The dot product divided by the product of the two lengths. Between −1-1 and 1. cos⁡−1\cos^{-1} turns a cosine value between −1-1 and 1 back into an angle between 0° and 180°. It is not 1/cos⁡1/\cos
a,b,u,w\mathbf{a}, \mathbf{b}, \mathbf{u}, \mathbf{w}example vectorsa=(3,1)\mathbf{a} = (3, 1), b=(1,2)\mathbf{b} = (1, 2), u=(1,2,2)\mathbf{u} = (1, 2, 2), w=(2,1,2)\mathbf{w} = (2, 1, 2). w\mathbf{w} is the name of an example, not a weight

School textbooks often write a vector as a⃗\vec{a} and its length as ∣a⃗∣\lvert\vec{a}\rvert. They are the same things as this page's a\mathbf{a} and ∥a∥\lVert\mathbf{a}\rVert, and the projection is also called the orthogonal projection.

The 2-D example a = (3, 1), b = (1, 2). Top: the sum a + b = (4, 3) (the diagonal of the dashed parallelogram) and the scalar multiple 2a = (6, 2) (dashed arrow). Bottom: the angle between the two vectors, φ = 45°, and the projection (1.5, 0.5) of b dropped perpendicularly onto a (thick arrow).
Figure 1 The 2-D example a=(3, 1)\mathbf{a} = (3,\ 1), b=(1, 2)\mathbf{b} = (1,\ 2). Top: the sum a+b=(4, 3)\mathbf{a} + \mathbf{b} = (4,\ 3) (the diagonal of the dashed parallelogram) and the scalar multiple 2a=(6, 2)2\mathbf{a} = (6,\ 2) (dashed arrow). Bottom: the angle between the two vectors, φ=45°\varphi = 45°, and the projection (1.5, 0.5)(1.5,\ 0.5) of b\mathbf{b} dropped perpendicularly onto a\mathbf{a} (thick arrow). figures/vector_geometry.py

The dot product is a single number — multiply matching components and add them all

The dot product of two vectors multiplies the components in the same position and adds everything up. The result is not a vector but a single number.

x⋅y=∑i=1nxiyi=x1y1+x2y2+⋯+xnyn(4)\mathbf{x} \cdot \mathbf{y} = \sum_{i=1}^{n} x_i y_i = x_1 y_1 + x_2 y_2 + \cdots + x_n y_n \tag{4}

∑i=1n\sum_{i=1}^{n} is the symbol (sigma) that says to change ii from 1 to nn one at a time and add up the expression after it. Sigma notation has its own page, Sigma Notation.

a⋅b=3×1+1×2=5\mathbf{a} \cdot \mathbf{b} = 3 \times 1 + 1 \times 2 = 5. The 3-D pair works the same way: u⋅w=1×2+2×1+2×2=8\mathbf{u} \cdot \mathbf{w} = 1 \times 2 + 2 \times 1 + 2 \times 2 = 8. For two vectors of different dimensions there are not enough components to pair up, so the dot product is not defined.

The dot product of a vector with itself adds up the squares of its components, so it is the square of the length in Eq. (3): a⋅a=9+1=10=∥a∥2\mathbf{a} \cdot \mathbf{a} = 9 + 1 = 10 = \lVert \mathbf{a} \rVert^2. Likewise u⋅u=1+4+4=9\mathbf{u} \cdot \mathbf{u} = 1 + 4 + 4 = 9, so ∥u∥=3\lVert \mathbf{u} \rVert = 3, and ∥w∥\lVert \mathbf{w} \rVert is also 3.

Swapping the order gives the same value, x⋅y=y⋅x\mathbf{x} \cdot \mathbf{y} = \mathbf{y} \cdot \mathbf{x}. Multiplying one side by ss multiplies the dot product by ss. The dot product of 2a2\mathbf{a} and b\mathbf{b} is 6+4=106 + 4 = 10, twice a⋅b\mathbf{a} \cdot \mathbf{b}.

Why the dot product is the product of two lengths and a cosine

Both vectors must be nonzero when referring to their angle.

The dot product equals the product of the two lengths times the cosine of the angle between them. For nonparallel vectors, it follows from computing a side length of the triangle formed by x\mathbf{x}, y\mathbf{y}, x−y\mathbf{x} - \mathbf{y} once with Eq. (4) and once with the law of cosines, and matching the two (folded derivation below).22 Many school textbooks take Eq. (5) as the definition of the dot product and teach Eq. (4) as its component form. Only the order differs; the value is the same.

x⋅y=∥x∥ ∥y∥cos⁡φ(5)\mathbf{x} \cdot \mathbf{y} = \lVert \mathbf{x} \rVert\, \lVert \mathbf{y} \rVert \cos\varphi \tag{5}

The cosine cos⁡φ\cos\varphi is the horizontal coordinate of the point on a circle of radius 1 reached by turning φ\varphi from the horizontal axis. It equals the signed shadow of a stick of length 1 held at the angle φ\varphi to the horizontal axis (the lying stick). It is 1 at 0°, 0 at 90°, and −1-1 at 180°, and in between it decreases as the angle grows (Figure 2).

Put in the a,b\mathbf{a}, \mathbf{b} from the lower panel of Figure 1 and you get 5=10×5×cos⁡φ5 = \sqrt{10} \times \sqrt{5} \times \cos\varphi. So cos⁡φ=5/50=1/2≈0.707\cos\varphi = 5 / \sqrt{50} = 1/\sqrt{2} \approx 0.707. The inverse cosine, which turns a cosine value back into an angle, gives φ=cos⁡−1(1/2)=45°\varphi = \cos^{-1}(1/\sqrt{2}) = 45°. NumPy's np.arccos returns the angle in radians (0.785), so to see it in degrees use np.degrees(np.arccos(x)).

Group Eq. (5) as ∥x∥×(∥y∥cos⁡φ)\lVert \mathbf{x} \rVert \times (\lVert \mathbf{y} \rVert \cos\varphi) and the shadow from the intuition appears. ∥y∥cos⁡φ\lVert \mathbf{y} \rVert \cos\varphi is the signed component of the shadow of y\mathbf{y} dropped perpendicularly onto the line in the direction of x\mathbf{x}. When the angle exceeds 90°, this value is negative, so the shadow falls on the opposite side.

Why Eq. (4) and Eq. (5) are equal — the law of cosines

Compute the length of the side x−y\mathbf{x} - \mathbf{y}, which joins the tips of the two vectors, in two ways. Expanding in components, Eq. (3) and Eq. (4) give the following.

∥x−y∥2=∑i=1n(xi−yi)2=∥x∥2+∥y∥2−2 x⋅y\lVert \mathbf{x} - \mathbf{y} \rVert^2 = \sum_{i=1}^{n} (x_i - y_i)^2 = \lVert \mathbf{x} \rVert^2 + \lVert \mathbf{y} \rVert^2 - 2\, \mathbf{x} \cdot \mathbf{y}

For two nonparallel vectors, applying the law of cosines to the triangle with sides x\mathbf{x}, y\mathbf{y}, x−y\mathbf{x} - \mathbf{y} gives the following.

∥x−y∥2=∥x∥2+∥y∥2−2 ∥x∥ ∥y∥cos⁡φ\lVert \mathbf{x} - \mathbf{y} \rVert^2 = \lVert \mathbf{x} \rVert^2 + \lVert \mathbf{y} \rVert^2 - 2\, \lVert \mathbf{x} \rVert\, \lVert \mathbf{y} \rVert \cos\varphi

Matching the last terms of the two equations gives Eq. (5). Checking with a,b\mathbf{a}, \mathbf{b}: a−b=(2,−1)\mathbf{a} - \mathbf{b} = (2, -1), so ∥a−b∥2=5\lVert \mathbf{a} - \mathbf{b} \rVert^2 = 5, and 10+5−2×5=510 + 5 - 2 \times 5 = 5 agrees.

For parallel nonzero vectors, write y=cx\mathbf{y}=c\mathbf{x} with c≠0c\ne0. Their dot product is c∥x∥2c\lVert\mathbf{x}\rVert^2, and ∥y∥=∣c∣∥x∥\lVert\mathbf{y}\rVert=|c|\lVert\mathbf{x}\rVert. The angle is 0° for c>0c>0 and 180° for c<0c<0, so Eq. (5) also holds in this case.

Above 3 dimensions you cannot draw the angle, so it goes the other way: Eq. (5) defines the angle. Thanks to the Cauchy–Schwarz inequality ∣x⋅y∣≤∥x∥ ∥y∥\lvert \mathbf{x} \cdot \mathbf{y} \rvert \le \lVert \mathbf{x} \rVert\, \lVert \mathbf{y} \rVert, cos⁡φ\cos\varphi falls between −1-1 and 1, so a single angle is determined (when neither vector is the zero vector). For the two pairs on this page, 5≤50≈7.075 \le \sqrt{50} \approx 7.07 and 8≤98 \le 9.

A projection is a shadow found with the dot product

The direction vector x\mathbf{x} must be nonzero. The length of the projection vector is ∣x⋅y∣/∥x∥\lvert\mathbf{x}\cdot\mathbf{y}\rvert/\lVert\mathbf{x}\rVert, which is nonnegative and differs from its signed component.

The shadow of y\mathbf{y} dropped perpendicularly onto the line in the direction of x\mathbf{x} is called the projection (orthogonal projection). It is the signed component x⋅y/∥x∥\mathbf{x} \cdot \mathbf{y} / \lVert \mathbf{x} \rVert times the vector of length 1 in the direction of x\mathbf{x}, x/∥x∥\mathbf{x} / \lVert \mathbf{x} \rVert, so the denominator is the squared length.

proj⁡xy=x⋅y∥x∥2 x(6)\operatorname{proj}_{\mathbf{x}} \mathbf{y} = \frac{\mathbf{x} \cdot \mathbf{y}}{\lVert \mathbf{x} \rVert^2}\, \mathbf{x} \tag{6}

Set c=(x⋅y)/∥x∥2c=(\mathbf{x}\cdot\mathbf{y})/\lVert\mathbf{x}\rVert^2, so the projection is cxc\mathbf{x}. The residual has dot product x⋅(y−cx)\mathbf{x}\cdot(\mathbf{y}-c\mathbf{x}) with the direction vector. Expanding gives x⋅y−c∥x∥2=0\mathbf{x}\cdot\mathbf{y}-c\lVert\mathbf{x}\rVert^2=0, so the residual is always perpendicular to the projection direction.

Dropping b\mathbf{b} onto a\mathbf{a}, the multiplier is 5/10=0.55 / 10 = 0.5, so proj⁡ab=0.5×(3,1)=(1.5,0.5)\operatorname{proj}_{\mathbf{a}} \mathbf{b} = 0.5 \times (3, 1) = (1.5, 0.5). This is the thick arrow in the lower panel of Figure 1. Its length is 5/10≈1.585 / \sqrt{10} \approx 1.58, the same as ∥b∥cos⁡φ=5/2\lVert \mathbf{b} \rVert \cos\varphi = \sqrt{5} / \sqrt{2}.

The remaining part, b−(1.5,0.5)=(−0.5,1.5)\mathbf{b} - (1.5, 0.5) = (-0.5, 1.5), has dot product −1.5+1.5=0-1.5 + 1.5 = 0 with a\mathbf{a}. A projection splits a vector into 'the part along another vector' and 'the part at right angles to it'. The dashed line and the small right-angle mark in the lower panel of Figure 1 are that second part.

What it means when the dot product is 0 or negative

For vectors that are not the zero vector the lengths are positive, so the sign of the dot product follows the sign of cos⁡φ\cos\varphi. It is positive when the angle is less than 90°, 0 at 90°, and negative when it is greater than 90° (Figure 2).

Two vectors whose dot product is 0 are called orthogonal. It means the same as perpendicular, or at right angles. a=(3,1)\mathbf{a} = (3, 1) and (−1,3)(-1, 3) are orthogonal because 3×(−1)+1×3=03 \times (-1) + 1 \times 3 = 0.

a\mathbf{a} and −a-\mathbf{a} point in exactly opposite directions, so cos⁡φ=−1\cos\varphi = -1, and the dot product is −∥a∥2=−10-\lVert \mathbf{a} \rVert^2 = -10. Figure 2 places the five pairs of this page on the cosine curve. 12u=(0.5,1,1)\tfrac{1}{2}\mathbf{u} = (0.5, 1, 1) halves every component of u\mathbf{u}, so it has the same direction, and the angle 27.3° between u\mathbf{u} and w\mathbf{w} is cos⁡−1(8/9)\cos^{-1}(8/9), the dot product 8 divided by the product of the two lengths 3×3=93 \times 3 = 9.

cosφ against the angle φ. The dots are the five pairs of this page: u and 1/2u give 1 at 0°, u and w give 0.889 at about 27.3°, a and b give 0.707 at 45°, a and (-1, 3) give 0 at 90°, and a and -a give -1 at 180°. The upper band (positive) is φ < 90°, and the lower band (negative) is φ > 90°.
Figure 2 cos⁡φ\cos\varphi against the angle φ\varphi. The dots are the five pairs of this page: u\mathbf{u} and 12u\tfrac{1}{2}\mathbf{u} give 1 at 0°, u\mathbf{u} and w\mathbf{w} give 0.889 at about 27.3°, a\mathbf{a} and b\mathbf{b} give 0.707 at 45°, a\mathbf{a} and (−1, 3)(-1,\ 3) give 0 at 90°, and a\mathbf{a} and −a-\mathbf{a} give −1-1 at 180°. The upper band (positive) is φ<90°\varphi < 90°, and the lower band (negative) is φ>90°\varphi > 90°. figures/cosine_curve.py

What does a dot product of 0 mean?

A dot product of 0 means the two vectors are orthogonal (perpendicular to each other, with an angle of 90°). The dot product is the product of the two lengths and cos⁡φ\cos\varphi, so if neither is the zero vector, cos⁡φ=0\cos\varphi = 0, and between 0° and 180° the only such angle is 90°. a=(3,1)\mathbf{a} = (3, 1) and (−1,3)(-1, 3) sit at the 90° spot in Figure 2. If one of them is the zero vector (all components 0), the angle is not defined but the dot product is 0, so by convention the zero vector is taken to be orthogonal to every vector.

Does a negative cosine similarity mean opposite?

No. A negative cosine similarity means the angle is greater than 90°; the vectors point in exactly opposite directions only when it is −1-1. a\mathbf{a} and −a-\mathbf{a} give −1-1 (180° in Figure 2). 0 is not opposite but orthogonal (90°).

Why cosine similarity divides the dot product by the two lengths

Cosine similarity is the dot product divided by the product of the two lengths (when neither vector is the zero vector). By Eq. (5) the dot product is also proportional to the two lengths, so only the direction is left after dividing. Among vectors with a positive dot product (at less than 90° to each other), one in the same direction gives a small value if it is short, and one that matches less gives a large value if it is long. 12u\tfrac{1}{2}\mathbf{u}, in exactly the same direction as the query u\mathbf{u}, has a dot product of 4.5, while the worse match 2w2\mathbf{w} has 16.

cos⁡φ=x⋅y∥x∥ ∥y∥(7)\cos\varphi = \frac{\mathbf{x} \cdot \mathbf{y}}{\lVert \mathbf{x} \rVert\, \lVert \mathbf{y} \rVert} \tag{7}

Figure 3 compares the query vector u=(1,2,2)\mathbf{u} = (1, 2, 2) with three candidates. The dot products are 4.5, 8, 16 for 12u\tfrac{1}{2}\mathbf{u} (0.5+2+20.5 + 2 + 2), w\mathbf{w}, 2w2\mathbf{w} (2×82 \times 8) in that order.

Measured by cosine similarity, the 1st place and the last place swap, and 12u\tfrac{1}{2}\mathbf{u} is the largest, at 1. w\mathbf{w} and 2w2\mathbf{w} tie at 0.889, because Eq. (7) divides the dot product 16 by the product of the two lengths 3×6=183 \times 6 = 18 and removes the share of the added length.

Dot products (left) and cosine similarities (right) between the query u = (1, 2, 2) and the candidates 1/2u, w, 2w. The dot products are 4.5, 8, 16, so 2w is the largest, but the cosine similarities are 1, 0.889, 0.889, so 1/2u, in the same direction as u, is the largest.
Figure 3 Dot products (left) and cosine similarities (right) between the query u=(1, 2, 2)\mathbf{u} = (1,\ 2,\ 2) and the candidates 12u\tfrac{1}{2}\mathbf{u}, w\mathbf{w}, 2w2\mathbf{w}. The dot products are 4.5, 8, 16, so 2w2\mathbf{w} is the largest, but the cosine similarities are 1, 0.889, 0.889, so 12u\tfrac{1}{2}\mathbf{u}, in the same direction as u\mathbf{u}, is the largest. figures/dot_vs_cosine.py

If you first divide each vector by its own length so that it becomes a vector of length 1 (a unit vector), which is normalization, the dot product is the cosine similarity, because the denominator of Eq. (7) becomes 1. For example, a/∥a∥=(3,1)/10≈(0.949,0.316)\mathbf{a} / \lVert \mathbf{a} \rVert = (3, 1)/\sqrt{10} \approx (0.949, 0.316) has length 1.

Key idea

When both vectors are nonzero, the dot product is the product of two lengths and a cosine, so it is also proportional to the lengths. Whether two directions are similar is measured by the cosine similarity, which divides by the two lengths.

What's the difference between cosine similarity and the dot product?

Cosine similarity is the dot product divided by the product of the two lengths (Eq. (7)), so it leaves out length and measures only direction. That is why it lies between −1-1 and 1, and stays the same when you stretch or shrink one vector by a positive factor. Stretching w\mathbf{w} to 2w2\mathbf{w} takes the dot product from 8 to 16, but the cosine similarity stays at 0.889. If both vectors already have length 1, the two are the same value.

What's the difference between cosine similarity and Euclidean distance?

Cosine similarity measures only direction, while the Euclidean distance ∥x−y∥\lVert \mathbf{x} - \mathbf{y} \rVert also measures the difference in length. Seen from u\mathbf{u}, w\mathbf{w} and 2w2\mathbf{w} have the same cosine similarity, 0.889, but their distances differ, about 1.41 and 3.61. If both vectors are normalized to length 1, ∥x−y∥2=2(1−cos⁡φ)\lVert \mathbf{x} - \mathbf{y} \rVert^2 = 2(1 - \cos\varphi), so the order from nearest is the same as the order from the largest cosine similarity.

Common misconceptions about vectors and the dot product

  • The result of a dot product is also a vector — Eq. (4) takes the element-wise products (3,2)(3, 2) and adds them up again into a single number, 5. In NumPy a * b is the element-wise product [3. 2.] (a vector), and the dot product is the 5.0 of a @ b or np.dot(a, b).
  • A large dot product means the two vectors are close — In Figure 3, 2w2\mathbf{w}, with the largest dot product (16), is the farthest of the three candidates from u\mathbf{u}: u−2w=(−3,0,−2)\mathbf{u} - 2\mathbf{w} = (-3, 0, -2), so the distance is 9+0+4=13≈3.61\sqrt{9 + 0 + 4} = \sqrt{13} \approx 3.61. 12u\tfrac{1}{2}\mathbf{u}, in exactly the same direction, has the smallest dot product (4.5).
  • It is a vector only if you can draw it as an arrow — By Eq. (1), any ordered list of numbers is a vector, so Eqs. (2)–(7) apply to a 768-dimensional embedding that cannot be drawn exactly as they do to the 2-D a,b\mathbf{a}, \mathbf{b}.

Does embedding search measure with the dot product or cosine similarity

With the similarity the search model was trained with. Often that is cosine similarity, and a model trained with the unnormalized dot product measures with the dot product as it is.

AI models turn words, sentences, and images into embedding vectors with hundreds to thousands of dimensions. A search for similar documents picks document vectors in order of their similarity to the query vector. If the vectors are normalized in advance, computing just the dot product gives the same order as cosine similarity.

There are also search models trained with the dot product of two unnormalized vectors as the similarity (DPR, Karpukhin et al. 2020). If you normalize the vectors of such a model and switch to cosine, the ranking can change, as in Figure 3.

Where vectors and the dot product show up in AI

Attention in a transformer uses the dot product of a query vector q\mathbf{q} and a key vector k\mathbf{k} as a score. It divides this score by dk\sqrt{d_k} and puts it through the softmax to turn it into weights that sum to 1. The softmax is covered on the Softmax page.

Definition
SymbolNameMeaning
q,k,dk\mathbf{q}, \mathbf{k}, d_kquery, key, key dimensionThe two vectors whose dot product attention takes, and their dimension. The subscript of dkd_k is a conventional notation, not a component index

dk\sqrt{d_k} divides the score of every pair by the same number, unlike cosine similarity, which divides each pair by its own two lengths. So in the attention scores of the original transformer (Vaswani et al.), the lengths of the vectors remain.

In training, too, the parameters and the gradient are vectors. One step of gradient descent is the operation of Eq. (2): multiply the gradient by the learning rate (a scalar multiple) and subtract it from the parameters (addition). The velocity of momentum is also a vector: some multiple of the previous velocity plus the gradient.

This pageNumPy · PyTorch · SciPy
Vector x\mathbf{x}1-D array np.array([3.0, 1.0]), torch.tensor([3.0, 1.0]). The shape is (2,) in NumPy and torch.Size([2]) in PyTorch
x+y\mathbf{x} + \mathbf{y}, s xs\,\mathbf{x} (Eq. (2))x + y, s * x
Element-wise product (x1y1, …, xnyn)(x_1 y_1,\ \dots,\ x_n y_n)x * y — not the dot product. (x * y).sum() is the dot product
x⋅y\mathbf{x} \cdot \mathbf{y} (Eq. (4))x @ y, np.dot(x, y), torch.dot(x, y) (torch.dot takes only two 1-D tensors)
∥x∥\lVert \mathbf{x} \rVert (Eq. (3))np.linalg.norm(x), torch.linalg.vector_norm(x) — the default is the 2-norm of Eq. (3)
cos⁡φ\cos\varphi (Eq. (7))x @ y / (np.linalg.norm(x) * np.linalg.norm(y)), torch.cosine_similarity(x1, x2, dim=1, eps=1e-8) (x1 and x2 are the two tensors to compare, not components). The default of dim is 1, so for two 1-D vectors pass dim=0 (or dim=-1)
1−cos⁡φ1 - \cos\varphiscipy.spatial.distance.cosine(x, y) — despite its name, it returns not the similarity but 1−cos⁡φ1 - \cos\varphi, often called the 'cosine distance'. It is a different quantity from this page's distance ∥x−y∥\lVert \mathbf{x} - \mathbf{y} \rVert; for vectors of length 1, ∥x−y∥2=2(1−cos⁡φ)\lVert \mathbf{x} - \mathbf{y} \rVert^2 = 2(1 - \cos\varphi)
Normalization x/∥x∥\mathbf{x} / \lVert \mathbf{x} \rVerttorch.nn.functional.normalize(x, dim=0) (defaults p=2, dim=1)
q⋅k\mathbf{q} \cdot \mathbf{k}q @ k. Real models compute many scores with a single matrix product (Matrix Operations)

What's the difference between np.dot, @, and *?

For two 1-D arrays, np.dot(a, b) and a @ b are the same dot product, and a * b is the element-wise product. With a=(3,1)\mathbf{a} = (3, 1) and b=(1,2)\mathbf{b} = (1, 2) the first two give 5.0. a * b gives the vector (3, 2), and you have to add it up with .sum() to get the dot product 5. For 2-D arrays (matrices), @ is the matrix product.

Check it in code

Click a symbol to highlight it in both the formula and the code. The first line of the output also shows that a * b is the element-wise product, not the dot product.

Math
a⋅b=∑i=12aibi(4)\sym{inner}{\mathbf{a} \cdot \mathbf{b}} = \sum_{i=1}^{2} a_i b_i \tag{4}proj⁡ab=a⋅b∥a∥2 a(6)\sym{proj}{\operatorname{proj}_{\mathbf{a}} \mathbf{b}} = \frac{\sym{inner}{\mathbf{a} \cdot \mathbf{b}}}{\sym{norm_a}{\lVert \mathbf{a} \rVert}^2}\, \mathbf{a} \tag{6}cos⁡φ=a⋅b∥a∥ ∥b∥(7)\sym{cos_phi}{\cos\varphi} = \frac{\sym{inner}{\mathbf{a} \cdot \mathbf{b}}}{\sym{norm_a}{\lVert \mathbf{a} \rVert}\, \sym{norm_b}{\lVert \mathbf{b} \rVert}} \tag{7}
Code figures/dot_product.py
import numpy as np

a = np.array([3.0, 1.0])            # 𝐚
b = np.array([1.0, 2.0])            # 𝐛

inner = np.sum(a * b)               # 식 (4): 성분끼리 곱해(a * b) 모두 더한다
norm_a = np.sqrt(np.sum(a * a))     # 식 (3): ‖a‖ = √(a·a)
norm_b = np.sqrt(np.sum(b * b))
proj = inner / np.sum(a * a) * a    # 식 (6): ‖a‖² = a·a. b 를 a 위로 내린 그림자
cos_phi = inner / (norm_a * norm_b) # 식 (7)
print("a * b =", a * b, "  <- 성분별 곱. 아직 벡터다")
print("a·b =", inner, "  a @ b =", a @ b, "  np.dot(a, b) =", np.dot(a, b))
print(f"‖a‖ = {norm_a:.3f}  ‖b‖ = {norm_b:.3f}")
print(f"cos φ = {cos_phi:.3f}  φ = {np.degrees(np.arccos(cos_phi)):.1f}°")
print("사영 =", proj, "  (b - 사영)·a =", (b - proj) @ a)

u = np.array([1.0, 2.0, 2.0])       # 𝐮 (3차원)
w = np.array([2.0, 1.0, 2.0])       # 𝐰
print("u·w =", u @ w, "  cos φ =", round(u @ w / (np.linalg.norm(u) * np.linalg.norm(w)), 3))
$ python dot_product.py
a * b = [3. 2.]   <- 성분별 곱. 아직 벡터다
a·b = 5.0   a @ b = 5.0   np.dot(a, b) = 5.0
‖a‖ = 3.162  ‖b‖ = 2.236
cos φ = 0.707  φ = 45.0°
사영 = [1.5 0.5]   (b - 사영)·a = 0.0
u·w = 8.0   cos φ = 0.889

Try it yourself

  1. Find the dot product of u=(1,2,2)\mathbf{u} = (1, 2, 2) and (0,3,4)(0, 3, 4), their two lengths, and cos⁡φ\cos\varphi. Roughly how many degrees is the angle, and compared with w\mathbf{w} (27.3°), which one is closer in direction to u\mathbf{u}?
  2. Is (2,−2,1)(2, -2, 1) orthogonal to u\mathbf{u}? What about w\mathbf{w}?
  3. This time, find the projection of a\mathbf{a} onto b\mathbf{b} with Eq. (6). What comes out, and why?
Answers
  1. The dot product is 0+6+8=140 + 6 + 8 = 14, and the lengths are ∥u∥=3\lVert \mathbf{u} \rVert = 3 and 0+9+16=5\sqrt{0 + 9 + 16} = 5. So cos⁡φ=14/15≈0.933\cos\varphi = 14/15 \approx 0.933, and the angle is cos⁡−1(14/15)≈21.0°\cos^{-1}(14/15) \approx 21.0°. The angle is smaller than the 27.3° of w\mathbf{w}, so (0,3,4)(0, 3, 4) is closer in direction to u\mathbf{u}. Its dot product, 14, is also larger than the 8 of w\mathbf{w}, but that is partly because it is longer, with length 5.
  2. u⋅(2,−2,1)=2−4+2=0\mathbf{u} \cdot (2, -2, 1) = 2 - 4 + 2 = 0, so they are orthogonal. w⋅(2,−2,1)=4−2+2=4\mathbf{w} \cdot (2, -2, 1) = 4 - 2 + 2 = 4, so it is not orthogonal to w\mathbf{w}.
  3. The multiplier is a⋅b/∥b∥2=5/5=1\mathbf{a} \cdot \mathbf{b} / \lVert \mathbf{b} \rVert^2 = 5/5 = 1, so the projection is b=(1,2)\mathbf{b} = (1, 2) itself. This means that the foot of the perpendicular dropped from the tip of a\mathbf{a} to the line along b\mathbf{b} happens to be the tip of b\mathbf{b}. Indeed, the remaining part a−b=(2,−1)\mathbf{a} - \mathbf{b} = (2, -1) has dot product 2−2=02 - 2 = 0 with b\mathbf{b}.

Further reading

  • Deep Learning, Chapter 2 Linear Algebra — Goodfellow, Bengio, Courville, 2016. Section 2.1 defines a vector as an array of numbers, Section 2.5 (Norms) rewrites the dot product as the product of two norms and a cosine, and Section 2.6 defines a dot product of 0 as orthogonality. The section 'A vector is numbers lined up in order' and Eq. (5) rely on it.
  • Introduction to Linear Algebra, Chapter 1 Introduction to Vectors — Gilbert Strang, 2016 (5th edition). The textbook of the MIT 18.06 course. Section 1.1 covers vector addition and scalar multiples, and Section 1.2 (Lengths and Dot Products) covers length, the dot product, the cosine formula, and the Schwarz inequality. Eqs. (2)–(5) and the folded derivation rely on it.
  • Introduction to Information Retrieval, 6.3.1 Dot products — Manning, Raghavan, Schütze, 2008. It treats documents as vectors and explains that they are compared with cosine similarity to remove the effect of document length. The sections 'Why cosine similarity divides the dot product by the two lengths' and 'Does embedding search …' and the section 'Common misconceptions about vectors and the dot product' rely on it.
  • Attention Is All You Need — Vaswani et al., 2017. Section 3.2.1 (Scaled Dot-Product Attention) defines attention, which divides the dot product of queries and keys by dk\sqrt{d_k} and puts it through the softmax. The section 'Where vectors and the dot product show up in AI' relies on it.
  • Dense Passage Retrieval for Open-Domain Question Answering — Karpukhin et al., 2020. EMNLP. It turns questions and passages into vectors separately and trains a search model with the dot product of the two unnormalized vectors as the similarity. The section 'Does embedding search measure with the dot product or cosine similarity' relies on it.

Video notes

Video notes
  • 00:00 Opening question — "Does a large dot product mean two vectors are similar?"
  • 00:40 Eqs. (1) and (2) — the vector as a list of numbers, the sum and the multiple by 2 in the upper panel of Figure 1
  • 02:00 Write Eq. (4) on the board — a⋅b=3+2=5\mathbf{a} \cdot \mathbf{b} = 3 + 2 = 5, the result is a single number
  • 03:20 Eq. (5) and the lower panel of Figure 1 — the angle 45°, the shadow (1.5, 0.5)(1.5,\ 0.5)
  • 05:00 Figure 2 — the sign of the dot product, and orthogonality
  • 06:10 Figure 3 and Eq. (7) — the 1st and last places swap between the dot product and cosine similarity; which one does search use
  • 07:30 Code lab — a * b and a @ b · wrap-up

This is the filming plan. After upload, the timestamps will be replaced with the real ones.

Where this sits on the map