Intuition — the shadow of a stick at noon
At noon, with sunlight falling straight down on the ground, lay one stick on the ground, stand another at its base, and tilt it only within the upright wall that stands right over the lying stick. The standing stick's shadow then falls along the lying stick: it gets longer the more you tilt the stick toward the ground, it shrinks to a point when the stick stands straight up, and it falls on the opposite side when you tip the stick backward. Give the shadow's length a negative sign when it falls in the opposite direction; multiplying this signed value by the lying stick's length gives the dot product.
Here the lying stick is the vector and the standing stick is . The wall is the plane in which the two vectors lie together (the plane of Figure 1). The angle between the two sticks is the angle . The shadow is dropped onto , and it comes back later under the name projection.
AI models turn words and sentences into vectors of hundreds of numbers. How well two vectors match is measured with the dot product. Search for similar documents and attention in a transformer both use this computation.
This page measures this shadow with numbers, using a 2-D pair , and a 3-D pair , . Then it asks whether a large dot product means 'similar'.
A vector is numbers lined up in order
A vector is numbers lined up in a fixed order. Each of those numbers is a component, and the count is the dimension.
is the set of all vectors of real numbers, and means 'belongs to it'. is 2-D, so it can be drawn as an arrow in the plane (Figure 1). An embedding, which turns a word or a sentence into 768 numbers, cannot be drawn, but it is just as much a 768-dimensional vector.11 In physics you learn a vector as 'an arrow with a magnitude and a direction'. The arrow is only one way to draw 2-D and 3-D vectors; the definition is the list of numbers in Eq. (1).
Two vectors of the same dimension are added component by component. Multiplying by a single number (a scalar) multiplies every component (a scalar multiple).
, and . In the upper panel of Figure 1, is where you land when you attach to the tip of , and is twice as long in the same direction. Subtraction also works component by component: .
The length of a vector (its magnitude, or norm) is the Pythagorean theorem extended to components.
and . , and the length of is twice that of . The distance between two vectors is measured by the length of their difference, .
| Symbol | Name | Meaning |
|---|---|---|
| vector | Numbers lined up in order. Written as bold lowercase letters | |
| component | The th number of . A plain letter with a subscript is used only for components | |
| dimension | The number of components | |
| -dimensional real coordinate space | The set of all vectors of real numbers | |
| scalar | A single number. Multiplying a vector by it multiplies every component | |
| norm (length, magnitude) | Eq. (3). The distance from the origin to the tip of the vector. Also called the L2 norm or the Euclidean norm. The distance between two vectors is | |
| dot product | Eq. (4). The result is a single number. Also called the inner product or the scalar product (a different operation from the scalar multiple). Texts also write it ( is the transpose; see the Matrix Operations page) or . The protagonist of this page | |
| angle | The angle between two vectors, between 0° and 180°. Textbooks often write it , but on this site is the parameter, so we write | |
| projection | Eq. (6). The shadow of dropped perpendicularly onto the line in the direction of | |
| , | cosine similarity, inverse cosine (arccos) | Eq. (7). The dot product divided by the product of the two lengths. Between and 1. turns a cosine value between and 1 back into an angle between 0° and 180°. It is not |
| example vectors | , , , . is the name of an example, not a weight |
School textbooks often write a vector as and its length as . They are the same things as this page's and , and the projection is also called the orthogonal projection.
figures/vector_geometry.pyThe dot product is a single number — multiply matching components and add them all
The dot product of two vectors multiplies the components in the same position and adds everything up. The result is not a vector but a single number.
is the symbol (sigma) that says to change from 1 to one at a time and add up the expression after it. Sigma notation has its own page, Sigma Notation.
. The 3-D pair works the same way: . For two vectors of different dimensions there are not enough components to pair up, so the dot product is not defined.
The dot product of a vector with itself adds up the squares of its components, so it is the square of the length in Eq. (3): . Likewise , so , and is also 3.
Swapping the order gives the same value, . Multiplying one side by multiplies the dot product by . The dot product of and is , twice .
Why the dot product is the product of two lengths and a cosine
Both vectors must be nonzero when referring to their angle.
The dot product equals the product of the two lengths times the cosine of the angle between them. For nonparallel vectors, it follows from computing a side length of the triangle formed by , , once with Eq. (4) and once with the law of cosines, and matching the two (folded derivation below).22 Many school textbooks take Eq. (5) as the definition of the dot product and teach Eq. (4) as its component form. Only the order differs; the value is the same.
The cosine is the horizontal coordinate of the point on a circle of radius 1 reached by turning from the horizontal axis. It equals the signed shadow of a stick of length 1 held at the angle to the horizontal axis (the lying stick). It is 1 at 0°, 0 at 90°, and at 180°, and in between it decreases as the angle grows (Figure 2).
Put in the from the lower panel of Figure 1 and you get . So . The inverse cosine, which turns a cosine value back into an angle, gives . NumPy's np.arccos returns the angle in radians (0.785), so to see it in degrees use np.degrees(np.arccos(x)).
Group Eq. (5) as and the shadow from the intuition appears. is the signed component of the shadow of dropped perpendicularly onto the line in the direction of . When the angle exceeds 90°, this value is negative, so the shadow falls on the opposite side.
Why Eq. (4) and Eq. (5) are equal — the law of cosines
Compute the length of the side , which joins the tips of the two vectors, in two ways. Expanding in components, Eq. (3) and Eq. (4) give the following.
For two nonparallel vectors, applying the law of cosines to the triangle with sides , , gives the following.
Matching the last terms of the two equations gives Eq. (5). Checking with : , so , and agrees.
For parallel nonzero vectors, write with . Their dot product is , and . The angle is 0° for and 180° for , so Eq. (5) also holds in this case.
Above 3 dimensions you cannot draw the angle, so it goes the other way: Eq. (5) defines the angle. Thanks to the Cauchy–Schwarz inequality , falls between and 1, so a single angle is determined (when neither vector is the zero vector). For the two pairs on this page, and .
A projection is a shadow found with the dot product
The direction vector must be nonzero. The length of the projection vector is , which is nonnegative and differs from its signed component.
The shadow of dropped perpendicularly onto the line in the direction of is called the projection (orthogonal projection). It is the signed component times the vector of length 1 in the direction of , , so the denominator is the squared length.
Set , so the projection is . The residual has dot product with the direction vector. Expanding gives , so the residual is always perpendicular to the projection direction.
Dropping onto , the multiplier is , so . This is the thick arrow in the lower panel of Figure 1. Its length is , the same as .
The remaining part, , has dot product with . A projection splits a vector into 'the part along another vector' and 'the part at right angles to it'. The dashed line and the small right-angle mark in the lower panel of Figure 1 are that second part.
What it means when the dot product is 0 or negative
For vectors that are not the zero vector the lengths are positive, so the sign of the dot product follows the sign of . It is positive when the angle is less than 90°, 0 at 90°, and negative when it is greater than 90° (Figure 2).
Two vectors whose dot product is 0 are called orthogonal. It means the same as perpendicular, or at right angles. and are orthogonal because .
and point in exactly opposite directions, so , and the dot product is . Figure 2 places the five pairs of this page on the cosine curve. halves every component of , so it has the same direction, and the angle 27.3° between and is , the dot product 8 divided by the product of the two lengths .
figures/cosine_curve.pyWhat does a dot product of 0 mean?
A dot product of 0 means the two vectors are orthogonal (perpendicular to each other, with an angle of 90°). The dot product is the product of the two lengths and , so if neither is the zero vector, , and between 0° and 180° the only such angle is 90°. and sit at the 90° spot in Figure 2. If one of them is the zero vector (all components 0), the angle is not defined but the dot product is 0, so by convention the zero vector is taken to be orthogonal to every vector.
Does a negative cosine similarity mean opposite?
No. A negative cosine similarity means the angle is greater than 90°; the vectors point in exactly opposite directions only when it is . and give (180° in Figure 2). 0 is not opposite but orthogonal (90°).
Why cosine similarity divides the dot product by the two lengths
Cosine similarity is the dot product divided by the product of the two lengths (when neither vector is the zero vector). By Eq. (5) the dot product is also proportional to the two lengths, so only the direction is left after dividing. Among vectors with a positive dot product (at less than 90° to each other), one in the same direction gives a small value if it is short, and one that matches less gives a large value if it is long. , in exactly the same direction as the query , has a dot product of 4.5, while the worse match has 16.
Figure 3 compares the query vector with three candidates. The dot products are 4.5, 8, 16 for (), , () in that order.
Measured by cosine similarity, the 1st place and the last place swap, and is the largest, at 1. and tie at 0.889, because Eq. (7) divides the dot product 16 by the product of the two lengths and removes the share of the added length.
figures/dot_vs_cosine.pyIf you first divide each vector by its own length so that it becomes a vector of length 1 (a unit vector), which is normalization, the dot product is the cosine similarity, because the denominator of Eq. (7) becomes 1. For example, has length 1.
When both vectors are nonzero, the dot product is the product of two lengths and a cosine, so it is also proportional to the lengths. Whether two directions are similar is measured by the cosine similarity, which divides by the two lengths.
What's the difference between cosine similarity and the dot product?
Cosine similarity is the dot product divided by the product of the two lengths (Eq. (7)), so it leaves out length and measures only direction. That is why it lies between and 1, and stays the same when you stretch or shrink one vector by a positive factor. Stretching to takes the dot product from 8 to 16, but the cosine similarity stays at 0.889. If both vectors already have length 1, the two are the same value.
What's the difference between cosine similarity and Euclidean distance?
Cosine similarity measures only direction, while the Euclidean distance also measures the difference in length. Seen from , and have the same cosine similarity, 0.889, but their distances differ, about 1.41 and 3.61. If both vectors are normalized to length 1, , so the order from nearest is the same as the order from the largest cosine similarity.
Common misconceptions about vectors and the dot product
- The result of a dot product is also a vector — Eq. (4) takes the element-wise products and adds them up again into a single number, 5. In NumPy
a * bis the element-wise product[3. 2.](a vector), and the dot product is the5.0ofa @ bornp.dot(a, b). - A large dot product means the two vectors are close — In Figure 3, , with the largest dot product (16), is the farthest of the three candidates from : , so the distance is . , in exactly the same direction, has the smallest dot product (4.5).
- It is a vector only if you can draw it as an arrow — By Eq. (1), any ordered list of numbers is a vector, so Eqs. (2)–(7) apply to a 768-dimensional embedding that cannot be drawn exactly as they do to the 2-D .
Does embedding search measure with the dot product or cosine similarity
With the similarity the search model was trained with. Often that is cosine similarity, and a model trained with the unnormalized dot product measures with the dot product as it is.
AI models turn words, sentences, and images into embedding vectors with hundreds to thousands of dimensions. A search for similar documents picks document vectors in order of their similarity to the query vector. If the vectors are normalized in advance, computing just the dot product gives the same order as cosine similarity.
There are also search models trained with the dot product of two unnormalized vectors as the similarity (DPR, Karpukhin et al. 2020). If you normalize the vectors of such a model and switch to cosine, the ranking can change, as in Figure 3.
Where vectors and the dot product show up in AI
Attention in a transformer uses the dot product of a query vector and a key vector as a score. It divides this score by and puts it through the softmax to turn it into weights that sum to 1. The softmax is covered on the Softmax page.
| Symbol | Name | Meaning |
|---|---|---|
| query, key, key dimension | The two vectors whose dot product attention takes, and their dimension. The subscript of is a conventional notation, not a component index |
divides the score of every pair by the same number, unlike cosine similarity, which divides each pair by its own two lengths. So in the attention scores of the original transformer (Vaswani et al.), the lengths of the vectors remain.
In training, too, the parameters and the gradient are vectors. One step of gradient descent is the operation of Eq. (2): multiply the gradient by the learning rate (a scalar multiple) and subtract it from the parameters (addition). The velocity of momentum is also a vector: some multiple of the previous velocity plus the gradient.
| This page | NumPy · PyTorch · SciPy |
|---|---|
| Vector | 1-D array np.array([3.0, 1.0]), torch.tensor([3.0, 1.0]). The shape is (2,) in NumPy and torch.Size([2]) in PyTorch |
| , (Eq. (2)) | x + y, s * x |
| Element-wise product | x * y — not the dot product. (x * y).sum() is the dot product |
| (Eq. (4)) | x @ y, np.dot(x, y), torch.dot(x, y) (torch.dot takes only two 1-D tensors) |
| (Eq. (3)) | np.linalg.norm(x), torch.linalg.vector_norm(x) — the default is the 2-norm of Eq. (3) |
| (Eq. (7)) | x @ y / (np.linalg.norm(x) * np.linalg.norm(y)), torch.cosine_similarity(x1, x2, dim=1, eps=1e-8) (x1 and x2 are the two tensors to compare, not components). The default of dim is 1, so for two 1-D vectors pass dim=0 (or dim=-1) |
scipy.spatial.distance.cosine(x, y) — despite its name, it returns not the similarity but , often called the 'cosine distance'. It is a different quantity from this page's distance ; for vectors of length 1, | |
| Normalization | torch.nn.functional.normalize(x, dim=0) (defaults p=2, dim=1) |
q @ k. Real models compute many scores with a single matrix product (Matrix Operations) |
What's the difference between np.dot, @, and *?
For two 1-D arrays, np.dot(a, b) and a @ b are the same dot product, and a * b is the element-wise product. With and the first two give 5.0. a * b gives the vector (3, 2), and you have to add it up with .sum() to get the dot product 5. For 2-D arrays (matrices), @ is the matrix product.
Check it in code
Click a symbol to highlight it in both the formula and the code. The first line of the output also shows that a * b is the element-wise product, not the dot product.
import numpy as np
a = np.array([3.0, 1.0]) # 𝐚
b = np.array([1.0, 2.0]) # 𝐛
inner = np.sum(a * b) # 식 (4): 성분끼리 곱해(a * b) 모두 더한다
norm_a = np.sqrt(np.sum(a * a)) # 식 (3): ‖a‖ = √(a·a)
norm_b = np.sqrt(np.sum(b * b))
proj = inner / np.sum(a * a) * a # 식 (6): ‖a‖² = a·a. b 를 a 위로 내린 그림자
cos_phi = inner / (norm_a * norm_b) # 식 (7)
print("a * b =", a * b, " <- 성분별 곱. 아직 벡터다")
print("a·b =", inner, " a @ b =", a @ b, " np.dot(a, b) =", np.dot(a, b))
print(f"‖a‖ = {norm_a:.3f} ‖b‖ = {norm_b:.3f}")
print(f"cos φ = {cos_phi:.3f} φ = {np.degrees(np.arccos(cos_phi)):.1f}°")
print("사영 =", proj, " (b - 사영)·a =", (b - proj) @ a)
u = np.array([1.0, 2.0, 2.0]) # 𝐮 (3차원)
w = np.array([2.0, 1.0, 2.0]) # 𝐰
print("u·w =", u @ w, " cos φ =", round(u @ w / (np.linalg.norm(u) * np.linalg.norm(w)), 3))$ python dot_product.py
a * b = [3. 2.] <- 성분별 곱. 아직 벡터다
a·b = 5.0 a @ b = 5.0 np.dot(a, b) = 5.0
‖a‖ = 3.162 ‖b‖ = 2.236
cos φ = 0.707 φ = 45.0°
사영 = [1.5 0.5] (b - 사영)·a = 0.0
u·w = 8.0 cos φ = 0.889Try it yourself
- Find the dot product of and , their two lengths, and . Roughly how many degrees is the angle, and compared with (27.3°), which one is closer in direction to ?
- Is orthogonal to ? What about ?
- This time, find the projection of onto with Eq. (6). What comes out, and why?
Answers
- The dot product is , and the lengths are and . So , and the angle is . The angle is smaller than the 27.3° of , so is closer in direction to . Its dot product, 14, is also larger than the 8 of , but that is partly because it is longer, with length 5.
- , so they are orthogonal. , so it is not orthogonal to .
- The multiplier is , so the projection is itself. This means that the foot of the perpendicular dropped from the tip of to the line along happens to be the tip of . Indeed, the remaining part has dot product with .
Further reading
- Deep Learning, Chapter 2 Linear Algebra — Goodfellow, Bengio, Courville, 2016. Section 2.1 defines a vector as an array of numbers, Section 2.5 (Norms) rewrites the dot product as the product of two norms and a cosine, and Section 2.6 defines a dot product of 0 as orthogonality. The section 'A vector is numbers lined up in order' and Eq. (5) rely on it.
- Introduction to Linear Algebra, Chapter 1 Introduction to Vectors — Gilbert Strang, 2016 (5th edition). The textbook of the MIT 18.06 course. Section 1.1 covers vector addition and scalar multiples, and Section 1.2 (Lengths and Dot Products) covers length, the dot product, the cosine formula, and the Schwarz inequality. Eqs. (2)–(5) and the folded derivation rely on it.
- Introduction to Information Retrieval, 6.3.1 Dot products — Manning, Raghavan, Schütze, 2008. It treats documents as vectors and explains that they are compared with cosine similarity to remove the effect of document length. The sections 'Why cosine similarity divides the dot product by the two lengths' and 'Does embedding search …' and the section 'Common misconceptions about vectors and the dot product' rely on it.
- Attention Is All You Need — Vaswani et al., 2017. Section 3.2.1 (Scaled Dot-Product Attention) defines attention, which divides the dot product of queries and keys by and puts it through the softmax. The section 'Where vectors and the dot product show up in AI' relies on it.
- Dense Passage Retrieval for Open-Domain Question Answering — Karpukhin et al., 2020. EMNLP. It turns questions and passages into vectors separately and trains a search model with the dot product of the two unnormalized vectors as the similarity. The section 'Does embedding search measure with the dot product or cosine similarity' relies on it.
Video notes
- 00:00 Opening question — "Does a large dot product mean two vectors are similar?"
- 00:40 Eqs. (1) and (2) — the vector as a list of numbers, the sum and the multiple by 2 in the upper panel of Figure 1
- 02:00 Write Eq. (4) on the board — , the result is a single number
- 03:20 Eq. (5) and the lower panel of Figure 1 — the angle 45°, the shadow
- 05:00 Figure 2 — the sign of the dot product, and orthogonality
- 06:10 Figure 3 and Eq. (7) — the 1st and last places swap between the dot product and cosine similarity; which one does search use
- 07:30 Code lab —
a * banda @ b· wrap-up
This is the filming plan. After upload, the timestamps will be replaced with the real ones.