how does the dot product determine similarity?

The dot product of two vectors $\mathbf{u}$ and $\mathbf{v}$ is defined as

$$\mathbf{u}\cdot\mathbf{v} = |\mathbf{u}|\,|\mathbf{v}|\cos \theta$$

It's perhaps easiest to visualize its use as a similarity measure when $|\mathbf{v}|=1$, as in the diagram below, where $\cos\theta = \mathbf{u}\cdot\mathbf{v}\,/\,|\mathbf{u}|\,|\mathbf{v}| = \mathbf{u}\cdot\mathbf{v}\,/\,|\mathbf{u}|$.

Here you can see that when $\theta=0$ and $\cos\theta=1$, i.e. the vectors are colinear, the dot product is the product of the magnitudes of the vectors. When $\theta$ is a right angle, and $\cos\theta=0$, i.e. the vectors are orthogonal, the dot product is $0$. In general $\cos\theta$ tells you the similarity in terms of the direction of the vectors (it is $-1$ when they point in opposite directions). This holds as the number of dimensions is increased, and $\cos\theta$ has important uses as a similarity measure in multi-dimensional space.

Here is the geometric intuition.

Dot product between $u$, $v$ can be interpreted as projecting $u$ onto $v$ (or vice-versa), and then taking product of projected length of $u$ ($|u|$) with length of $v$ ($|v|$).

When $u$ is orthogonal to $v$, projection of $u$ onto $v$ is a 0 length vector, yielding 0 product. If you visualize all possible rotations of $u$ while keeping $v$ fixed, the dot product gives:

• Zero value when $u$ is orthogonal to $v$ as the projection of $u$ onto $v$ yields a vector of zero length. This corresponds to the intuition of zero similarity.
• Largest value of $|u||v|$ when $u$ and $v$ point in the same direction.
• Lowest value of $-|u||v|$ when $u$ and $v$ point in opposite direction.

Dividing the $u \cdot v$ by max value of $|u||v|$ nicely limits the range to $[-1, 1]$, making it scale invariant.