Understanding precisely the dot product...

Sometimes to understand a mathematical concept it is helpful to start at the end of the calculation and work your way back to the beginning. The end in this case is the fact that when two vectors are perpendicular, their dot product is zero. Try to convince yourself that both formulas you gave in your question have this property.

For example, when you have a pair of unit vectors $(a,b)$ and $(c,d)$, the first formula will give $ac+bd$ which will be zero if $b=-c$ and $d=a$ for example, correspoding to the known fact that vectors $(x,y)$ and $(-y,x)$ in the plane are perpendicular.

Two arbitrary unit vectors can be written in the form $(\cos \alpha, \sin\alpha)$ and $(\cos\beta,\sin\beta)$. Then the relation you mentioned follows from the addition (or more precisely subtraction) formula for cosine.

If you are familiar with polar form of complex numbers there is a more elegant calculation that one can write down.

The intuitive background is that a 2-dimensional subspace of $R^n$ (with $n\geq2$) should be geometrically identical to $R^2$. The distance from $x=(x_1,...,x_n)$ to $y=(y_1,...,y_n)$ is $\|x-y\|=(\sum_{i=1}^n (x_i-y_i)^2)^{1/2}$. Consider a triangle with vertices at $x, y$, and the origin $0$, with $x\ne 0\ne y $ and $x\ne k y$ for any real $k$. The lengths of the sides of the triangle are $\|x\|, \|y\|,$ and $\|x-y\|$. The cosine triangle law will show that the angle $\theta=$"x0y" satisfies $\cos \theta= \frac {x\cdot y}{\|x\|\|y\|}$ where $x\cdot y $ is the value in Def'n (1).