Intuition behind how the Cauchy-Schwarz inequality's proof was obtained

I don't know about anybody else, but I share your dissatisfaction with the standard slick proof, and I personally find it helpful to think instead of expressing $x$ as a sum of a multiple of $y$ and a vector orthogonal to $y$. This kind of resolution of a vector into two mutually orthogonal components is a common and natural operation.

If $\lambda$ is real, then $x - \lambda y$ is orthogonal to $y$ if and only if (in your notation) $(x - \lambda y) \cdot y = 0$, i.e., $$ \lambda \|y\|^2 = x \cdot y. $$

For any value of $\lambda$ satisfying that condition ($\lambda$ may be chosen arbitrarily if $y = 0$, and there is a unique solution for $\lambda$ if $y \ne 0$), write $u = x - \lambda y$ and $v = \lambda y$, so that $x = u + v$ and $u \cdot v = 0$. Then: \begin{align*} \|x\|^2 & = (u + v) \cdot (u + v) \\ & = u \cdot u + 2u \cdot v + v \cdot v \\ & = \|u\|^2 + \|v\|^2 \\ & \geqslant \|v\|^2. \end{align*} Therefore, using the definitions of $v$ and $\lambda$: $$ \|x\|^2\|y\|^2 \geqslant \|v\|^2\|y\|^2 = \lambda^2\|y\|^4 = (x \cdot y)^2 = |x \cdot y|^2, $$ and the result follows. So the selection of the value $-\lambda$ for $\alpha$ does make some intuitive sense (to me, at least).

You could arrive at this value of $\alpha$ less intuitively by "completing the square" in the expression you derived for $\|x + \alpha y\|^2$, thus, multiplying by $\|y\|^2$, to avoid a possible division by zero: \begin{align*} \|x + \alpha y\|^2\|y\|^2 & = \|x\|^2\|y\|^2 + 2(x \cdot y)\alpha\|y\|^2 + \alpha^2\|y\|^4 \\ & = (\alpha\|y\|^2 + x \cdot y)^2 + \|x\|^2\|y\|^2 - (x \cdot y)^2 \\ & = \|x\|^2\|y\|^2 - (x \cdot y)^2, \end{align*} if $$\alpha\|y\|^2 + x \cdot y = 0. $$ So the proof you quoted can be seen as the proof by resolution into orthogonal components in heavy disguise.

Proving theorems about general vector spaces, or general inner product spaces, can begin by considering a familiar $2$- or $3$-dimensional space. But then you need to abstract the intuition so it's pure algebra, no diagrams required. So your question comes down to what sort of preamble may have helped here.

If you think about vectors in a space you can visualize, all the theorem says is that the angle $\theta$ between two vectors satisfies $-1\le\cos\theta\le 1$, which by the cosine rule is equivalent to the triangle inequality. Since the cosine rule can be stated in terms of dot products, it makes sense to see what you learn from one more equivalent result, $\Vert x-y\Vert^2\ge 0$.

But $\Vert x-\alpha y\Vert^2\ge 0$ is a natural generalisation, and connects the issue to extremising quadratics, with the extremum giving us the most inequality we can get. And we don't need to think about a specific vector space to use $\Vert v\Vert^2\ge 0$, so it's a general starting point.