How to find the vector formula for the bisector of given two vectors?

Here's a purely geometric argument.

By definition, the sum of two vectors is equal to the diagonal of the parallelogram spanned by the vectors.

Now, observe that the two vectors $|b|\vec{a}$ and $|a|\vec{b}$ have exactly the same length. Therefore the parallelogram they span is a rhombus. The result then follows from the fact that the diagonal of a rhombus bisects its angles.