Why is my alternate method of calculating scalar products not working?

In the first method you assume at the end that the angle between $\vec{(a+2b)}$ and $\vec{(2a-b)}$ is $60^\circ$. If you were going to do this approach you would need the angle between them instead of $\cos 60^\circ$ for the last line.

I think it is failed because here

$(\vec{a} + 2\vec{b}) \cdot (2\vec{a} - \vec{b}) = |\vec{a} + 2\vec{b}| \cdot |2\vec{a} - \vec{b}| \cdot \cos60^{\circ} = 5 \cdot 2\sqrt{7} \cdot \frac{1}{2} = 5\sqrt{7}$

you are assuming that the angle between vectors $(\vec{a} + 2\vec{b})$ and $(2\vec{a} - \vec{b})$ is $60^\circ$ but it may not be (and apparently it is not).

Note $$2\vec {b}\cdot 2\vec {b}=4b^2$$ Compare this with what you have in your first approach when calculating $$ |\vec {a}+2\vec {b}| $$