Show that $(2,0,4) , (4,1,-1) , (6,7,7)$ form a right triangle

It's enough to show the following: $$(2,1,-5)\cdot(4,7,3)=0$$ and we are done!

The distances satisfy the Pythagorean theorem.

$d(A, B) = \sqrt{2^2 + 1^2 + 5^2} = \sqrt{30}$

$d(A, C) = \sqrt{4^2 + 7^2 + 3^2} = \sqrt{74}$

$d(B, C) = \sqrt{2^2 + 6^2 + 8^2} = \sqrt{104}$

And indeed:

$\sqrt{30}^2 + \sqrt{74}^2 = \sqrt{104}^2$

Therefore it is a right triangle (link).

To find $\alpha_3=\angle ABC$, you should consider $\overrightarrow{BA}\cdot \overrightarrow{BC}$ instead of $\overrightarrow{AB}\cdot \overrightarrow{BC}$. That's why you have a negative cosine and obtained the supplementary angle of the correct answer.