Why doesn't my derivation of the arcsec derivative formula not work?

The sloppy reasoning occurs with the right triangle you wrote. You must be careful about the range (values) of the arcsec function. $y=\text{arcsec}(x)$ can lie either in $[0,\pi/2)$ or in $(\pi/2,\pi]$. (These correspond, respectively, to $x>0$ and $x<0$.)

$\tan(y)<0$ when $y\in (\pi/2,\pi]$, and so $\tan(y)=-\sqrt{x^2-1}$ in that event. Working with your formula for $y'$, we note that when $x<0$, $$\sec(y)\tan(y)= x(-\sqrt{x^2-1})= (-x)\sqrt{x^2-1} =|x|\sqrt{x^2-1},$$ and this explains the formula. To be honest, it's a bit sneaky to move the negative to the other term, but it allows us to write a single formula, rather than writing down cases. (There's no problem, of course, when $x>0$.)