Find the zeros of $h(z)=z^6-5z^4+3z^2-1$ within the unit disc - Verification

Your first approach with $f(z) = -5z^4$ and $g(z) = z^6 + 3z^2 - 1$ doesn't work, because $g(i) = -1 - 3 - 1 = - 5$, hence $|f(i)| = |g(i)|$

So you don't have a strict inequality on the boundary of the region. True that $|f(z)| >= |g(z)|$, but that's not enough for Rouche's theorem.

On the exam, your choice of functions does work for applying Rouche's theorem! Taking $f(z) = -5z^4 + 3z^2$ and $g(z) = z^6 -1$, we now have that $|f(z)|$ takes its minimum of $2$ when and only when $z^2 = 1$, but at those points $|g(z)|$ is also at its minimum of $0$. Meanwhile $|g(z)|$ takes its maximum at $2$, but this never happens when $z^2 = 1$. We conclude that $|f(z)| > |g(z)|$ everywhere.

And $f(z)$ does have all four of its roots in the unit circle $(0, \pm\sqrt\frac{3}{5})$