How find the minimum of $ab+\frac{1}{a^2}+\frac{1}{b^2}$

Here care must be taken while using AM-GM so that the constraint and equality condition is not violated. Hence rewrite the objective as:

$$\left(\frac12 ab+ \frac12 ab+\frac{1}{32a^2}+\frac{1}{32b^2}\right)+ \frac{31}{32}\left( \frac1{a^2}+\frac1{b^2}\right)$$ The first part is a sum of four terms with constant product. Hence this achieves minimum when each term is the same. Hence for this part, $$\implies 16ab^3=16a^3b=1 \implies a=b=\tfrac12$$

So for the first part, the minimum is $\frac12$. For the second part, we have $$\frac1{a^2}+\frac1{b^2} = \left(\frac1a-\frac1b\right)^2+\frac2{a^2b^2} \ge 0+ \frac8{(a+b)^2} \ge 8$$ which is also achieved when $a=b=\frac12$.

Hence the minimum of the objective within the constraints is achieved when $a=b=\frac12$, and the value is $\frac{33}4$.

By AM-GM $$16ab+16ab+a^{-2}+b^{-2}\ge16$$ But $$2\sqrt{ab}\le a+b\le 1\implies ab\le\frac14\implies -31ab\ge-\frac{31}4$$ Summing we get, with equality iff $a=b=\frac12$ $$S\ge\frac{33}4$$