Why is the answer different with energy conservation vs forces?

First, I am assuming that there is no kinetic friction acting on the insect as it moves up the bowl. If kinetic friction were involved, you would have energy dissipation, but I will not consider that here.

Your mistake is in assuming that the static friction force is equal to its maximum value during the entire process. $\mu N$ only determines the maximum magnitude the static friction force can have before slipping occurs; it doesn't always hold for the static friction force magnitude. Before slipping, the static friction force is just equal to the force needed to prevent slipping, i.e. $mg\sin\theta$.

Doing this correctly, you will then see that the integral will give you a true expression, but it won't help you find where the ant slips because the integral is true for any angle $\alpha$ before slipping occurs, and the integral doesn't tell you anything about when the static friction force fails. i.e. energy conservation doesn't apply only when slipping occurs, so energy conservation won't help you solve this problem.

Also, technically the static friction force can't do work because the point of contact between the ant and the bowl doesn't move as the force is being applied, but that point isn't important here, as the (correct) integral will still give the work done by the insect's legs on the rest of the insect, even if the physical interpretation isn't correct.

The difference in energy between the two static equilibrium positions may only be some potential energy difference. You may assume the friction force is $F=\mu N$ during sliding, where $\mu$ is the kinetic friction coefficient (taken equal to the static friction coefficient) but since this force is non conservative, the work done this force will not account for any potential energy change, instead, it's lost. The balance in energy between the two positions will thus only tell you that the change in potential energy is the work of the weight force, which is not helpful for the determination of $\alpha$.