How to solve $\sin^2(x)+\sin2x+2\cos^2(x)=0$

What you've done here is actually perfect! We know that the only way for this expression to be $0$ is if

$$\sin(x)+\cos(x)=0$$ and $$\cos^2(x)=0\to\cos(x)=0$$

We know both can't be possible because that would imply $\sin(x)=\cos(x)=0$, which is false. So the solution of given equation does not exist.

Rushabh's answer is perfect, though I would do it slightly differently. Note that $\sin 2x = 2\sin x\cos x$, so your equation can be rewritten as $$ \sin^2 x + 2\sin x\cos x + 2\cos^2 x = 0.$$ If you treat $\sin x$ and $\cos x$ as independent variables $u$ and $v$, this becomes the quadratic equation $$ u^2 + 2uv + 2v^2 = 0, $$ where it becomes clear there are no solutions, as the discriminant is $\triangle = 2^2-4\times 2 = -4<0$.

From $$(\sin x + \cos x )^2 + \cos ^2x =0$$ we get $$(\sin x + \cos x )^2=0$$ and $$\cos ^2x =0$$

The first identity implies $$\tan x =-1$$ and the second identity implies $$\cos x=0$$ which makes $\tan x = \pm \infty $.

Thus there is no solution which satisfies both identities, that is your equation has no solution.