Problem with definite integral $\int_{0}^{\frac{\pi}{6}}\cos x\sqrt{1-2\sin x} dx$

Well, @Nilan has the better way to go. But, here is another "Brute Force," double substitution method that works.

Write, $\sin x = \sqrt{1-\cos^2 x}$ (the positive square root is appropriate here since $0\le x\le \frac{\pi}{6}$. Then, with $u = \cos x$, $du = -\sin x dx=-\sqrt{1-u^2}du$, and the limits of integration extend from $u=1$ to $u=\frac{\sqrt{3}}{2}$. Thus,

$$\int_0^{\frac{\pi}{6}}\cos x\sqrt{1-2\sin x}dx= \int_{\frac{\sqrt{3}}{2}}^1 \frac{u\sqrt{1-2\sqrt{1-u^2}}}{\sqrt{1-u^2}}du$$

Now, we make a second substitution. Let $y=\sqrt{1-u^2}$. Then, $dy=\frac{-2u}{\sqrt{1-u^2}}du$ and the limits of integration go from $y=\frac12$ to $y=0$. (Note: This second substitution is identical to making the original substitution $u=\sin x$). Thus,

$$\int_0^{\frac{\pi}{6}}\cos x\sqrt{1-2\sin x}dx=\int_0^{\frac12} \sqrt{1-2y}dy=\frac13$$

Consider the integral \begin{align} \int_{0}^{\frac{\pi}{6}}\cos x\sqrt{1-2\sin x} dx \end{align} with the substitution $u = \cos(x)$. Making the desired substitution the integral becomes \begin{align} I = \int_{\sqrt{3}/2}^{1} u \sqrt{ \frac{1 - 2 \sqrt{1-u^{2}}}{1 - u^{2} } } du \end{align} Now make the substitution $t = 1 - u^{2}$ to obtain the integral \begin{align} I = \int_{0}^{1/2} \sqrt{1 - 2t} \, dt. \end{align} Making one last change of $y = 1-2t$ which leads to \begin{align} I = \frac{1}{2} \int_{0}^{1} \sqrt{y} \, dy = \frac{1}{3}. \end{align}