Square root of the square of the cosine: absolute value or not

The function $(b-a)\sin^2 u$ is not one-to-one. So in order to make the substitution $x-a=(b-a)\sin^2 u$ you have to restrict the domain of $\sin^2 u$: otherwise, you won't know which value of $u$ corresponds to a given $x$.

The most natural choice of domain where $(b-a)\sin^2 u$ is one-to-one is the interval $\left[0,\frac{\pi}{2}\right]$. Since $(b-a)\sin^2 u$ attains its full range for $u$ in this domain, there is no harm in restricting $u$ to this domain when making the substitution.

But on this interval, $\sin u \cos u$ is nonnegative, because $\sin u$ and $\cos u$ are individually non-negative. So it is genuinely true that $\sqrt{\sin^2 u \cos^2 u}=\sin u \cos u$, without absolute values.

Note that in the second integral, if you are using real numbers, then $a\le x\le b$ or $b\le x\le a$ (otherwise the argument of the square root is negative). The the substitution you are making implies $0\le u\le \pi/2$, so both $\sin$ and $\cos$ have positive values.