How prove this $\int_{0}^{\infty}\sin{x}\sin{\sqrt{x}}\,dx=\frac{\sqrt{\pi}}{2}\sin{\left(\frac{3\pi-1}{4}\right)}$

First make the substitution $x=u^2$ to get:

$\displaystyle \int _{0}^{\infty }\!\sin \left( x \right) \sin \left( \sqrt {x} \right) {dx}=\int _{0}^{\infty }\!2\,\sin \left( {u}^{2} \right) \sin \left( u \right) u{du}$,

$\displaystyle=-\int _{0}^{\infty }\!u\cos \left( u \left( u+1 \right) \right) {du}+ \int _{0}^{\infty }\!u\cos \left( u \left( u-1 \right) \right) {du}$,

and changing variable again in the second integral on the R.H.S such that $u\rightarrow u+1$ this becomes:

$=\displaystyle\int _{0}^{\infty }\!-u\cos \left( u \left( u+1 \right) \right) {du}+ \int _{-1}^{\infty }\!\left(u+1\right)\cos \left( u \left( u+1 \right) \right) {du} $,

$\displaystyle=\int _{0}^{\infty }\!\cos \left( u \left( u+1 \right) \right) {du}+ \int _{-1}^{0}\! \left( u+1 \right) \cos \left( u \left( u+1 \right) \right) {du} $.

Now we write $u=v-1/2$ and this becomes:

$\displaystyle\int _{1/2}^{\infty }\!\cos \left( {v}^{2}-1/4 \right) {dv}+\int _{-1/ 2}^{1/2}\! \left( v+1/2 \right) \cos \left( {v}^{2}-1/4 \right) {dv}=$

$\displaystyle \left\{\int _{0}^{\infty }\!\cos \left( {v}^{2}-1/4 \right) {dv}\right\}$

$\displaystyle +\left\{\int _{-1/2} ^{1/2}\!v\cos \left( {v}^{2}-1/4 \right) {dv}+\int _{-1/2}^{0}\!1/2\, \cos \left( {v}^{2}-1/4 \right) {dv}-1/2\,\int _{0}^{1/2}\!\cos \left( {v}^{2}-1/4 \right) {dv}\right\},$

but the second curly bracket is zero by symmetry and so:

$\displaystyle \int _{0}^{\infty }\!\sin \left( x \right) \sin \left( \sqrt {x} \right) {dx}=\displaystyle \int _{0}^{\infty }\!\cos \left( {v}^{2}-1/4 \right) {dv}$,

$\displaystyle =\int _{0}^{ \infty }\!\cos \left( {v}^{2} \right) {dv}\cos \left( 1/4 \right) + \int _{0}^{\infty }\!\sin \left( {v}^{2} \right) {dv}\sin \left( 1/4 \right) $.

We now quote the limit of Fresnel integrals:

$\displaystyle\int _{0}^{\infty }\!\cos \left( {v}^{2} \right) {dv}=\int _{0}^{ \infty }\!\sin \left( {v}^{2} \right) {dv}=\dfrac{\sqrt{2\pi}}{4}$,

to obtain:

$\displaystyle \int _{0}^{\infty }\!\sin \left( x \right) \sin \left( \sqrt {x} \right) {dx}=\dfrac{\sqrt{2\pi}}{4}\left(\cos\left(\dfrac{1}{4}\right)+\sin\left(\dfrac{1}{4}\right)\right)=\dfrac{\sqrt{\pi}}{2}\sin{\left(\dfrac{3\pi-1}{4}\right)}$.

Numerical calculation shows that the graph of

$$ y = \int_{0}^{x} \sin t \sin \sqrt{t} \, dt $$

is given by

Though a graph cannot constitute a proof, it strongly suggests that $y$ cannot converge as $x \to \infty$. Indeed, we can show that

$$ y(x) = -\cos x \sin \sqrt{x} + \frac{\sqrt{\pi}}{2} \sin \left( \frac{3\pi - 1}{4} \right) + o(1) $$

as $x \to \infty$. Thus in ordinary sense, the integral

$$ \int_{0}^{\infty} \sin x \sin \sqrt{x} \, dx $$

diverges. But if we understand the integral in Abel summation senas as follows

$$ \int_{0}^{\infty} \sin x \sin \sqrt{x} \, dx := \lim_{s \to 0^{+}} \int_{(0, \infty)} \sin x \sin \sqrt{x} \; e^{-sx} \, dx, $$

then the oscillating part vanishes and we obtain the identity. In fact, it converges to the proposed limit as weak as in Cesaro summation sense.

Now let us show that the integral converges in Able summation sense. By some calculation, we have

\begin{align*} \int_{0}^{\infty} \sin x \sin \sqrt{x} \; e^{-sx} \, dx &= \int_{0}^{\infty} 2x \sin x \sin (x^2) \; e^{-sx^{2}} \, dx \\ &= \int_{0}^{\infty} x \Re \left( e^{-i(x^2-x)} - e^{-i(x^2+x)} \right) e^{-sx^{2}} \, dx \\ &= \Re \int_{-\infty}^{\infty} x e^{-(s+i)x^2 + ix} \, dx. \end{align*}

By noting that

$$ z e^{-(s+i)z^2 + iz} = z \exp \left\{ -(s+i) \left( z - \tfrac{i}{2(s+i)} \right)^2 + \tfrac{i}{4(1-is)} \right\} $$

is an entire function with a nice vanishing speed as $ \left| \Re z \right| \to \infty$, we find that we can shift the contour of integration so that

\begin{align*} \int_{0}^{\infty} \sin x \sin \sqrt{x} \; e^{-sx} \, dx &= \Re \left[ e^{\frac{i}{4(1-is)}} \int_{-\infty}^{\infty} \left(x + \frac{i}{2(s+i)} \right) e^{-(s+i)x^2} \, dx \right] \\ &= \Re \left[ e^{\frac{i}{4(1-is)}} \frac{i}{2(s+i)} \int_{-\infty}^{\infty} e^{-(s+i)x^2} \, dx \right] \\ &= \Re \left[ e^{\frac{i}{4(1-is)}} \frac{i}{2(s+i)} \frac{\sqrt{\pi}}{\sqrt{s + i}} \right]. \end{align*}

Taking $s \to 0^{+}$, we find that in Abel summation sense,

\begin{align*} \int_{0}^{\infty} \sin x \sin \sqrt{x} \, dx &= \frac{\sqrt{\pi}}{2} \Re \left( e^{\frac{i-i\pi}{4}} \right) = \frac{\sqrt{\pi}}{2} \cos \left( \frac{1-\pi}{4} \right) = \frac{\sqrt{\pi}}{2} \sin \left( \frac{3\pi - 1}{4} \right) \end{align*}

as desired.

No, this integral is not convergent.

Consider $I(k):=\int_{0}^{\pi} \sin x\sin \sqrt{2k\pi +x}dx$. For $k$ large enough, $\sqrt{2k\pi +x}-\sqrt{2k\pi}, x\in [0,\pi]$ is small enough, so

$$\int_{0}^{\pi} \sin x\sin \sqrt{2k\pi +x}dx\sim \int_{0}^{\pi}\sin x \sin \sqrt{2k\pi }dx=2\sin \sqrt{2k\pi }\not\to 0.$$