Is there no formula for $\cos(x^2)$?

There's a simpler reason: Every rational function in $\cos x, \sin x$ has (not necessarily minimal) period $2 \pi$, but $\cos(x^2)$ is not periodic.

Much easier: any rational function of $\sin$, $\cos$ will be periodic. Your function isn't.