Estimate $\sum_{n = 0}^N \cos (\alpha n^2)$

(Incomplete answer)

a) $\alpha\ll N^{-1}$

$$\begin{aligned}S&=\sum_{n=0}^N\cos(\alpha n^2)\\ &=\int_{-1/2}^{N-1/2}\cos(\alpha x^2)dx+\sum_{n=0}^{N-1}\int_{n-1/2}^{n+1/2}(\cos(\alpha x^2)-\cos(\alpha n^2))dx\\ &=\sqrt{\frac{\pi }{2\alpha}} C\left( (N-1/2) \sqrt{\frac{2}{\pi \alpha}}\right)-\sqrt{\frac{\pi }{2\alpha}} C\left( -1/2 \sqrt{\frac{2}{\pi \alpha}}\right)+r_\alpha(N) \end{aligned}$$where $C$ denotes the Fresnel C function.
Now, we estimate $r_\alpha(N)$.
$$\begin{aligned}\left|\int_{n-1/2}^{n+1/2}(\cos(\alpha x^2)-\cos(\alpha n^2))dx\right|&=\left|\int_{n-1/2}^{n+1/2}2\sin\frac{\alpha(x+n)(x-n)}{2}\sin\frac{\alpha(x^2+n^2)}2dx\right|\\ &\le\left|\int_{n-1/2}^{n+1/2}\alpha(x+n)(x-n)dx\right|\\ &=\alpha/12\end{aligned}$$ Sum them together, we get $|r_\alpha(N)|\le\alpha N/12\ll 1$.

b) $\alpha\not\ll N^{-1}$

I strongly believe that there is no good estimation as $\cos(\alpha x^2)$ starts to oscillate extremely quick when $\alpha x^2\gg 1$ for almost all $\alpha\in\mathbb R$. $\cos(\alpha n^2)$ starts to have "randomness". This MO link gives more information.