How does the series $\sum_{n=1}^\infty \frac{(-1)^n \cos(n^2 x)}{n}$ behave?

The series diverges for $x=\frac{2\pi}8\approx 0.78$. Since the squares modulo 8 are 0,1,4,1,0,1,4,1, the series for this $x$ becomes $$ - \frac{\sqrt{1/2}}{1} - \frac1{2} - \frac{\sqrt{1/2}}{3} + \frac1{4} - \frac{\sqrt{1/2}}{5} - \frac1{6} - \frac{\sqrt{1/2}}{7} + \cdots$$

The terms with even denominator form an alternating series -- their $(-1)^n$ factor is always $1$ but the cosine alternates between $-1$ and $1$. So their sum converges. However, the terms with odd denominator all have the same sign because the $(-1)^n$ factor is always $-1$ and the cosine is always $\cos\frac{\pi}{4} = \sqrt{1/2}$. So they diverge logarithmically towards $-\infty$.

I would expect (without having checked) that a similar divergence will happen at many other rational multiples of $\pi$, such that your series diverges for a dense set of $x$s.

Note that logarithmic divergence is slow. Even for $1000$ terms, you shouldn't expect it to have reached more than $\log 1000\approx 7$ times the first term, so the $2.8$ maximum you've found numerically does not really point towards convergence (note that in the above case only half of the terms participate in the divergence).

The series is real part of $$ f(x) = \sum_{n=1}^{\infty} \frac{(-1)^n e^{in^2 x}}n=\sum_{n=1}^{\infty} \frac{e^{ i(2n)^2x}}{2n} - \sum_{n=1}^{\infty} \frac{e^{i(2n-1)^2x}}{2n-1}. $$ By the Theorem 2 of Lebeque, the series for $f$ converges for almost all $x\in [-\pi,\pi]$. This means that the set $$ D:=\{x\in[-\pi,\pi] : \sum_{n=1}^{\infty} \frac{(-1)^n e^{in^2 x}}n \ \ \textrm{diverges}\} $$ has a measure $0$.

Also, by the discussion in MathOverflow, this set $D$ is a measure-zero $G_{\delta\sigma}$-set.

Determining convergence or divergence for specific values of $x$ is not easy. As @Henning pointed out, $x=\pi/4 \in D$. Similarly, we can prove that $$\{\pi/4, \pi/2, 3\pi/4, -\pi, -3\pi/4, -\pi/2, -\pi/4\}\subset D.$$If we use finiteness of the irrationality measure of $\pi$, then we can prove the convergence for any $x\in\mathbb{Q}$, that is $$\mathbb{Q}\subset [-\pi,\pi]-D.$$ This is once discussed in @DavidSpeyer's answer in here.