On the convergence of $\sum_{n = 1}^\infty\frac{\sin\left(n^a\right)}{n^b}$

If $a > 0$ then the series converges if $b > \max(a,1-a)$.

The general principle is that the series $\sum_{n=1}^\infty f(n)$ and the integral $\int_1^\infty f(x) \, dx$ converge and diverge together if $\int_1^\infty |f'(x)| \, dx < \infty$. This is proved here.

In this case, $f(x) = \sin x^a /x^b$ and

$$\int_1^\infty |f'(x)| \, dx= \int_1^\infty \left| \frac{-b\sin x^a}{x^{b+1}} + \frac{a\cos x^a}{x^{b-a+1}} \right| \, dx \\ \leqslant \int_1^\infty \frac{b}{x^{b+1}} \, dx + \int_1^\infty \frac{a}{x^{b-a+1}} \, dx, $$

so the integrals on the RHS converge and theorem is applicable if $b > 0$ and $b > a$.

We have

$$\int_1^\infty \frac{\sin x^a}{x^b}\, dx = \frac{1}{a}\int_1^\infty \frac{\sin u}{u^{(b+a-1)/a}}\, du, $$

which converges by the Dirichlet test when $a > 0$ and $b+a-1 > 0 \implies b > 1 - a.$

If $b \leqslant 1 - a$ then the integral diverges. In this case we have an integral of the form $\int_1^\infty u^\alpha \sin u \, du$ where $\alpha \geqslant 0$. Divergence is obvious if $\alpha = 0$ since $\int_1^c \sin u \, du = \cos 1 - \cos c$.

For $\alpha > 0$ we have for any positive integer $k$,

$$\left|\int_{2k\pi}^{2(k+1)\pi} u^\alpha \sin u \, du \right| = \int_{2k\pi}^{2(k+1)\pi} u^\alpha \sin u \, du \geqslant (2 k \pi)^\alpha\int_{2k\pi}^{2(k+1)\pi} \sin u \, du = 2(2k\pi)^\alpha.$$

Since the RHS tends to $\infty$ as $k \to \infty$, the Cauchy criterion is violated and the improper integral must diverge.

Hence, the series diverges when $0 < a < b \leqslant 1 - a$.

The series diverges if $0<a<1$, $b>0$ and $a+b\leq 1$

This idea is already discussed in the link provided at $a=b=1/2$ case. Let $k$ be a positive integer. If $$ 2\pi k + \pi/6\leq n^a\leq 2\pi k + 5\pi/6, $$ then $\sin (n^a)\geq 1/2$.

The number of $n$'s in this interval is $\asymp k^{(1/a) -1}$. In the interval, we have $n^b\asymp k^{b/a}$. Thus, the contribution of $n$'s to the sum in this interval is $\asymp k^{(1/a)-(b/a)-1}$.

If $(1/a)-(b/a)-1 \geq 0$, then we have divergence. Note that the partial sum of the series fails to be Cauchy sequence.

The series converges if $0<a<1$ and $a+b>1$

The case particularly requires a good estimation of the sum by an integral. A crucial lemma for this case is Lemma 4.8 of The Theory of the Riemann Zeta-function written by Titchmarsh.

Let $f(x)$ be a real differentiable function in the interval $[a,b]$, let $f'(x)$ be monotonic, and let $|f'(x)|\leq \theta <1$. Then $$ \sum_{a<n\leq b}e^{2\pi i f(n)} = \int_a^b e^{2\pi i f(x)} \ dx + O(1). $$

Let $0<a<1$. We apply the above with $f(x)=\frac{x^a}{2\pi}$. The assumptions of the lemma are satisfied for $x\geq X_0>0$. Then we obtain

$$ \sum_{X_0<n\leq N} e^{in^a}=\int_{X_0}^N e^{ix^a} \ dx + O(1). $$

Apply change of variable $x^a=t$, then $$ \int_{X_0}^N e^{ix^a} \ dx = \int_{X_0^a}^{N^a} e^{it} \frac1a t^{\frac1a -1} \ dt. $$

Now, integration by parts shows that the above integral is $O(N^{1-a})$. Thus, we have for $t\geq 1$, $$ \sum_{n\leq t} \sin(n^a) = O(t^{1-a}). $$

Then the partial summation yields $$ \sum_{n=1}^N \frac{\sin(n^a)}{n^b} = O(N^{1-a-b})+b\int_{1-}^N \frac{\sum_{n\leq t} \sin(n^a)}{t^{b+1}} \ dt. $$

Hence, we have the convergence when $b>1-a$.

The series converges if $k\in\mathbb{Z}\cap [2,\infty)$, $k-1<a<k$, and $b>1-\frac k{2^k-2}+\frac a{2^k-2}$

We apply Theorems 5.11-5.13 of The Theory of the Riemann Zeta-function written by Titchmarsh. These theorems are proved by Van Der Corput's method. If the method is applied to integer $a$, we obtain Weyl's inequality, which was used in David Speyer's answer. The method is outlined in Chapter 5 of Titchmarsh's book.

Let $f(x)$ be real and have continuous derivatives up to $k$-th order, where $k\geq 2$. Let $\lambda_k\leq f^{(k)}(x)\leq h\lambda_k$ (or the same for $-f^{(k)}(x)$). Let $b-a\geq 1$, $K=2^{k-1}$. Then $$ \sum_{a<n\leq b} e^{2\pi i f(n)}=O(h^{\frac2K}(b-a)\lambda_k^{\frac1{2K-2}})+O((b-a)^{1-\frac2K}\lambda_k^{-\frac1{2K-2}}).$$

For the series in question, use $f(n)=\frac{n^a}{2\pi}$ over diadic intervals $(N,2N]$, and $\lambda_k\asymp N^{a-k}$. Then for $N\geq 1$, $$ \sum_{N<n\leq 2N} \sin(n^a)=O(N^{1-\frac{k-a}{2^k-2}}). $$ Here, the implied constant depends only on $a$. Then adding up the results for diadic intervals, we obtain for $t\geq 1$, $$ \sum_{n\leq t} \sin(n^a)=O(t^{1-\frac{k-a}{2^k-2}}). $$ Applying partial summation, we obtain $$ \sum_{n=1}^N \frac{\sin(n^a)}{n^b} = O(N^{1-\frac{k-a}{2^k-2}-b})+b\int_{1-}^N \frac{\sum_{n\le t} \sin(n^a)}{t^{b+1}}\ dt. $$ Then the convergence of the series follows if $b>1-\frac k{2^k-2}+\frac a{2^k-2}$.