Which recursively-defined predicates can be expressed in Presburger Arithmetic?
Presburger arithmetic admits elimination of quantifiers, if one expands the language to include truncated minus and the unary relations for divisibility-by-2, divisibility-by-3 and so on, which are definable in Presburger arithmetic. (One can equivalently expand the language to include congruence $\equiv_k$ modulo $k$ for each natural number $k$.) That is to say, every assertion in the language of Presburger arithmetic is equivalent to a quantifier-free assertion in the expanded language.
It follows that the definable subsets of $\mathbb{N}$ in the language of Presburger arithmetic are exactly the eventually periodic sets. These are comparatively trivial sets, of course, and it means that the set of prime numbers and other interesting sets of natural numbers are simply not expressible in the language of Presburger arithmetic. A similar analysis holds in higher dimensions, and for this reason, we usually think of Presburger arithmetic as a weak theory.
The quantifier elimination argument leads directly to the conclusion that Presburger arithmetic is a decidable theory: given any sentence, one finds the quantifier-free equivalent formulation, and such sentences are easily recognized as true or false.
There is an interesting account in these slides for a talk by Cesare Tinelli.
Presburger Arithmetic (over the naturals) describes exactly the semi-linear sets. A set ($\subseteq \mathbb{N}^k$) is linear if it can be expressed as $$L(\vec c, P=\{\vec{p_1}, \ldots, \vec{p_n}\}) = \{ \vec c + \sum_{i=1}^pc_i\vec{p_i} \;|\; c_i \in \mathbb{N}\}.$$
A set is semi-linear if it can be expressed as a finite union of linear sets. One can deduce a few techniques from this characterization (due to Ginsburg & Spanier).
In addition to the answers above, it is worth mentioning that the existential fragment of Presburger arithmetic can actually be extended by a full divisibility predicate while retaining decidability [1]. Satisfiability in the resulting fragment is in NEXP, the lower bound being NP [2].
Also, it is possible to define a family of formulas $\Phi_n(x,m)$ such that for every natural number $m$ representable using $n$-bits, $\Phi_n(x,m)$ holds if and only if $x \equiv 0 \bmod m$ [3].
[1] L. Lipshitz, “The Diophantine problem for addition and divisibility,” Transactions of the American Mathematical Society, vol. 235, pp. 271– 283, 1976.
[2] Lechner, A.; Ouaknine, J.; Worrell, J., "On the Complexity of Linear Arithmetic with Divisibility," in Proceedings of the 30th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS), pp. 667-676, 2015.
[3] Haase, C., "Subclasses of Presburger arithmetic and the weak EXP hierarchy," in Proceedings of the Joint Meeting of the Twenty-Third EACSL Annual Conference on Computer Science Logic (CSL) and the Twenty-Ninth Annual ACM/IEEE Symposium on Logic in Computer Science (LICS) (CSL-LICS '14). ACM, New York, NY, USA, Article 47, 2014.