Polynomial maps between noncommutative groups
Polynomial mappings of groups were investigated by Leibman in connection with density Ramsey theory.
Definition. Let $G$ be a group and $f:\mathbb{Z}\to G$ any map. The discrete derivative of $f$ is the map $D_a f(b) = f(b+a) f(b)^{-1}$. The map is called polynomial of degree $d$ if all its $d+1$-th discrete derivatives vanish identically (i.e. equal $1_G$).
One could replace $\mathbb{Z}$ by a more general group here; I stick to the integers for simplicity.
Theorem Assume that $G$ is nilpotent. Then the polynomial mappings form a group under pointwise operations.
Different proofs can be found in Leibman's article, an appendix in an article by Green, Tao, and Ziegler and in an article that I wrote. The appendix contains probably the most systematic approach based on Host-Kra cube groups, in particular it is shown that the composition of polynomial maps between nilpotent groups is also polynomial.
Note that the theorem does not say that the polynomial mappings of a given degree form a group, to recover a result of this kind one has to refine the notion of degree, see the references above.
The theorem fails already for the dihedral group $D_3$ that is the smallest non-nilpotent group. Indeed, let δ be a rotation and σ a reflection in $D_3$ so that δ³ = σ² = (σδ)² = 1. The sequences (…, σ, 1, σ, 1,…) and (…, σδ, 1, σδ, 1,…) are polynomial (of degree 1) but their pointwise product (…, δ, 1, δ, 1,…) is not polynomial (of any degree).
I am not sure whether this presents an obstruction to polynomiality of composition of polynomial maps between arbitrary groups.
See a paper of Anokhin. He considered only the case where $H$ is abelian (and $G$ is arbitrary) and his definition of degree is slightly different. For instance, he says that the degree of $f$ is at most one if $$ f(x)-f(xy)+f(xyz)-f(xz)=0 $$ for all $x,y,z\in G$ (see Lemma 2).
In that paper, it was proved, e.g., that
all functions from a nontrivial group $G$ to a nontrivial abelian group $H$ are polynomial iff, for some prime $p$, $G$ is a finite $p$-group and $H$ is an abelian $p$-group (Theorem 2).