What are the subcategories of ordered sets / groups?

Interesting questions. Let start be saying that your observations on the subcategories of groups are correct: if you exclude the empty category all the subcategories of a group are exactly the submonoids of the group (by the way there is a subcategory that is full: the group itself).

About the posets, a subcategory of a poset-category $P$ is a poset $Q$ such that as sets $Q \subseteq P$ (that means that $\text{Ob}(Q) \subseteq \text{Ob}(P)$) and for every pair of elements $x,y \in Q$ if $x \leq_Q y$ then $x \leq_P y$.

That doesn't mean that $Q$ is a subposet of $P$ in the classical sense: usually by a subposet of $P$ one would mean a subset $Q \subseteq P$ with the ordering $$x \leq_Q y \iff x \leq_P y\ .$$ These subposets correspond exactly to the full-subcategories of $P$.

Finally about the existance of the functor $Z \colon \mathbf{Grp} \to \mathbf{Grp}$, it will surprise you but the answer is no, such functor does not exists. Here is a proof by contradiction. Assume that such a functor $Z$ does exists, then consider the following group-homomorphisms $$\mathbb Z/2\mathbb Z \stackrel{f}{\longrightarrow} S_3 \stackrel{g}{\longrightarrow} \mathbb Z/2 \mathbb Z$$ where $f(i)=(1,2)^i$ and $g$ is the projection of $S_3$ onto $S_3/A_3\cong \mathbb Z/2\mathbb Z$. You can easily see that $g \circ f=1_{\mathbb Z/2\mathbb Z}$ so we should have $$Z(g)\circ Z(f)=Z(g \circ f)=Z(1_{\mathbb Z/2\mathbb Z})=1_{Z(\mathbb Z/2\mathbb Z)}=1_{\mathbb Z/2\mathbb Z}$$ on the other hand we have the situation shown in the following diagram $$Z(\mathbb Z/2\mathbb Z) \stackrel{Z(f)}{\longrightarrow} Z(S_3)=0 \stackrel{Z(g)}{\longrightarrow} Z(\mathbb Z/2\mathbb Z)$$ so $Z(g)\circ Z(f)$ should be the zero-morphism from $\mathbb Z/2\mathbb Z$ in itself, which is not the identity, hence we arrived to a contradiction. We have to conclude that the functor $Z$ cannot exist.

Hope this help.

Regarding the second question, the center crops up in a different way:

If $G$ is a group viewed as a category and $1_G$ is the identity functor, then $Z(G)$ is the group of natural transformations $1_G \to 1_G$.

It can be useful to extend this to more general categories: for an arbitrary category $\mathbf{C}$, we can define $Z(\mathbf{C}) = \hom(1_\mathbf{C}, 1_\mathbf{C})$.

As a neat example, for any group $G$, if $G{-}\mathbf{Set}$ is the category of left $G$ actions on sets, then $Z(G{-}\mathbf{Set}) \cong Z(G)$.