Construct a nonabelian group of order 44

No element of $\mathbb Z_{10}$ has order four (why not?) and there is one element of order 2 ($5\in\mathbb Z_{10}$ under addition, $10\cong -1\in \mathbb Z_{11}^x$ under multiplication.), so our possibilities are quite limited. There are two groups of order $4$, and either will work as our $R$. To give a nonabelian group, we need to pick a nontrivial homomorphism, as you pointed out.

So at least one generator of $R$ has to map to our order-two element. In the case of the Klein four group, there appear to be three possibilities, but I claim that up to an isomorphism of the Klein four group, there is only one possibility.

Actually, this exhausts the possibilities for groups of order 44: we have two abelian groups, $\mathbb Z_{11}\times\mathbb{Z}_4$, $\mathbb Z_{11}\times \mathbb{Z}_2\times \mathbb Z_2$, and two nonabelian groups: $\mathbb{Z}_{11} \rtimes \mathbb Z_4 = \langle a,b \mid a^{11}, b^4, b^{-1}ab = a^{-1}\rangle$ and $\mathbb Z_{11}\rtimes(\mathbb Z_2 \times \mathbb Z_2) = \langle a, b, c \mid a^{11}, b^2, c^2, [b,c], b^{-1}ab = c^{-1}ac = a^{-1} \rangle$. I think the latter is $D_{22}$ (for 22-gon, not order of group), while the former has an element of order $4$.

You're on the right track (but NB your semidirect product is written in the wrong order). To analyze the possible maps $\gamma : P \to \operatorname{Aut}(\Bbb Z_{11}) \cong (\Bbb Z_{10}, +)$, we consider separately the cases $P \cong \Bbb Z_2 \times \Bbb Z_2$ and $P \cong \Bbb Z_4$.

In the case $P \cong \Bbb Z_4$, $P$ is generated by a single element, $[1]$, of order $4$, and so $$\operatorname{id}_{\Bbb Z_{11}} = \gamma([0]) = \gamma([1] + [1] + [1] + [1]) = \gamma([1])^4 .$$ So, $\gamma([1])$ has order dividing $4$, and the only such elements of $\operatorname{Aut}(\Bbb Z_{11}) \cong (\Bbb Z_{10}, +)$ are $\operatorname{id}_{\Bbb Z_{11}} \leftrightarrow [0]$ and $(x \mapsto -x) \leftrightarrow [5]$.

• If $\gamma([1]) = \operatorname{id}_{\Bbb Z_{11}}$, then $\gamma$ is the trivial homomorphism $\Bbb Z_4 \to \operatorname{Aut}(\Bbb Z_{11})$. This gives the direct product $G \cong \Bbb Z_{11} \times \Bbb Z_4 \cong \Bbb Z_{44} .$

• If $\gamma([1]) = (x \mapsto -x)$, then $\gamma([b])([c]) = (-1)^b [c]$, and the semidirect product $G = \Bbb Z_{11} \rtimes_{\gamma} \Bbb Z_4$ is defined by $$([a], [b]) \cdot ([c], [d]) = ([a] + (-1)^b [c], [b] + [d]) .$$ It's apparent from the multiplication rule that this group is nonabelian. The fact that the group is generated by $u := ([1], [0])$ and $v := ([0], [1])$ can be used to construct an explicit presentation of this group and to show that $G$ is the dicyclic group of order $44$.

One can analyze the case $P \cong \Bbb Z_2 \times \Bbb Z_2$ similarly, and this case gives rise to two more groups up to isomorphism, namely the abelian group $\Bbb Z_{11} \times \Bbb Z_2 \times \Bbb Z_2$ and the (nonabelian) dihedral group $D_{44} \cong D_{22} \times \Bbb Z_2$.