Isomorphism of $\mathbb{Z}\ltimes_A \mathbb{Z}^m$ and $\mathbb{Z}\ltimes_B \mathbb{Z}^m$

$\newcommand{\IZ}{\mathbb{Z}}$ One can easily verify that $G_A' = \{0\}\times \operatorname{im}(A-1_{m\times m})$. Moreover $G_A$ acts on $G_A'$ by conjugation. The elements of $\IZ^m$ act trivially and the extra $\IZ$ acts by multiplication with $A$. The normal subgroup $K_A:=\operatorname{ord}(A)\IZ \times \IZ^m$ is the kernel of this action, i.e. the subgroup of all elements that act trivially on $G_A'$.

Therefore any isomorphism $G_A \to G_B$ must map $K_A$ to $K_B$. In particular $ord(A)=|G_A/K_A| = |G_B/K_B|=\operatorname{ord}(B)$, let's call that $n$, and $G_A/K_A \cong G_B/K_B \cong \IZ/n\IZ$.

Now consider the conjugation action of $G_A$ on $K_A$ instead of $G_A'$. Since $K_A$ is abelian, this is really an action of $G_A/K_A\cong \IZ/n\IZ$ on $K_A\cong \IZ \times\IZ^m$ given by multiplication with the block matrix $A':=\begin{pmatrix}1&\\&A\end{pmatrix}$.

By considering the induced action on $K_A \otimes \mathbb{Q}$, we find that the two $\mathbb{Q}[\IZ/n]$-modules $K_A \otimes \mathbb{Q}$ and $K_B\otimes \mathbb{Q}$ must be isomorphic. That means that $A'$ and $B'$ are $\mathrm{GL}_{1+m}(\mathbb{Q})$-conjugated at the very least. I'm not sure how one would go from there.

I believe now that David Speyer's example can be adapted to provide a counterexample to the original question. (So I retract my earlier comment on the question and will delete it soon.)

In David's example, $A$ is a degree $\phi(m)$ matrix of order $m$ defining the action by multiplication of $\zeta_m$ on the ideal $I$ of the number field ${\mathbb Q}[\zeta_m]$, and $B$ is the action on the ideal $\sigma(I)$, and $A$ and $B$ are not conjugate to each other or to their inverses in ${\rm GL}_{\phi(m)}({\mathbb Z})$. A specific example is $m=37$, $\phi(m)=36$.

We define degree $n:=\phi(m)+1$ matrices $A'$ and $B'$ as the diagonal joins of $A$ and $B$ with the identity matrix $I_1$. So the corresponding ${\mathbb Z}$-modules can be thought of as $I \oplus \langle y \rangle$ and $\sigma(I) \oplus \langle z \rangle$, with trivial action on the second factors. These modules cannot be isomorphic, because an isomorphism would have to map the fixed points submodule $\langle y \rangle$ onto $\langle z \rangle$ and then their quotients $I$ and $\sigma(I)$ would be isomorphic, which they are not. So $A'$ and $B'$ are not conjugate in ${\rm GL}_{n}({\mathbb Z})$.

I claim (at least in some cases) that we can choose $A$ and $B$ such that the corresponding semidirect products $\langle \alpha \rangle \ltimes_{A'} {\mathbb Z}^n$ and $\langle \beta \rangle \ltimes_{B'} {\mathbb Z}^n$ are isomorphic, where $\alpha$ and $\beta$ generate infinite cyclic groups. We can (in some cases?) choose $A = B^a$ with $a$ coprime to $m$ and $2 \le a < \phi(m)-1$ such that $B$ is not conjugate in ${\rm GL}_{\phi(m)}({\mathbb Z})$ to $A$ or to $A^{-1}$, and choose integers $r,s$ with $ra-sm=1$.

Then we can define a isomorphism from $\langle \alpha \rangle \ltimes_{A'} {\mathbb Z}^n$ to $\langle \beta \rangle \ltimes_{B'} {\mathbb Z}^n$ by mapping $I$ to $\sigma(I)$ as in David's example, $y$ to $\beta^m z^r$ and $\alpha$ to $\beta^a z^s$. Note that this induces an isomorphism from the free abelian group $\langle \alpha, y \rangle$ to $\langle \beta, z \rangle$, such that the image of $y$ centralizes $\sigma(I)$.

I did some calculations in Magma in the case $m=37$, and found a degree 36 integer matrix $A$ that is not conjugate to $A^a$ for any $a$ with $2 \le a \le 36$.

For completeness, here is the matrix $A$ in machine readable format. I used the Magma function $\mathsf{AreGLConjugate}$ to test $A$ for conjugacy with $A^i$. This uses a fairly new algorithm published in Bettina Eick, Tommy Hofmann, and E.A. O'Brien. The conjugacy problem in ${\rm GL}(n,{\mathbb Z})$. J. London Math. Soc., 2019.

[
 [-4,149,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-1,4,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-4,133,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-2,64,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-2,42,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-4,130,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-3,76,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-4,143,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-1,24,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-2,53,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-3,86,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-3,103,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-1,35,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-1,9,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-4,113,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-4,144,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-1,20,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-2,69,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-1,22,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-2,61,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-2,54,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-3,82,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0],
 [-4,119,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0],
 [-4,120,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0],
 [-4,116,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0],
 [-4,132,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0],
 [-2,68,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0],
 [-1,26,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0],
 [-2,45,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0],
 [-4,118,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0],
 [-4,124,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0],
 [-3,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0],
 [-2,47,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0],
 [-3,110,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0],
 [-1,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1],
 [15,120,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
        -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
]

$\def\QQ{\mathbb{Q}}\def\ZZ{\mathbb{Z}}$I misread the question as asking about $C_m \ltimes_A \ZZ^n$ and $C_m \ltimes_B \ZZ^n$, where $m$ is the order of $A$ and $B$. If we work with $\ZZ \ltimes_A \ZZ^n$ and $\ZZ \ltimes_B \ZZ^n$, I'm not sure what happens.

Working with $C_m \ltimes_A \ZZ^n$, this is not true. Let $m$ be the order of $A$ and $B$, let $\zeta_m$ be a primitive $m$-th root of unity, let $K$ be the cylotomic field $\QQ(\zeta_m)$. Let $G$ be the Galois group of $K$ over $\QQ$, so $G \cong (\ZZ/m \ZZ)^{\times}$. Let $H$ be the class group of $K$. Suppose that $H$ contains a class $h$ whose $G$-orbit is larger than $h^{\pm 1}$; say $\sigma(h) \neq h^{\pm 1}$.

Let $I$ be an ideal representing the class $h$, so $I$ is a free $\ZZ$-module of rank $\phi(m)$. Let $A$ be the matrix of multiplication by $\zeta_m$ on $I$, and let $B$ be the matrix of multiplication by $\zeta_m$ on $\sigma(I)$. Since $I^{\pm 1}$ and $\sigma(I)$ are not isomorphic as $\ZZ[\zeta_m]$ modules, $A^{\pm 1}$ and $B$ are not conjugate.

However, $C_m \ltimes_A \ZZ^{\phi(m)} \cong \langle \zeta \rangle \ltimes I$ and $C_m \ltimes_B \ZZ^{\phi(m)} \cong \langle \zeta \rangle \ltimes \sigma(I)$, and these are isomorphic by $(\zeta^j, x) \mapsto (\sigma(\zeta)^j, \sigma(x))$.

This occurs for $m=37$, where $H \cong \ZZ/37 \ZZ$. If I recall correctly, if $\sigma(\zeta) = \zeta^a$ then $\sigma(h) = h^{a^{21}}$. Since $\mathrm{GCD(21,36)} = 3$, the monomial $a^{21}$ takes $12$ different values modulo $37$ so, taking $h$ a generator of the class group, there are values of than $h^{\pm 1}$ in the $G$ orbit of $h$.