Deciding if $\mathbb{Z}\ltimes_A \mathbb{Z}^5$ and $\mathbb{Z}\ltimes_B \mathbb{Z}^5$ are isomorphic or not
Claim. The groups $G_A$ and $G_B$ are not isomorphic.
We will use the following lemmas.
Lemma 1. Let $A \in \text{GL}_n(\mathbb{Z})$ and let $G_A \Doteq \mathbb{Z} \ltimes_A \mathbb{Z}^n$. Then the following hold:
- The center $Z(G_A)$ of $G_A$ is generated by $\{0\} \times \ker(A - 1_n)$ and $(\omega, (0, \dots, 0))$ where $1_n$ is the $n \times n$ identity matrix and $\omega$ is the order of $A$ in $\text{GL}_n(\mathbb{Z})$ if $A$ has finite order, zero otherwise.
- The derived subgroup $[G_A, G_A]$ of $G_A$ is $\{0\} \times (A - 1_n)\mathbb{Z}^n$. More generally, setting $\gamma_{i + 1}(G_A) \Doteq [\gamma_i(G_A), G_A]$ with $\gamma_1(G_A) \Doteq G_A$, we have $\gamma_{i + 1}(G_A) = \{0\} \times (A - 1_n)^i \mathbb{Z}^n$.
Proof. Straightforward.
For $A$ and $B$ as in OP's question, we have thus $$Z(G_A) = 4\mathbb{Z} \times \ker(A - 1_5), \, Z(G_B) = 4\mathbb{Z} \times \ker(B - 1_5)$$ with $\ker(A - 1_5) = \ker(B - 1_5) = \mathbb{Z} \times \{ (0, 0, 0, 0) \} \subset \mathbb{Z}^5$.
Lemma 2. Let $A$ and $B$ as in OP's question and set $\Gamma_A \Doteq G_A / Z(G_A)$ and $\Gamma_B \Doteq G_B / Z(G_B)$. Then we have $\Gamma_A/ [\Gamma_A, \Gamma_A] \simeq (\mathbb{Z}/ 2 \mathbb{Z})^3 \times \mathbb{Z}/ 4 \mathbb{Z}$ and $\Gamma_B/ [\Gamma_B, \Gamma_B] \simeq \mathbb{Z}/ 2 \mathbb{Z} \times (\mathbb{Z}/ 4 \mathbb{Z})^2$.
Proof. Write $\Gamma_A = \mathbb{Z} / 4 \mathbb{Z} \ltimes_{A'} \mathbb{Z}^4$ and $\Gamma_B = \mathbb{Z} / 4 \mathbb{Z} \ltimes_{B'} \mathbb{Z}^4$ where $A', B' \in \text{GL}_4(\mathbb{Z})$ are obtained from $A$ and $B$ by removing the first row and the first column. Use then the description of the derived subgroup of Lemma 1 which still applies to $\Gamma_A$ and $\Gamma_B$ if we replace $A$ by $A'$ and $B$ by $B'$.
Proof of the claim. If $G_A$ and $G_B$ are isomorphic, then so are $\Gamma_A$ and $\Gamma_B$. This is impossible since the two latter groups have non-isomorphic abelianizations by Lemma 2.
Addendum. Let $C_A$ be the cyclic subgroup of $G_A$ generated by $a \Doteq (1, (0, \dots, 0))$ and $K_A$ the $\mathbb{Z}[C_A]$-module defined as in Johannes Hahn's answer (and subsequently mine) to this MO question. Let $\omega(A)$ be the order of $A$ in $\text{GL}_n(\mathbb{Z})$, that we assume to be finite, and set $e_0 \Doteq (\omega(A), (0, \dots, 0)) \in G_A$. Let us denote by $(e_1, \dots, e_n)$ the canonical basis of $\mathbb{Z}^n \triangleleft G_A$.
It has been established that the pair $\{K_A, K_{A^{-1}}\}$ of $\mathbb{Z}[C]$-modules is an isomorphism invariant of $G_A$, where $C = C_A \simeq C_{A^{-1}}$ with the identification $a \mapsto (1, (0, \dots,0)) \in G_{A^{-1}}$.
For the instances of this MO question, straightforward computations show that $$\left\langle e_0, e_2, e_3, e_5 \, \vert \, (a - 1)e_0 = (a + 1)e_2 = (a + 1)e_3 = (a^3 -a^2 + a - 1)e_5 = 0\right\rangle$$ is a presentation of both $K_A$ and $K_{A^{-1}}$ and $$\left\langle e_0, e_1, e_2, e_3, e_5 \, \vert \, (a - 1)e_0 = (a -1)e_1 = (a + 1)e_2 = (a + 1)e_3 = (a^2 + 1)e_5 + e_1 + e_2 = 0\right\rangle$$ is a presentation of $K_B$.
From the above presentations, we easily infer the following isomorphisms of Abelian groups: $K_A/(a + 1)K_A \simeq \mathbb{Z}/2\mathbb{Z} \times \mathbb{Z}/4 \mathbb{Z} \times \mathbb{Z}^2$ and $K_B/(a + 1)K_B \simeq (\mathbb{Z}/2\mathbb{Z})^2 \times \mathbb{Z}^2$.
As result, the groups $G_A$ and $G_B$ are not isomorphic.
Here is my Magma calculation - I did the $2$-quotient calculations to class 3. Please check that I have entered the group presentations correctly. Note that $(a,b)$ is Magma's notation for the commutator $a^{-1}b^{-1}ab$, and $a^t$ means $t^{-1}at$.
> G1 := Group<a,b,c,d,e,t | (a,b), (a,c), (a,d), (a,e), (b,c), (b,d), (b,e),
> (c,d), (c,e), (d,e), a^t=a, b^t=b^-1, c^t=c^-1, d^t=e*a, e^t=d^-1 >;
>
> G2 := Group<a,b,c,d,e,t | (a,b), (a,c), (a,d), (a,e), (b,c), (b,d), (b,e),
> (c,d), (c,e), (d,e), a^t=a, b^t=b^-1, c^t=c^-1, d^t=e*a*b, e^t=d^-1 >;
> P1 := pQuotient(G1,2,3 : Print:=1);
Lower exponent-2 central series for G1
Group: G1 to lower exponent-2 central class 1 has order 2^4
Group: G1 to lower exponent-2 central class 2 has order 2^9
Group: G1 to lower exponent-2 central class 3 has order 2^14
> P2 := pQuotient(G2,2,3 : Print:=1);
Lower exponent-2 central series for G2
Group: G2 to lower exponent-2 central class 1 has order 2^4
Group: G2 to lower exponent-2 central class 2 has order 2^8
Group: G2 to lower exponent-2 central class 3 has order 2^13
Here is Derek Holt's computation done in GAP:
gap> LoadPackage("anupq");
gap> F := FreeGroup("a","b","c","d","e","t");;
gap> AssignGeneratorVariables(F);
gap> comms := List(Combinations(GeneratorsOfGroup(F){[1..5]},2),Comm);;
gap> G1 := F/Concatenation(comms,
> [Comm(a,t),b^t*b,c^t*c,d^t*a^-1*e^-1, e^t*d]);;
gap> G2 := F/Concatenation(comms,
> [Comm(a,t),b^t*b,c^t*c,d^t*b^-1*a^-1*e^-1,e^t*d]);;
gap> Pq(G1:Prime:=2,ClassBound:=2);
<pc group of size 512 with 9 generators>
gap> StructureDescription(last);
"(C4 x C4 x C4 x C2) : C4"
gap> Pq(G2:Prime:=2,ClassBound:=2);
<pc group of size 256 with 8 generators>
gap> StructureDescription(last);
"C2 x ((C4 x C4 x C2) : C4)"