A "subtle" isomorphism testing problem: $\mathbb{Z}\ltimes_{A} \mathbb{Z}^5\cong \mathbb{Z}\ltimes_{B}\mathbb{Z}^5$ or not?
The following calculations seem to distinguish between them.
> #LowIndexSubgroups(GA,4);
30
> #LowIndexSubgroups(GB,4);
26
They have different numbers of homomorphisms onto $A_4$:
> #Homomorphisms(GA,Alt(4),Sym(4));
5
> #Homomorphisms(GB,Alt(4),Sym(4));
1
(The options third argument $\mathsf{Sym(4)}$ means count (surjective homomorphisms) up to conjugacy in $\mathsf{Sym(4)}$.)
Here is yet another approach:
> P,phi:=pQuotient(GA,3,1);
> AQInvariants(Kernel(phi));
[ 2, 2, 0, 0, 0, 0 ]
> P,phi:=pQuotient(GB,3,1);
> AQInvariants(Kernel(phi));
[ 0, 0, 0, 0 ]
In fact these three methods are all detecting the same difference in finite quotients of the groups, but I included them all to give you an indication of possible techniques for proving non-isomorphism.
Ultimately, all of these techniques rely on looking at various types of computable quotients of the groups. Unfortunately there are examples of pairs of non-isomorphic finitely presented groups which cannot be distinguished in this fashion by their computable quotients (in fact the unsolvability of the general isomorphism problem implies that such examples must exist.)
Claim. The groups $G_A$ and $G_B$ are not isomorphic.
We will use the following lemma.
Lemma. Let $\Gamma_A = G_A/Z(G_A) = C_6 \ltimes_{A'} \mathbb{Z}^4$ and $\Gamma_B = G_B/Z(G_B) = C_6 \ltimes_{B'} \mathbb{Z}^4$, where $C_6 = \langle \alpha \rangle$ is the cyclic group of order $6$ and $A'$ and $B'$ are obtained from $A$ and $B$ respectively by removing the first row and the first column. Let $M_A \Doteq [\Gamma_A, \Gamma_A]$ and $M_B \Doteq [\Gamma_B, \Gamma_B]$ be the corresponding derived subgroups considered as $\mathbb{Z}[C_6]$-modules where $\alpha$ acts as $A'$ on $M_A$ and as $B'$ on $M_B$. Then we have the following $\mathbb{Z}[C_6]$-module presentations: $$M_A = \langle x, y \vert \, (\alpha^2 + \alpha + 1)x = (\alpha^2 - \alpha + 1)y = 0\rangle $$ and $$ M_B = \langle x \,\vert \, (\alpha^4 + \alpha^2 + 1)x = 0\rangle $$
Proof. Use the description of $M_A$ as $(A' - 1_4) \mathbb{Z}^4$ and observe how $A'$ transforms the column vectors of $A' - 1_4$. Do the same for $M_B$.
We are now in position to prove the claim.
Proof of the claim. It suffices to show that $\Gamma_A$ and $\Gamma_B$ are not isomorphic. An isomorphism $\phi: \Gamma_A \rightarrow \Gamma_B$ would induce an isomorphism $M_A \rightarrow M_B$ of Abelian groups. As we necessarily have $\phi((\alpha, (0, 0, 0, 0))) = (\alpha^{\pm 1}, z)$ for some $z \in \mathbb{Z}^4$ and since the presentations of the above lemma remain unchanged if we replace $\alpha$ by $\alpha^{-1}$, the isomorphism $\phi$ would induce an isomorphism of $\mathbb{Z}[C_6]$-modules. This is impossible because $M_A$ cannot be generated by less than two elements whereas $M_B$ is cyclic over $\mathbb{Z}[C_6]$. Observe indeed that $M_A$ surjects onto $\mathbb{F}_4 \times \mathbb{F}_4$ where $\mathbb{F}_4 \simeq \mathbb{Z}[C_6]/(2, \alpha^2 + \alpha + 1)$ is the field with four elements.
Addendum 1. Let $G$ be finitely generated group $G$. We denote by $d(G)$ the rank of $G$, i.e., the minimum number of generators of $G$. For these two special instances, we actually have $d(G_A) = 4$ and $d(G_B) = 3$.
In general, it can be difficult to compute the rank of a group, but some knowledge is available for $G_A$ and some other split extensions by cyclic groups, see [1, Corollary 2.4] and [2, Theorem A and Section 3.1].
Let us set $G_A = \mathbb{Z} \ltimes_A N_A$ with $N_A \Doteq \mathbb{Z}^n$. Then $N_A$ can be endowed with the structure of a $\mathbb{Z}[C]$ module where $C \subset G_A$ is the infinite cyclic group generated by $a \Doteq (1, (0, \dots, 0)) \in G_A$ which acts on $N_A$ via conjugation, or equally, via multiplication by $A$.
Let $R$ be a unital ring and let $M$ be a finitely generated $R$-module. We denote by $d_R(M)$ the minimum number of generators of $M$ over $R$. Then the aforementioned results implies that $$d(G_A) = d_{\mathbb{Z}[C]}(N_A) + 1.$$
Let us denote by $(e_1, \dots, e_n)$ the canonical basis of $\mathbb{Z}^n$. For $A$ and $B$ as in OP's question, it is easy to derive the following $\mathbb{Z}[C]$-module presentations: $$N_A = \langle e_1, e_2, e_4 \, \vert (a - 1)e_1 = (a^2 + a + 1)e_2 = (a^2 - a + 1)e_4 = 0 \rangle$$ and $$N_B = \langle e_1, e_5 \, \vert (a - 1)e_1 = (a^4 + a^2 + 1)e_5 = 0 \rangle.$$
From these presentations and the above rank formula, we can easily infer the claimed identities, that is, $d(G_A) = 4$ and $d(G_B) = 3$.
Addendum 2. Let $C_A$ be the cyclic subgroup of $G_A$ generated by $a \Doteq (1, (0, \dots, 0))$ and $K_A$ the $\mathbb{Z}[C_A]$-module defined as in Johannes Hahn's answer (and subsequently mine) to this MO question. Let $\omega(A)$ be the order of $A$ in $\text{GL}_n(\mathbb{Z})$, that we assume to be finite, and set $e_0 \Doteq (\omega(A), (0, \dots, 0)) \in G_A$. Let us denote by $(e_1, \dots, e_n)$ the canonical basis of $\mathbb{Z}^n \triangleleft G_A$.
It has been established that the pair $\{K_A, K_{A^{-1}}\}$ of $\mathbb{Z}[C]$-modules is an isomorphism invariant of $G_A$, where $C = C_A \simeq C_{A^{-1}}$ with the identification $a \mapsto (1, (0, \dots,0)) \in G_{A^{-1}}$. It can be used to address the previous example and this one as well.
For the instances of this MO question, straightforward computations show that $$K_A = K_{A^{-1}}= \langle e_0, e_1, e_2, e_4 \, \vert \, (a - 1)e_0 = (a - 1)e_1 = (a^2 + a + 1)e_2 = (a^2 - a + 1)e_4 = 0\rangle$$ and $$K_B = \langle e_0, e_1, e_5 \, \vert \, (a - 1)e_0 = (a -1)e_1 = (a^4 + a^2 + 1)e_5 = 0\rangle.$$ Since $d_{\mathbb{Z}[C_A]}(K_A) = 4$ and $d_{\mathbb{Z}[C_B]}(K_B) = 3$ the groups $G_A$ and $G_B$ are not isomorphic.
[1] G. Levitt and V. Metaftsis, "Rank of mapping tori and companion matrices", 2010.
[2] L. Guyot, "Generators of split extensions of Abelians groups by cyclic groups", 2018.