Structure of the group generated by two specific symplectic matrices

Your representation $p$ is not faithful, since we have $$ (ABA^{-1}BA^{-1}BAB^{-1})^3 \ = \ 1. $$ In particular, this means that $$ (aba^{-1}ba^{-1}bab^{-1})^3 \ = \ \left(\begin{array}{rr}% -24587&42408\\% 15048&-25955\\% \end{array}\right) $$ lies in the kernel of $p$.

After talking to Gabriela Weitze-Schmithuesen, I think that we can show the arithmeticity of $\langle A, B\rangle$ using the argument in Section 2 of this paper of Singh and Venkataramana here (http://www.ams.org/mathscinet-getitem?mr=3165424).

Indeed, let us consider the permutation matrix $P=\left(\begin{array}{cccc} 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & 1 & 0 \\ 0 & 1 & 0 & 0 \end{array}\right)$ exchanging the second and fourth basis vectors and let us show that the conjugate $P\cdot \langle A, B\rangle\cdot P$ of $\langle A, B \rangle$ is arithmetic, i.e., it has finite-index in $Sp(4,\mathbb{Z})$.

For this sake, we asked Sage to look words on $A$, $B$, $A^2$ and $B^2$ of size $\leq 10$ fixing the first basis vector, and we found that the matrices $x=P(A^2 B)^2(AB^2)^2P$, $y=PABA^2BA(AB^2)^2P$ and $z=PA^2BA^2(B^2A)^2BP$ are interesting because $$[y,x]=yxy^{-1}x^{-1} = \left(\begin{array}{cccc} 1 & 0 & 0 & 18 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{array}\right), \quad x^6[y,x] = \left(\begin{array}{cccc} 1 & 0 & 18 & 0 \\ 0 & 1 & 0 & 18 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{array}\right)$$ $$y^6[y,x]^{-1} = \left(\begin{array}{cccc} 1 & 18 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & -18 \\ 0 & 0 & 0 & 1 \end{array}\right), \quad z^6 \beta^{-1} = z^6 (x^6 [y,x])^{-1} = \left(\begin{array}{cccc} 1 & 0 & 0 & 0 \\ 0 & 1 & -18 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{array}\right)$$ generate the positive root groups of $Sp(4,\mathbb{R})$ and, thus, $P\cdot\langle A, B \rangle\cdot P$ intersects the subgroup $U(\mathbb{Z})$ of unipotent upper triangular matrices of $Sp(4,\mathbb{Z})$ in a finite-index subgroup.

Since we know that $\langle A, B\rangle$ is Zariski-dense (see my comment above to a question of Venkataramana), we can apply a result of Tits (http://www.ams.org/mathscinet-getitem?mr=424966) saying that Zariski dense subgroups of $Sp(4,\mathbb{Z})$ containing a finite-index subgroup of $U(\mathbb{Z})$ are arithmetic to get the desired conclusion.