Basic proof that $GL(2, \mathbb{Z})$ is not nilpotent

One possibility is to show that there is a proper, self-normalizing subgroup, $$ \left\{ \begin{pmatrix} a&b \\ 0&d \end{pmatrix} : a, d = \pm 1, b \in \mathbb{Z}\right\} $$ being perhaps the easiest candidate.

Another possibility is to show that the group has a non-nilpotent homomorphic image. Consider the reduction modulo two mapping, and identify $GL_2(\Bbb{F}_2)$ as the smallest non-nilpotent group.

Let's find the upper central series. For ease of notation, let $$ T_k = \begin{pmatrix} 1 & k\\0 & 1\end{pmatrix}$$ and $$ R = \begin{pmatrix} 0 & 1 \\ 1 & 0\end{pmatrix}$$ be translation and reflection matrices, with inverses $T_k^{-1} = T_{-k}$ and $R^{-1}=R$. The center of $GL(2,\mathbb{Z}$) must surely commute with all $T_k$ (that is, for all $k$), and also with $R$. If we let $$ X = \begin{pmatrix} a & b \\ c & d\end{pmatrix}$$ then $X$ being in the center implies \begin{align} T_kXT_{-k} &= X\\ RXR &= X \end{align} This is enough to imply $b=c=0$ and $a=d$ [to see this, just do the multiplcations and equate matrix coefficients]; that is, the center is $Z_1=\{I, -I\}$, with $I$ the $2\times2$ identity matrix.

What about the second center, $Z_2$? These are elements that are central "mod $Z_1$", or elements $X$ such that for all $Y\in GL(2,\mathbb{Z})$, we have $$ YXY^{-1} = \pm X$$ [Note: the choice of $+$ or $-$ on the right hand side is dependent on $Y$.]

Again, using $T_k$ and $R$, show that $X$ would still have to be $\pm I$. This means $Z_2=Z_1$, and thus our upper central series has stalled (and hasn't become the whole group); the group $GL(2,\mathbb{Z})$ thus is not nilpotent.

EDIT: Adding in more details.

First center

If $X$ is in the center of $GL(2,\mathbb{Z})$, then it commutes with all matrices. In particular, we have $$ T_kXT_{-k}=X$$ for all $T_k$. Expanding that out, we get four equations: \begin{align} a+ck &= a\\ b+k(d-a-ck) &= b\\ c &= c\\ d-ck &= d \end{align} Since this holds for any $k$, we get from the first equation $a+ck_1=a+ck_2$ for $k_1\neq k_2$, which immediately implies $c=0$. Knowing $c=0$, the second equation similarly implies $a=d$. To get $b=c$, we expand $RXR=X$: \begin{equation} \begin{pmatrix} d & c\\ b & a \end{pmatrix} = \begin{pmatrix} a & b\\c & d\end{pmatrix} \end{equation}

Thus, our center is $\{I, -I\}$ (we actually need to check these commute with all matrices, but that's easy).

Second center

The second center is defined as $Z_2(G)/Z_1(G)=Z(G/Z_1(G))$; that is, it consists of the matrices that commute with all others, "mod $Z_1$". This means that if $X\in Z_2$, and $Y\in GL(2,\mathbb{Z})$, then there exists $Z_Y\in Z_1$ such that $$ YXY^{-1} = Z_YX$$ The element from $Z_1$ will probably depend on $Y$; the point is that it always exists. (If you want to think about this more abstractly, the above is just saying that $[Z_2, G]\subset Z_1$).

Let's find the second center. If $X\in Z_2$, then for every $k$, there exists $Z_k\in Z_1=\{I, -I\}$ such that $$ T_kXT_{-k} = Z_kX$$

Since there are only two choices for $Z_k$ (namely, $I$ or $-I$), and since there are infinitely many $k$s, we can choose $k_1, k_2$ such that $Z_{k_1}=Z_{k_2}$, and thus just as above, we get (after possibly cancelling a negative sign if $Z_{k_1}=Z_{k_2}=-I$) $$ a + ck_1 = a + ck_2$$ which again implies $c=0$. Using $R$ we have \begin{equation} \begin{pmatrix} d & c\\ b & a \end{pmatrix} = Z_R\begin{pmatrix} a & b\\c & d\end{pmatrix} \end{equation} where $Z_R$ is $I$ or $-I$. Thus we get $b=\pm c=0$, and $a=\pm d$.

That means the only thing left to show that $Z_2=Z_1=\{I, -I\}$ is to show that $a=-d$ cannot happen. That is, we need to show the matrix $$ \begin{pmatrix} -1 & 0\\ 0 & 1 \end{pmatrix} $$ is not in $Z_2$. I'll leave this as an exercise (hint: try checking what happens with $T_1$).