How to find the determinant of this $3 \times 3$ Hankel matrix?

Here is a proof that is decidedly not from the book. The determinant is obviously a polynomial in n of degree at most 6. Therefore, to prove it is constant, you need only plug in 7 values. In fact, -4, -3, ..., 0 are easy to calculate, so you only have to drudge through 1 and 2 to do it this way !

Recall that $a^2-b^2=(a+b)(a-b)$. Subtracting $\operatorname{Row}_1$ from $\operatorname{Row}_2$ and from $\operatorname{Row}_3$ gives $$ \begin{bmatrix} n^2 & (n+1)^2 & (n+2)^2 \\ 2n+1 & 2n+3 & 2n+5 \\ 4n+4 & 4n+8 & 4n+12 \end{bmatrix} $$ Then subtracting $2\cdot\operatorname{Row}_2$ from $\operatorname{Row}_3$ gives $$ \begin{bmatrix} n^2 & (n+1)^2 & (n+2)^2 \\ 2n+1 & 2n+3 & 2n+5 \\ 2 & 2 & 2 \end{bmatrix} $$ Now, subtracting $\operatorname{Col}_1$ from $\operatorname{Col}_2$ and $\operatorname{Col}_3$ gives $$ \begin{bmatrix} n^2 & 2n+1 & 4n+4 \\ 2n+1 & 2 & 4 \\ 2 & 0 & 0 \end{bmatrix} $$ Finally, subtracting $2\cdot\operatorname{Col}_2$ from $\operatorname{Col}_3$ gives $$ \begin{bmatrix} n^2 & 2n+1 & 2 \\ 2n+1 & 2 & 0 \\ 2 & 0 & 0 \end{bmatrix} $$ Expanding the determinant about $\operatorname{Row}_3$ gives $$ \det A = 2\cdot\det \begin{bmatrix} 2n+1 & 2\\ 2 & 0 \end{bmatrix} =2\cdot(-4)=-8 $$ as advertised.

That's not the nicest way to do it but it's quite simple and works.

To avoid developping the squares, I used $a^2-b^2 =(a+b)(a-b)$ (and $a-b$ was always $1$ when I used it). Then, once I had enough $0$s, I simply exapended by the last column.

$$\begin{array}{l} \begin{vmatrix} n^{2} & (n+1)^{2} &(n+2)^{2} \\ (n+1)^{2} &(n+2)^{2} & (n+3)^{2}\\ (n+2)^{2} & (n+3)^{2} & (n+4)^{2} \end{vmatrix} &= \begin{vmatrix} n^{2} & (n+1)^{2} &(n+2)^{2}-(n+1)^{2} \\ (n+1)^{2} &(n+2)^{2} & (n+3)^{2}-(n+2)^{2}\\ (n+2)^{2} & (n+3)^{2} & (n+4)^{2}-(n+3)^{2} \end{vmatrix}\\ &= \begin{vmatrix} n^{2} & (n+1)^{2} &2n+3 \\ (n+1)^{2} &(n+2)^{2} & 2n+5\\ (n+2)^{2} & (n+3)^{2} & 2n+7 \end{vmatrix}\\ &= \begin{vmatrix} n^{2} & (n+1)^{2}-n^{2} &2n+3 \\ (n+1)^{2} &(n+2)^{2}-(n+1)^{2} & 2n+5\\ (n+2)^{2} & (n+3)^{2}-(n+2)^{2} & 2n+7 \end{vmatrix}\\ &= \begin{vmatrix} n^{2} & 2n+1 &2n+3 \\ (n+1)^{2} &2n+3 & 2n+5\\ (n+2)^{2} & 2n+5 & 2n+7 \end{vmatrix}\\ &= \begin{vmatrix} n^{2} & 2n+1 &2n+3 \\ (n+1)^{2} &2n+3 & 2n+5\\ (n+2)^{2}-(n+1)^{2} & 2n+5-(2n+3) & 2n+7-(2n+5) \end{vmatrix}\\ &= \begin{vmatrix} n^{2} & 2n+1 &2n+3 \\ (n+1)^{2} &2n+3 & 2n+5\\ 2n+3 & 2 & 2 \end{vmatrix}\\ &= \begin{vmatrix} n^{2} & 2n+1 &2n+3 \\ (n+1)^{2}-n^{2} &2n+3-(2n+1) & 2n+5-(2n+3)\\ 2n+3 & 2 & 2 \end{vmatrix}\\ &= \begin{vmatrix} n^{2} & 2n+1 &2n+3 \\ 2n+1 &2 & 2\\ 2n+3 & 2 & 2 \end{vmatrix}\\ &= \begin{vmatrix} n^{2} & 2n+1 &2n+3-(2n+1) \\ 2n+1 &2 & 2-2\\ 2n+3 & 2 & 2-2 \end{vmatrix}\\ &= \begin{vmatrix} n^{2} & 2n+1 &2 \\ 2n+1 &2 & 0\\ 2n+3 & 2 & 0 \end{vmatrix}\\ &= \begin{vmatrix} n^{2} & 2n+1 &2 \\ 2n+1 &2 & 0\\ 2n+3-(2n+1) & 2-2 & 0-0 \end{vmatrix}\\ &= \begin{vmatrix} n^{2} & 2n+1 &2 \\ 2n+1 &2 & 0\\ 2 & 0 & 0 \end{vmatrix}\\ &= 2\begin{vmatrix} 2n+1 &2\\ 2 & 0 \end{vmatrix}\\ &= -8 \end{array}$$