Which positive integers $a$ and $b$ make $(ab)^2-4(a+b) $ a square of an integer?
We can find an upper bound for their "product" in the following way: \begin{align} {\left( {ab} \right)^2} - 4\left( {a + b} \right) &< {\left( {ab} \right)^2} \ \ (\because a,b>0) \\ {\left( {ab} \right)^2} - 4\left( {a + b} \right) &\le {\left( {ab - 1} \right)^2} \\ (2ab-1)-4(a+b) &\le 0\\ ab - 2(a+b) - \frac 12 \color{blue}{+4} &\le 0 \color{blue}{+4}\\ \left( {a - 2} \right)\left( {b - 2} \right) &\le \frac{9}{2} \\ \left( {a - 2} \right)\left( {b - 2} \right) &\le 4 \\ \end{align}
Now, seeing some cases should finish the work.
If $(ab)^2-4(a+b)$ is greater than $(ab-1)^2$ then it cannot be a square, since it is strictly between two consecutive squares. Hence
$$(ab)^2-4(a+b) \le (ab-1)^2=(ab)^2-2ab+1$$ $$2ab-4a-4b-1\le0$$ $$2(a-2)(b-2)=2ab-4a-4b+8\le 9$$
which, again, gives a finite set of possibilities to be checked.
WLOG suppose $a \ge b$. We need only consider the cases:
- $b=1,2$
- $b=3, a\le6$
- $b\ge 4, a < 3$ (this case contradicts $a\ge b$)
For $b=1$, $(ab)^2-4(a+b) = a^2-4a-4$. For $a\ge7$, $a^2-4a-4> a^2-6a+9=(a-3)^2$. But $a^2 -4a-4 < a^2-4a+4=(a+2)^2$. So we only need to check $1 \le a \le 6$.
For $b=2$, $(ab)^2-4(a+b) = 4a^2-4a-8$, which is a square only if $a^2-a-2$ is. For $a\ge4$, $a^2-a-2> a^2-2a+1 =(a-1)^2$. But $a^2-a-2<a^2$, so we only need to check $a=2,3$.
For $b=3$, $(ab)^2-4(a+b) = 9a^2-4a-12$, and we only need to check $3 \le a \le 6$.