An Interesting Number Theory Problem: Solve $2520=(x+y+xy)^2+2xy+2y-3x, x, y \in \mathbb Z$

$$2520=x^2+y^2+(xy)^2+2x^2y+2y^2x+2xy+2xy+2y-3x$$ so, if we group in terms of $y$ $$2520=y^2(x^2+2x+1)+y(2x^2+4x+2)+x^2-3x$$ so $$0=y^2\cdot(x+1)^2+2y(x+1)^2+x^2-3x-2520$$ so solving the quadratic equation in the usual way:

$$\Delta=4(x+1)^4-4(x+1)^2(x^2-3x)=4(x+1)^2(x^2+2x+1-x^2+3x+2520)$$ so we have found out that $$\Delta=(2x+2)^2\cdot(5x+2521)$$

but we know

$$y=\frac{-2(x+1)^2\pm\sqrt{\Delta}}{2(x+1)^2}=\pm\frac{\sqrt{\Delta}}{2(x+1)^2}-1=\pm\frac{2(x+1)\sqrt{5x+2521}}{2(x+1)^2}-1=\pm\frac{\sqrt{5x+2521}}{(x+1)}-1$$ so for $y$ to be an integer, $x+1$ must divide $\sqrt{5x+2521}$ and $5x+2521$ must be a perfect square. Let $5x+2521=k^2$ so $x=\frac{k^2-2521}{5}$ so $$\big(\frac{k^2-2521}{5}+1\big)^2\text{ divides } k$$ so $$(k^2-2516)^2\text{ divides } 25k$$ so $$(k^2-2516)^2\leq 25|k|$$ which gives us $k=\pm50,\pm51$. If $k=\pm 50$, then because $x=\frac{k^2-2521}{5}$, x wouldn't be an integer so contradiction. So $k=51$, so $x=\frac{51^2-2521}{5}=16$

Because $x=16$ wecan now see that because $y=\pm\frac{\sqrt{5x+2521}}{(x+1)}-1=\pm3-1$ so $y$ is $2$ or $-4$.

So the solutions are $(x,y)=(16,-4)$ and $(16,2)$

Denote $z=x+1, w=y+1$ then your result $$t^2+2t-5x=2520 \implies (t+1)^2-5(z-1)=2519 \\\implies w^2z^2-5z=2516=z(w^2 z - 5) \implies z|2516=2^2\cdot 17\cdot 37$$

If $z$ is even then $w^2z-5$ is odd, so $4|z, z=4u, 4u^2w^2=5u+629 \implies u\equiv -1 \pmod 4$, and $u|17\cdot 37$, therefore $u=-1, -17, -37, -629$, none of which yields a perfect square for $5u+629$.

If $z$ is odd then $wz$ is odd, $(wz)^2 \equiv 1 \pmod 4$, so $z\equiv 1 \pmod 4, z=1, 17, 37, 629$. We have the following four cases:

$z=1, w^2-5=2516$, no solution;

$z=17, 17w^2-5=148 \implies w=\pm 3 \implies x=16, y=2 \text{ or } -4;$

$z=37, 37w^2-5=68$, no solution;

$z=629, 629w^2-5=4$, no solution.

So the only solutions are $(16, 2)$ and $(16, -4)$.