Functional Equation: When $f(x+y)=f(x)+f(y)-(xy-1)^2$

$$f(x+y)=f(x)+f(y)-(xy-1)^2\\ f(2)=2f(1)\\ f(3)=f(2)+f(1)-1=3f(1)-1\\ f(2)+f(2)-9=f(4)=f(1)+f(3)-4\\ 4f(1)-9=4f(1)-5$$

First put $x=y=0$ into the functional equation to obtain that

$$f(0)=1 \tag{1}$$

Then suppose that $y \ne 0$ so dividing the functional equation by $y$ we can write

$$\frac{f(x+y)-f(x)}{y}=\frac{f(y)-1}{y}-\frac{(xy-1)^2-1}{y} \tag{2}$$

$$\frac{f(x+y)-f(x)}{y}=\frac{f(y)-f(0)}{y-0}-\frac{x^2y^2-2xy}{y} \tag{3}$$

Next, take the limit when $y \to 0$ to obtain

$$f'(x)=f'(0)+2x \tag{4}$$

Where we have assumed that $f(x)$ is differentiable over $\mathbb{R}$. Next, integrating $(4)$ we get

$$f(x)-f(0)=f'(0)x+x^2 \tag{5}$$

Or equivalently

$$f(x)=x^2+f'(0)x+1 \tag{6}$$

But we can observe that $(6)$ will never satisfy the functional equation since $f(x)$ is a second degree polynomial and there are fourth degree terms like $x^2y^2$ in the functional equation . So the only thing that we are left with is that

There is no differentiable $f(x)$ which satisfies the functional equation!