A generalization of the (in)famous IMO 1988 problem 6: If $\frac{a^2 + b^2 - abc}{ab + 1}$ is a positive integer then it is a square.

Yes, this fact is true. We have: $$\frac{a^2+b^2-abc}{ab+1}=k \implies a^2-b(c+k)a+(b^2-k)=0$$ Following the standard procedure for Vieta jumping, assume that $(a,b)$ is the smallest solution with respect to the sum of solutions $a+b$, such that $k$ is not a perfect square. If one root of $$x^2-b(c+k)x+(b^2-k)$$ is $a$, then: $$x=b(c+k)-a$$ $$x=\frac{b^2-k}{a}$$ WLOG let $a>b$. We can note that the first equation gives $x$ is an integer, while the second gives: $$x<\frac{b^2}{a}<\frac{a^2}{a}=a \implies x<a$$ By the minimality of the solution $(a,b)$, we cannot have $(b,x)$ as a solution. Thus, we must have $x$ to be non-negative. We cannot have $x=0$ as this would give $b^2=k$ contradicting the assumption that $k$ is not a square.

If $x<0$, we must have $b^2<k$. This gives: $$(k-b^2)(ab+1)=a^2+b^2-abc-ab^3-b^2=a(a-bc-b^3) \implies \frac{k-b^2}{a}=\frac{a-bc-b^3}{ab+1}$$ $$-x=\frac{a-bc-b^3}{ab+1}<\frac{a}{ab+1}<1 \implies x>-1$$ This is a contradiction since there is no integer $0>x>-1$. Thus, there were no solutions to begin with, where $k$ is not a square.


We will adapt the proof for $c=0$. So let us consider $c\ge 0$ to be fixed.

Assume we have a solution $(a,b)$ of the problem, so that the number $$ k =\frac{a^2+b^2-abc}{ab+1}\tag{$1$} $$ is an integer, $k\in \Bbb Z$, and it is positive, $k>0$.

We can and do assume that $a>b\ge 0$. (The case $a=b$ is easily eliminated, since then in $a^2(2-c)/(a^2+1)$ the factor $a^2$ of the numerator, and the denominator are relatively prime, so $2-c\in \{0,1,2\}$ is a multiple of the denominator. The case $c=2$ leads to $k=0$, not allowed. If $c=1$, we obtain $a=b=0$ as unique case of an integer $k$, but then again $k=0$. It remains $c=0$, the OIM case, then $a=b=1$. We observe that in this case there is also the solution $(1,0)$ near $(1,1)$, a solution which makes the sum $a+b$ smaller.)

We furthermore can and do assume, that $(a,b)$ with $a>b\ge 0$ is a solution that minimizes $$ a+b\ . $$ Let us show that $b=0$.

Assume the contrary, $a>b>0$.

We formally substitute $a$ in the equation $(1)$ with an indeterminate $X$ and write the corresponding equation of degree two in $X$ explicitly: $$ \underbrace{X^2-b(c+k)X + b^2-k}_{f(X)} = 0\ . $$ One solution is known, $x_1=a\in \Bbb Z$, there is by Vieta an other solution, making their sum $b(c+k)$, so the second solution is $x_2b(c+k)-a$. Let us show that $0\le x_2< x_1=a$. We first compute $$ \begin{aligned} f(a+1) &=a^2+2a+1\ -\ a(bc+bk)\ +\ (bc+bk)\ +\ b^2-k\\ &=2a+1 \ - \ (bc+bk)\ ,\\ af(a+1) &=2a^2+a \ -\ a(bc+bk) \\ &=2a^2+a \ -\ (a^2+b^2-k) \\ &=2a^2-b^2+a +k \\ &>0\ . \end{aligned} $$ Since $x_1=a$ is a root, and the other root $x_2$ is an integer, and (on $\Bbb Z$) in the interval $I$ between the roots $x_1,x_2$ (in the correct order) the function $f$ takes values $\le 0$, and outside $I$ values $>0$, the second root $x_2=bc+bk-a$ is $\le x_1=a$. (Or use $f(a+1)>0$ combined with the above computation of $f(a+1)$.)

The initial discussion about the case $a=b$ shows that this case ($x_1=x_2)$ is here excluded. So we have the strict inequality $$ x_1 = bc+bk-a<a=x_2 \ .\tag{$2$}$$ Let us see that $0\le x_2=bc+bk-a$. (And the case of equality is of course possible.) Assume this is not the case, so that $\color{blue}{x_2<0}$. In the equality $$ \frac{x_2^2+b^2-x_2bc}{x_2b+1}=k>0 $$ the denominator is then $<0$, so the numerator $x_2^2+b^2-x_2bc$ is also strictly negative. This implies $c>0$, and the expression $x_2/b$ is then between the two roots of the equation $Y^2-cY+1$, which have the same sign (Vieta product is one), and thus positive (Vieta sum is $c> 0$). We get $\color{red}{x_2>0}$. Contradiction with the assumption made in blue. So we have indeed $x_2\ge 0$.

It remains to see that the solution $(x_2,b)$ is breaking the chosen minimality of $(a,b)=(x_1, b)$, since $x_2+b<x_1+b$.

Contradiction with the assumption $b>0$. So $b=0$, which gives $k=a^2$, a square.

$\square$


This is an adaptation of the solution given in https://en.wikipedia.org/wiki/Vieta_jumping#Standard_Vieta_jumping for the case $c=0$.