When is $f(a,b)=\frac{a^2+b^2}{1+ab}$ a perfect square rational number?
Edit: in the original formulation, it wasn't clear that $a,b$ were supposed to be positive integers. This answer solves the question for $a,b$ rational instead.
The function $f$ takes every square value, once rational $a,b$ are allowed. Let $t$ be any rational number, and take the equation $(a^2+b^2)/(1+ab)=t^2$. Then $$ a^2 + b^2 = t^2 + t^2ab. $$ For a given non-zero rational $t$, this is a smooth conic with a point $(a,b)=(t,0)$. So it has infinitely many solutions $(a,b)$, so in particular infinitely many non-trivial ones, and moreover they form a parametrizable set. (For $t=0$, the only solution is $a=b=0$.)
You take a line $b = v(a-t)$ which intersects the conic in the $(a,b)$-plane in one rational point besides $(t,0)$. Solving for this point gives $$ a^2 + v^2(a-t)^2 = t^2 + t^2v a(a-t) $$ which leads to $$ (1-t^2v+v^2)a^2 + (t^3v - 2tv^2) a + t^2v^2 - t^2 = 0. $$ The two solutions must add up to minus the linear coefficient divided by the quadratic coefficient, so the second solution $a$ satisfies $$ a = \frac{2tv^2 - t^3v}{1-t^2v+v^2} - t $$ while $$ b = v(a-t) $$ which is your way of completely listing all pairs $(a,b)$ for which $f(a,b)$ is square.
For finding solutions in the case if the square is rational.
$$\frac{a^2+b^2}{ab+1}=\left(\frac{t}{k}\right)^2$$
Use the same formula. For finding all solutions need to consider all possible solutions of the equation Pell. Consider the case where these solutions exist. To do this, lay on the multiplier coefficients.
$$t=cd$$
$$k=nv$$
And you need to check when this Pell equation has the solutions?
$$p^2-(t^4-4k^4)s^2=dv$$
And then the solution is substituted in the above formula.
$$b=4cnps$$
$$a=\frac{c}{nv^2}(p^2+2t^2ps+(t^4-4k^4)s^2)$$
Consider the case which invited.
$$\frac{a^2+b^2}{ab+1}=\left(\frac{t}{k}\right)^2=\left(\frac{58}{41}\right)^2$$
$t=58$ ; $c=2$ ; $d=29$ ;
$k=41$ ; $n=1$ ; $v=41$ ;
$$p^2-(58^4-4*41^4)s^2=p^2-13452s^2=1189$$
Use the first solution. $p=121$ ; $s=1$
Then ;
$$b=4*2*121=968$$
$$a=\frac{2}{41^2}(121^2+2*58^2*121+13452)=1002$$
For such equations:
$$\frac{x^2+y^2}{xy+1}=t^2$$
Using the solutions of the Pell equation. $p^2-(t^4-4)s^2=1$
You can write the solution.
$$x=4tps$$
$$y=t(p^2+2t^2ps+(t^4-4)s^2)$$
It all comes down to the Pell equation - as I said.
Considering specifically the equation:
$$\frac{x^2+y^2}{xy-1}=5$$
Decisions are determined such consistency. Where the next value is determined using the previous one.
$$p_2=55p_1+252s_1$$
$$s_2=12p_1+55s_1$$
You start with numbers. $(p_1;s_1) - (55 ; 12)$
Using these numbers, the solution can be written according to a formula.
$$y=p^2+2ps+21s^2$$
$$x=3p^2+26ps+63s^2$$
If you use an initial $(p_1 ; s_1) - (1 ; 1)$
Then the solutions are and are determined by formula.
$$y=s$$
$$x=\frac{p+5s}{2}$$
As the sequence it is possible to write endlessly. Then the solutions of the equation, too, can be infinite.
If you use a sequence with the first element. $ (p ; s ) $ - $( 4 ; 1 )$ Then decisions can be recorded.
$$y=2s$$
$$x=p+5s$$
If you use a sequence with the first element. $( p ; s )$ - $( 55 ; 12 )$
Using this sequence can be different. On its basis with the first element. $( z ; q )$ - $(2 ; 1 )$
$$z_2=pz_1+7sq_1$$
$$q_2=pq_1+3sz_1$$
Then decisions will be.
$$x=z-q$$
$$y=z+q$$
It is necessary to take into account that the number can have a different sign. - $(p ; s )$