functional equation $f(f(x))=af(x)+bx$

Assume $f(x_1)=f(x_2)$. Then $$ bx_1=f(f(x_1))-af(x_1)=f(f(x_2))-af(x_2)=bx_2$$ and hence (as $b\ne 0$) $x_1=x_2$. We conclude that $f$ is injective. As $f$ is also continuous, $f$ is strictly monotonous. Thereefore $f\circ f$ is strictly increasing.

Assume $L:=\lim_{x\to +\infty}f(x)$ exists. Then $f(L)=\lim_{x\to+\infty}(af(x)+bx)=\infty$, contradiction. Similarly, $\lim_{x\to -\infty}f(x)$ cannot exist. We conclude that $f$ is a continuous bijection $\Bbb R\to\Bbb R$.

Pick $x_0\in \Bbb R$ and define recursively $x_{n+1}=f(x_n)$. Then $x_{n+2}=ax_{n+1}+bx_n$ for all $n$. It is well-known that the solutiosn of this recursion are of the form $$x_n=\alpha \lambda_1^n+\beta\lambda_2^n, $$ where $\lambda_{1,2}=\frac{a\pm\sqrt{a^2+4b}}{2}$ are the solutions of $X^2-aX-b=0$ and $\alpha,\beta$ are found by solving $x_0=\alpha+\beta$, $x_1=\alpha\lambda_1+\beta\lambda_2$. From the given restrictions for $a,b$, we find $-\lambda_1<\lambda_2<0<\lambda_1<1$. This implies that $x_n\to 0$, independent of $x_0\in\Bbb R$. By continuity, $f(0)=0$.

Assume $x_n>0$ and $x_{n+2}\ge x_n$. As $f\circ f$ is strictly increasing, this implies that the subsequence $x_{n+2k}$ is non-decreasing, contradicting convergence to $0$. Therefore $x_n>0$ implies $x_{n+2}<x_n$. Similarly, $x_n<0$ implies $x_{n+2}>x_n$.

As $f$ is bijective, we can extend the sequence $x_n$ no $n\in\Bbb Z$.

• If $\alpha\ne 0$, the $\lambda_1^n$ terms is dominant for $n\gg 0$. In particular, all $x_n$ with $n\gg 0$ have the same sign. It follows that the sequences $x_{2n}$ and $x_{2n+1}$ are both decreasing or both increasing.

• If $\beta\ne 0$, the$\lambda_2^n$ term is dominant for $n\ll 0$. In particular, $x_n$ flips signs as $n\ll 0$. It follows that the sequences $x_{2n}$ and $x_{2n+1}$ have opposite monotonicity.

Both these observations together imply that $\alpha$ and $\beta$ cannot both be non-zero. Therefore, either $f(x)=\lambda_1x$ or $f(x)=\lambda_2 x$. By continuity of $f$, a switch between these two conditions can at most occur at $x=0$, but then $f$ would not be monotonous. We conclude that no switch occurs, so that the only possible solutions are $$ f(x)=\lambda_1x\text{ for all }x\in\Bbb R$$ and $$ f(x)=\lambda_2x\text{ for all }x\in\Bbb R.$$