Solving (quadratic) equations of iterated functions, such as $f(f(x))=f(x)+x$

One way to think about this is that you are assuming that $f(x) = cx$ and then solving for the value of $c$.

But there is something much more interesting going on; you have started doing some abstract algebra without knowing it. $f$ is not a real number, but it does live in something called an algebra over a field, which is a special type of ring. In particular, $f$ lives in the algebra of continuous functions $\mathbb{R} \to \mathbb{R}$. In this algebra there is a multiplicative identity $f(x) = x$ which plays the role of the zeroth power, an addition, a multiplication, and a scalar multiplication by real numbers.

Many manipulations which are possible with ordinary real numbers are possible in rings and algebras; in particular, the first half of the proof of the quadratic formula carries through totally abstractly (the part where you complete the square).

Unfortunately, the second half does not. In other words, it is not true that there are only two solutions to the equation $f^2 = a$ in a general algebra. This is because algebras are not in general integral domains. There may be none or infinitely many!

However, in this special case $a$ is a positive real multiple of the identity, so we know it has at least two square roots (even though there may be more). These are the solutions that you found, and this method for finding them is perfectly valid.

This is a very important technique. It is often used in the case where $f$ is a differential operator as a concise way to solve linear homogeneous ODEs.

Assume $f$ is a linear transformation, then linear algebra lets us give it a matrix representation $f(x) = F \cdot x$. It is also the case that $f \circ g = F G$, function composition is represented by matrix multiplication.

In your derivation you do not make any distinction between $f$ and $F$, this is fine as long as you know you are doing it. Also the matrix $F$ is $1 \times 1$ because $\mathbb{R}$ is a $1$-dimensional vector space.

That is why we have:

$$ \begin{eqnarray} f(f(x)) &= f(x) + x \\\\ \iff \\\\ F^2 &= F + I \\\\ \iff \\\\ \varphi^2 &= \varphi + 1 \end{eqnarray} $$

(Where $f(x) = F \cdot x = \varphi x$)

The only problem is that we cannot be sure $f$ is a linear transform just based on what was told. So it is not known whether this gives all solutions.

In fact this belongs to a functional equation of the form http://eqworld.ipmnet.ru/en/solutions/fe/fe1220.pdf.

Let $\begin{cases}x=u(t)\\f=u(t+1)\end{cases}$ ,

Then $u(t+2)=u(t+1)+u(t)$

$u(t+2)-u(t+1)-u(t)=0$

$u(t)=C_1(t)\left(\dfrac{1+\sqrt{5}}{2}\right)^t+C_2(t)\left(\dfrac{1-\sqrt{5}}{2}\right)^t$ , where $C_1(t)$ and $C_2(t)$ are arbitrary periodic functions with unit period

$\therefore\begin{cases}x=C_1(t)\left(\dfrac{1+\sqrt{5}}{2}\right)^t+C_2(t)\left(\dfrac{1-\sqrt{5}}{2}\right)^t\\f=C_1(t)\left(\dfrac{1+\sqrt{5}}{2}\right)^{t+1}+C_2(t)\left(\dfrac{1-\sqrt{5}}{2}\right)^{t+1}\end{cases}$ , where $C_1(t)$ and $C_2(t)$ are arbitrary periodic functions with unit period