If $f(f(x))=x^2-x+1$, what is $f(0)$?
We have that
$$f(f(f(x)))=f(x)^2-f(x)+1.$$ Since $f(f(0))=1$ we get that $$f(1)=f(0)^2-f(0)+1.$$ That is
$$f(0)^2-f(0)+1-f(1)=0.$$ Repeating the process, since $f(f(1))=1$ we get that $$f(1)=f(1)^2-f(1)+1.$$ That is
$$(f(1)-1)^2=f(1)^2-2f(1)+1=0.$$ Thus, $f(1)=1.$ So it is $$f(0)^2=f(0),$$ from where $f(0)=0$ or $f(0)=1.$
If $f(0)=0$ then $f(f(0))=f(0)=0$ which contradicts $f(f(0))=1.$ So, it must be $f(0)=1.$
$f(f(0))=f(f(1))=1$. Apply $f$ once again: $f(f(f(0)))=f(f(f(1)))=f(1)=f(0)^2-f(0)+1=f(1)^2-f(1)+1$.
That leads to $f(1)=1$, hence $f(0)^2-f(0)=0$ and $f(0)$ can only be $0$ or $1$.
But $f(0)=0$ leads to $f(f(0))=0$, contra $f(f(0))=1$, so $\color{red}{f(0)=1}$.
If such a function $f$ exists, then $f(0) = 1$, but such a function $f$ does not exist. See the paper:
- When is $f(f(x)) = az^2 +bz+c$?
R. E. Rice, B. Schweizer and A. Sklar
The American Mathematical Monthly
Vol. 87, No. 4 (Apr., 1980), pp. 252-263
(Link to PDF not behind the JSTOR paywall)
Edit: Such a function does not exist in $\mathbb{C}$! Or in any algebraically closed field of characteristic zero. But you can have such a function in the reals. See the epilogue of the paper (page 262).