Impossible integral?

You are correct that this does mean there are no continuous functions satisfying the equation. (There's a quicker way to see this: set $x=1$, then $LHS=0$, but $RHS=5$.)

Suppose $f(t)=at^2+bt+c$

we have $$\int_0^{x^2-1} \left(a t^2+b t+c\right) \, dt=$$ $$=x^6+x^4+3 x^2\equiv \frac{a x^6}{3}+x^4 \left(\frac{b}{2}-a\right)+x^2 (a-b+c)-\frac{a}{3}+\frac{b}{2}-c$$ and $$ \begin{cases} \frac{a }{3}=1\\ \frac{b}{2}-a=1\\ a-b+c=3\\ -\frac{a}{3}+\frac{b}{2}-c=0\\ \end{cases} $$ the system has no solution, therefore there is no polynomial $f(t)=at^2+bt+c$ which satisfies the equation.