Derivative of a bounded Integral?

$$f (x)=x^2+x\int_0^xf (t)dt-\int_0^xtf (t)dt $$ and by FTC, $$f'(x)=2x+\int_0^xf (t)dt+xf (x)-xf (x)$$ $$=2x+\int_0^xf (t)dt $$

$$f''(x)=2+f (x) $$

$$f (x)=Ae^x+Be^{-x}-2$$ and since $f (0)=f'(0)=0$, we find

$$f (x)=e^x+e^{-x}-2=4\sinh^2 (\frac {x}{2}) $$

You're on the right track, see this for more details. You're right about the $(x-x) f(x)$ term, but the second term should be $\int_0^x \frac{\partial}{\partial x} (x-t) f(t) dt = \int_0^x f(t) dt$. The ODE you derive turns out to be second order.

Note $$ f(x) = x^2 + \int_0^x(x − t)f(t) dt=f(x) = x^2 + x\int_0^x f(t)dt-\int_0^x tf(t) dt. $$ So $f(0)=0$. Taking derivative gives $$　f'(x) = ２x + \int_0^x f(t)dt+xf(x)- xf(x)=2x+\int_0^xf(t)dt $$ and hence $f'(0)=0$. Taking derivative again gives $$　f''(x) = 2+f(x). $$ So $f(x)$ satisfies the following 2nd order DE $$ y''-y=2,y(0)=y'(0)=0 $$ whose solution is $$ y=e^x+e^{-x}-2. $$