a problem in functional analysis or operator theory

Hint Proceed in two steps, the strategy being to find all the functions $u$ such that $T(u)=\lambda.u$ and then conclude.

First step (Find a condition - with derivatives - that solutions of $T(u)=\lambda u$ must satisfy and find one of them) If you set $$F(x):=\int_{0}^x u(\sqrt{s})ds\ .$$ Then $T(u)=F(x^2)$. If $u$ is s.t. $$T(u)=\lambda u\qquad (1),$$ taking derivatives, one gets. $$ \lambda u'(x)=T(u)'=2xF'(x^2)=2x.u(x)\qquad (2) $$ One first supposes $\lambda\not=0$. Then if $u$ satisfies (1), it satisfies (2) and $$ u'=\frac{2x}{\lambda}.u\qquad (3) $$ then, we have $u=e^{\frac{x^2}{\lambda}}$ as particular solution (call this solution $Z_\lambda$).

Second step (Find all solutions of $T(u)=\lambda u$ must satisfy and conclude) Consider any solution of $T(u)=\lambda u$. We form $y=(Z_\lambda)^{-1}u=e^{-\frac{x^2}{\lambda}}u$ and compute. As it must satisfy (3), one has $$ y'=(-\frac{2x}{\lambda})e^{-\frac{x^2}{\lambda}}u+e^{-\frac{x^2}{\lambda}}u'=(-\frac{2x}{\lambda})e^{-\frac{x^2}{\lambda}}u+e^{-\frac{x^2}{\lambda}}(\frac{2x}{\lambda})u=0 $$ then $y=c$ constant and $u=c.e^{\frac{x^2}{\lambda}}$. Now $$ T(u)=\int_{0}^{x^2} u(\sqrt{s})ds=c.\int_{0}^{x^2} e^{\frac{s}{\lambda}}ds=c.[\lambda.e^{\frac{s}{\lambda}}]_0^{x^2}=\lambda.c.(e^{\frac{x^2}{\lambda}}-1) $$ if $T(u)=\lambda.u$ this forces $\lambda.c=0$ and then $c=0$ hence the claim.

For $\lambda=0$, from (2), one gets $2x.u(x)=0$ and then $u\equiv 0$.

Remark (After Ranc's post) The operator $T$ is the conjugate of a Volterra operator (but not, in the usual sense, a Volterra operator). The conjugacy factor being the isometry $u\to U(u)$ with $U(u)(x)=u(x^2)$ ($T=UWU^{-1}$).

It's much easier than solving ODE:

Note that $ T(u)(x)=2\int_{0}^{x} su (s) ds $,

Hence, for each $0<\lambda\leq 1$ you have $$ \sup_{x\in [0,\lambda] }| T (u)(x)|\leq 2\lambda sup_{x\in [0,\lambda]}|u (x)| $$ which shows that any eigenvalue must be smaller than $2\lambda $ for all $\lambda\in (0,1]$. In other words, zero is the only possible eigenvalue.

To see that zero isn't an eigenvalue, note that $[T(u)(x)]'=2xu (x) $. Hence, if $ T (u)\equiv 0$, then $ 2x u (x)=0$ for all $ x $, so $ u (x) =0$ for all $ x> 0$ and, by continuity, also for $ x=0$

I would like to suggest a solution that is more functional-analysis oriented, which is based on analysis of the spectrum of Volterra operators. This can be finished in a sentence:

It turns out $T$ is Volterra operator from $C([0,1])$ to itself, so the spectrum is $\{0\}$, but $0$ is not an eigenvalue.

Ofcourse, this is not a hint, nor it is easy, but I decided it is reasonable to show it, since you already got many good hints and answers. I will outline most of the proof, a major part (Neumann series THM) is omitted, but it is not difficult if you are familiar with some operator theory over Banach spaces.

Proof: Note $Tf(x) = \int_0 ^{x^2} f(\sqrt{y}) \mathrm{d}y = \int_0 ^x 2 f(\tau)\tau \mathrm{d}\tau$ which is to say that $T$ is a Volterra operator (and we can say alot about these). Recall that Neumann THM about operator series: Suppose $T\colon X \rightarrow X$ is an operator defined on a Banach space $X$, then if $\sum_{n=0}^\infty\| T^n \| < \infty$ we have $I-T$ is invertible, and its inverse is $\sum_{n=0}^\infty T ^n$.

Let us evaluate $\|T^n\|$, (recall $x\in(0,1)$): $$|Tf(x)|=\left| \int_0 ^x 2\tau f(\tau) \mathrm{d}\tau \right| \leq \int_0 ^x |2 f(\tau)| \mathrm{d}\tau \leq \int_0^x2\|f\|_\infty \mathrm{d}\tau=2\|f\|_\infty x $$ which gives a point-wise bound for $Tf(x)$ and also shows $\|T\| \leq 2$. Let us evaluate $T^2$ in a similar manner:

$$|T^2f(x)|= \left| \int_0^x 2\tau \cdot Tf(\tau) \mathrm{d}\tau \right| \leq \int_0^x |2\tau \cdot Tf(\tau) |\mathrm{d}\tau\leq 2^2\|f\|_\infty \int_0^x\tau \mathrm{d}\tau = 2^2 \|f\|_\infty \frac{x^2}{2}$$

which again gives a point-wise bound $|T^2f(x)| \leq 2\|f\|_\infty x^2$ and $\|T^2\| \leq 2\|f\|_\infty$. Inductive reasoning shows $$|T^nf(x)| \leq 2^n \|f\|_\infty \cdot \frac{x^{n}}{n!}$$ that gives $\|T^n\| \leq 2^n/(n-1)!$, so the series converge and $I-T$ is invertible, and $T-I$ is invertible.

Take $\lambda \in \mathbb{C} \setminus{0}$, and look at $T-\lambda I$. Is it invertible? Yes. $T-\lambda I = \lambda ( \lambda^{-1} T -I)$, so it is enough to show $ \lambda^{-1} T -I$ is invertible, but it is (since $\lambda^{-1}T$ is Volterra operator).

Since the spectrum $\sigma(T)$ is not empty - it must contain the point $\{0\}$, and we are left to determine if $0$ is an eigenvalue.

Although not immediate, showing $0$ is not an eigenvalue is easy, and I take the privilige to stop here.