How to integrate $\int_{0}^{2} \sqrt{x+\sqrt{x+\sqrt{x+\cdots}}}\,dx$?

Let $L(x)=\sqrt{x+\sqrt{x+\cdots}}$.

$L(x)^2=x+L(x)$, hence, $L(x)^2-L(x)-x=0$ and $L(x)=\frac{1\pm\sqrt{1+4x}}{2}$.

Because $L(x)\ge0$, $L(x)=\frac{1+\sqrt{1+4x}}{2}$ and then the intergal is straightforward.

Your approach is fine. Just note that $x(u)=u^2-u$ maps $[1,2]$ onto $[0,2]$, and therefore, after the substitution, the lower limit of integration is $1$. Hence $$\int_{0}^{2} \sqrt{x+\sqrt{x+\sqrt{x+\cdots}}}dx=\int_{1}^{2}u(2u-1)du=\frac{19}{6}.$$