Integrate $\int_0^{\int_0^{\vdots}\frac{1}{\sqrt{x}}\text{d}x}\frac{1}{\sqrt{x}}\text{d}x$ and monotonicity of integrals

1)

I'll try to make it more rigorous. Let $$ a_0= \int_0^\alpha \frac{1}{\sqrt{x}} \mathrm{d}x = 2 \sqrt{\alpha}\\[3ex] a_{n+1}=\int_0^{a_n}\frac{1}{\sqrt{x}} \mathrm{d}x = 2 \sqrt{a_n} $$ for some $\alpha \ge 0$. We are interested in evalulate

$$L=\lim_{n \to \infty} a_n. $$ From the general equation we have $$L =2 \sqrt{L} $$ so if we suppose the limit exists, then a priori we have three possibilities:

1. $L=0$;
2. $L=4$;
3. $L= +\infty$.

Notice that for the first terms of the sequence we have $$ a_0 = 2 \alpha^{\frac{1}{2}}\\ a_1 = 2 \sqrt{a_0}= 2 \sqrt{2 \alpha^{\frac{1}{2}}} = 2 \cdot 2^{\frac{1}{2}} \cdot\alpha^{\frac{1}{2^2}}\\ a_2 = 2 \sqrt{a_1} = 2 \cdot 2^{\frac{1}{2}} \cdot 2^{\frac{1}{2^2}}\cdot\alpha^{\frac{1}{2^3}} $$

and, in general, $$ a_n=2^{\sum_{i=0}^n \big(\frac{1}{2}\big)^i} \cdot \alpha^{\frac{1}{2^{n+1}}} $$ (note that if $\alpha = 0$ then $a_n = 0$ for every $n > 0$ ).

So, since $$ \sum_{i=0}^n \bigg(\frac{1}{2}\bigg)^i = \frac{1-\frac{1}{2^{n+1}}}{1 - \frac{1}{2}} = 2 - \frac{1}{2^n} $$ (see Formula for geometric series ), we get

$$ a_n = 4 \cdot 2^\frac{1}{2^{n+1}} \cdot \alpha^{\frac{1}{2^{n+1}}} $$ Now $$\lim_{n \to \infty} 2^\frac{1}{2^{n+1}}=2^0=1$$ and hence if $\alpha \neq 0$, $$ \lim_{n \to \infty}a_n = 4 \cdot \lim_{n \to \infty} 2^\frac{1}{2^{n+1}}\cdot \lim_{n \to \infty}\alpha^{\frac{1}{2^{n+1}}} = 4 \cdot\lim_{n \to \infty}\exp \left( \frac{\ln \alpha}{2^{n+1}} \right) = 4 \cdot\exp \left(\ln \alpha\cdot \lim_{n \to \infty} \frac{1}{2^{n+1}} \right)=4 \cdot e^0 = 4 $$ finally

$$ \lim_{n \to \infty} a_n = \begin{cases} 0 & \text{if $\alpha = 0$}; \\ 4 & \text{if $\alpha > 0$} \\ \end{cases} $$

2) Riemann integral has the monotonicity proprierties, see here for more details.

1) is my argument right? I'm not sure if it is rigorous, especially when I "substitute" the upper bound with L; maybe I can approach it with sequences.

The question isn't even properly posed, as

$$\int_0^{\int_0^\vdots\frac1{\sqrt x}~\mathrm dx}\frac1{\sqrt x}~\mathrm dx$$

isn't a meaningful expression. You want to instead consider something such as:

$$a_0=\alpha\ge0\\a_{n+1}=\int_0^{a_n}\frac1{\sqrt x}~\mathrm dx=2\sqrt{a_n}$$

and ask what

$$L=\lim_{n\to\infty}a_n$$

is. It can easily be seen that if $\alpha=0$, then $L=a_n=0$ for all $n$. If $0<a_n<4$, then we have

$$a_n<2\sqrt{a_n}=a_{n+1}<4$$

and if $a_n>4$, then we have

$$4<a_{n+1}=2\sqrt{a_n}<a_n$$

so for any $\alpha>0$, it converges, since $a_n$ is monotone and bounded. Now that you know that it converges, it suffices to solve

$$L=2\sqrt L$$

restricted to the interval that we know the solution is in (from the bounds), which in this case gives $L=4$.

2) In this case the square root was at the denominator so somehow I've excluded the fact that the integrand could be $\ge0$ (if my argument is correct), but in general how can I prove that if $f(x)>0$ then $\int^b_af(x)dx>0$ and not $\int^b_af(x)dx\ge0$ (if this is true)?

This is not true unless $a<b$ and $f$ is Riemann integrable, in which case you can see Is the Riemann integral of a strictly positive function positive?.