Limit of a sequence of integrals involving continued fractions

If I were taking that exam, I'd speculate covergence and write the integrand for $A_\infty$ as $$ S_\infty(x) = \frac{1}{ 1 + \frac{1}{1+\frac{1}{1+\frac{1}{\ddots} }}} = \frac{1}{1+S_\infty(x)}$$ Solve the resulting quadratic for $S_\infty^2(x) + S_\infty(x) -1 = 0$ for $S_\infty(x)=\frac{-1+\sqrt{5}}{2}$. Then we immediately have $A_\infty = S_\infty$.

Then, I'd sit there and wonder what they intended for me to actually show on a freshman calculus exam.

As requested by the OP in a comment deleted by a moderator, switching of limit and integral sign is avoided as it requires a higher-than-expected level of knowledge for justification. Thus, a ‘simpler’ approach is presented.

By the substitution $t=\sqrt x$, $$A_n=\int^1_0f_n(t^2)(2tdt)$$

$f_n(t^2)$ is of the form $$f_n(t^2)=\frac{a_n+b_nt}{c_n+d_nt}$$

We have the recurrence relation $$a_{n+1}=c_n$$ $$b_{n+1}=d_n$$ $$c_{n+1}=a_n+c_n$$ $$d_{n+1}=b_n+d_n$$

Or $$c_{n+1}=c_n+c_{n-1}$$ $$d_{n+1}=d_n+d_{n-1}$$

which are the Fibonacci recurrence with initial conditions $$c_0=1, c_1=1$$ $$d_0=0, d_1=1$$

I think you can now proceed.

Also, the general term of Fibonacci sequence $0,1,1,\cdots$ is $$\frac{\phi^n-\overline\phi^n}{\sqrt5}$$ where $\phi=\frac{1+\sqrt 5}2$.

Let $A_n = \int\limits_0^1 f_n(x) dx$. First show that each $f_n$ is a monotonically increasing or decreasing function in the range $[0,1]$, and hence $A_n$ lies between $f_n(0)$ and $f_n(1)$. Then note that

$f_n(0) = 1, \frac{1}{2}, \frac{2}{3}, \frac{3}{5}, \dots$

are the convergents of the continued fraction expansion of $\phi$, and $f_n(1) = f_{n+1}(0)$. So we have

$\frac{1}{2} \le A_1 \le 1$

$\frac{1}{2} \le A_2 \le \frac{2}{3}$

$\frac{3}{5} \le A_3 \le \frac{2}{3}$

and so on.

So the sequence $A_n$ is squeezed between consecutive convergents of the continued fraction expansion of $\phi$. And so $\lim_{n \to \infty} A_n = \phi$.