Closed form for the limit of the iterated sequence $a_{n+1}=\frac{\sqrt{(a_n+b_n)(a_n+c_n)}}{2}$

Here is an observation: As in your link, if $b_0 = c_0$ then we can prove recursively that $b_n = c_n$ for all $n \geq 0$. Then plugging this to OP's recurrence relation, we find that the sequence $(a_n, b_n)$ satisfies the Schwab-Borchardt recurrence relation

$$ a_{n+1} = \frac{a_n + b_n}{2}, \qquad b_{n+1} = \sqrt{a_{n+1}b_n}. $$

So if we write $(a_0, b_0) = (a, b)$, then the limit is given by the Schwab-Borchardt mean

$$ \lim_{n\to\infty} a_n = \lim_{n\to\infty} b_n = SB(a_0, b_0) = \begin{cases} \dfrac{\sqrt{a^2 - b^2}}{\operatorname{arccosh}(a/b)}, & a > b \\ \dfrac{\sqrt{b^2 - a^2}}{\operatorname{arccos}(a/b)}, & a < b \\ a, & a = b \end{cases} \tag{*} $$

---

Anyway, let me write down my failed attempt to mimic Carlson's proof of $\text{(*)}$ so that future me do not repeat this mistake.

Define $I(a,b,c)$ by

$$ I(a,b,c) := \lim_{R\to\infty} \int_{-R}^{R} \frac{dx}{(x+a^2)^{1/3}(x+b^2)^{1/3}(x+c^2)^{1/3}}, $$

where $z^{1/3} = \exp(\frac{1}{3}\log z)$ with the branch cut $-\pi < \arg(z) \leq \pi$. Then we can check that $I(a, b, c)$ converges. Now using the substitution $x \mapsto 4x + ab + bc + ca$, we find that

$$ I(a, b, c) = I\left( \frac{\sqrt{(a+b)(a+c)}}{2}, \frac{\sqrt{(b+c)(b+a)}}{2}, \frac{\sqrt{(c+a)(c+b)}}{2} \right). $$

This tells us that $I(a_n,b_n,c_n) = \cdots = I(a_0,b_0,c_0)$. If we recall how the AGM is computed from Landen's transformation, this may possibly allow us to analyze $L$ using $I$.

Well, it turns out that the function $I$ has a serious issue, which is that $I$ is identically zero! This can be checked either by using the fact $I(a,a,a) = 0$ or by interpreting $I(a,b,c)$ as a value of the Schwarz–Christoffel mapping.