Evaluating elliptic integrals
This question has been out for a while, and I think it deserves a thorough answer, even tho I came quite late to this party.
As both the classical Legendre-Jacobi theory and the Carlson theory have been mentioned by other users, I'll treat the OP's integral from both viewpoints.
Legendre-Jacobi
The OP came pretty close to using the correct substitution. One thing that could have been done instead is to recall the Pythagorean identity $1+\tan^2 u=\sec^2 u$, so that the substitution that should have been used is $x=a \tan^2 u$, where $a=2$ or $a=3$. Taking the smaller value of $a$, and after some amount of algebra, we obtain
$$\begin{align*}\int_0^\infty\frac{\mathrm dx}{\sqrt{x(x+2)(x+3)}}&=\int_0^{\pi/2}\frac{2}{\sqrt{3-\sin^2u}}\mathrm du\\ &=\frac2{\sqrt{3}}\int_0^{\pi/2}\frac{\mathrm du}{\sqrt{1-\frac13\sin^2u}}\\&=\frac2{\sqrt{3}}K\left(\frac13\right)\end{align*}$$
where I use the parameter convention for elliptic integrals. (This is the same convention used in Abramowitz and Stegun. Relatedly, see here for an extended discussion on the notational confusion surrounding elliptic integrals.)
Had we chosen the substitution with $a=3$ instead, we would have instead obtained the result $\sqrt{2}K\left(-\frac12\right)$, which is equivalent through the imaginary modulus transformation
$$K(-m)=\frac1{\sqrt{1+m}}K\left(\frac{m}{m+1}\right),\quad m>0$$
Carlson
In Carlson's theory, there is the general hypergeometric function
$$R_{-a}(b_1,\dots,b_k;z_1,\dots,z_k)=\frac1{\mathbf B\left(a,-a+\sum_j b_j\right)}\int_0^\infty u^{-a-1+\sum_j b_j}\prod_j \left(u+z_j\right)^{-b_j}\mathrm du$$
where $\mathbf B(x,y)=\frac{\Gamma(x)\Gamma(y)}{\Gamma(x+y)}$ is the usual Euler beta function.
This multivariate hypergeometric function is a homogenized/symmetrized version of the classical Lauricella $F_D$ function (see e.g. this paper where Carlson introduced his function, tho that reference uses an opposite sign convention for $a$).
With this consideration, the OP's original integral can be expressed either as a three-variable Carlson integral, or a two-variable Carlson integral:
$$\begin{align*}\int_0^\infty\frac{\mathrm dx}{\sqrt{x(x+2)(x+3)}}&=2\,R_{-\frac12}\left(\frac12,\frac12,\frac12;0,2,3\right)\\&=\pi\,R_{-\frac12}\left(\frac12,\frac12;2,3\right)\end{align*}$$
The three-variable (incomplete) case occurs often enough that it is given the notation
$$\begin{align*}R_F(x,y,z)&=\frac12\int_0^\infty\frac{\mathrm du}{\sqrt{(u+x)(u+y)(u+z)}}\\&=R_{-\frac12}\left(\frac12,\frac12,\frac12;x,y,z\right)\end{align*}$$
and the two-variable form corresponds to the so-called "complete case",
$$\begin{align*}R_K(x,y)&=R_{-\frac12}\left(\frac12,\frac12;x,y\right)\\&=\frac2{\pi}R_F(0,x,y)\end{align*}$$
In fact, the two-variable form has the integral representation
$$R_K(x,y)=\frac2{\pi}\int_0^{\pi/2}\frac{\mathrm du}{\sqrt{x\cos^2 u+y\sin^2 u}}$$
which one recognizes to be related to the integral representation of Gauss's arithmetic-geometric mean (AGM):
$$R_K(x,y)=\frac1{\operatorname{agm}(\sqrt{x},\sqrt{y})}$$
Thus, the OP's integral is $\pi/\operatorname{agm}(\sqrt{2},\sqrt{3})$.
Additionally, the two-variable Carlson function is also related to the Gauss hypergeometric function, being a homogeneous version of it:
$$R_{-a}(b_1,b_2;x,y)=y^{-a}{}_2 F_1\left({{a,b_1}\atop{b_1+b_2}}\middle|1-\frac{x}{y}\right)$$
so one has
$$\begin{align*}\pi\,R_{-\frac12}\left(\frac12,\frac12;2,3\right)&=\frac{\pi}{\sqrt{3}}{}_2 F_1\left({{\frac12,\frac12}\atop{1}}\middle|\frac13\right)\\&=\frac2{\sqrt{3}}K\left(\frac13\right)\end{align*}$$
where we have used the hypergeometric representation of the complete elliptic integral of the first kind.
It seems to be known as symmetric elliptic integrals of Carlson. Look in the NIST book, 19.15 and further. There are a lot of formulas in it. It seems you seek for exactly the formula 19.22.8 on page 505, note that in it $R_F$ is defined by 19.16.1-19.16.4 and AGM is exactly the complete Legendre integral $K$ as you suggested.