Integration of $\sqrt{x+\sqrt{x^2+3x}}$

we set $$t=\sqrt{x+\sqrt{x^2+3x}}$$ then by squaring, We get $$t^2-x=\sqrt{x^2+3x}$$ by squaring again, We get $$t^4-2t^2x=3x$$ thus $$x=\frac{t^4}{3+2t^2}$$ and $$dx=4\,{\frac {{t}^{3} \left( {t}^{2}+3 \right) }{ \left( 2\,{t}^{2}+3 \right) ^{2}}} dt$$ for the integration use that the integrand can written as $${t}^{2}-{\frac {9}{4\,{t}^{2}+6}}+{\frac {27}{2\, \left( 2\,{t} ^{2}+3 \right) ^{2}}} $$

See the other answer. To find $\int \frac{27}{2(2t^2+3)^2}\, dt$, you can avoid using non-real numbers (notice that $2t^2+3=at^2+bt+c$, where $b^2-4ac<0$, has no real roots and no non-trivial factorization over the real numbers. We can't use partial fractions with real numbers).

See this link (link) (Wikipedia (link) also has some formulas), which shows the integration formulas over the real numbers of $\int \frac{1}{(x^2+bx+c)^n}\, dx$, $\int \frac{x}{(x^2+bx+c)^n}\, dx$, and see the examples there, in particular the example $\int \frac{1}{(x^2+1)^2}\, dx$. It generalizes. Notice that

$$\left(\frac{1}{\left(x^2+bx+c\right)^n}\right)'=\frac{-(2x+b)n}{\left(x^2+bx+c\right)^{n+1}}$$

$$\left(\frac{x}{\left(x^2+bx+c\right)^n}\right)'=\frac{x^2+bx+c-nx(2x+b)}{\left(x^2+bx+c\right)^{n+1}}$$

Use integration by parts, partial fractions. $$\int \frac{1}{2t^2+3}\, dt$$

$$\int u\, dv=uv-\int v\, du$$

$$u=\frac{1}{2t^2+3}$$

$$du=\frac{-4t}{(2t^2+3)^2}\, dt$$

$$dv=dt, v=t$$

$$\int \frac{1}{2t^2+3}\, dt=\frac{t}{2t^2+3}-$$

$$-\int \frac{-4t^2}{(2t^2+3)^2}\, dt$$

Use partial fractions. You could use WolframAlpha (link) if you want, but it's not needed.

$$\frac{-4t^2}{(2t^2+3)^2}=\frac{6}{(2t^2+3)^2}-\frac{2}{2t^2+3}$$

Now find $\int \frac{1}{(2t^2+3)^2}\, dt$ in terms of $$\int\frac{1}{2t^2+3}\, dt=\frac{1}{3}\sqrt{\frac{3}{2}}\int\frac{d\left(\sqrt{\frac{2}{3}}t\right)}{\left(\sqrt{\frac{2}{3}}t\right)^2+1}=$$

$$=\frac{1}{\sqrt{6}}\arctan\left(\sqrt{\frac{2}{3}}t\right)+C$$