Integral $I=\int \frac{dx}{(x^2+1)\sqrt{x^2-4}} $

Use $x=2\cosh t$, so the integral becomes $$ \int\frac{1}{1+4\cosh^2t}\,dt= \int\frac{1}{1+e^{2t}+2+e^{-2t}}= \int\frac{e^{2t}}{e^{4t}+3e^{2t}+1}\,dt $$ and then set $u=e^{2t}$ so you get $$ \frac{1}{2}\int\frac{1}{u^2+3u+1}\,du $$ that can be computed by partial fractions.

HINT:

Your integral:

$$\text{I}=\int\frac{1}{\left(x^2+1\right)\sqrt{x^2-4}}\space\text{d}x=$$

---

Subsitute $x=2\sec(u)$ and $\text{d}x=2\tan(u)\sec(u)\space\text{d}u$.

Then $\sqrt{x^2-4}=\sqrt{4\sec^2(u)-4}=2\tan(u)$ and $u=\text{arcsec}\left(\frac{x}{2}\right)$:

---

$$\int\frac{\sec(u)}{4\sec^2(u)+1}\space\text{d}u=\int\frac{\cos(u)}{5-\sin^2(u)}\space\text{d}u=$$

---

Substitute $s=\sin(u)$ and $\text{d}s=\cos(u)\space\text{d}u$:

---

$$\int\frac{1}{5-s^2}\space\text{d}s=\frac{1}{5}\int\frac{1}{1-\frac{s^2}{5}}\space\text{d}s=$$

---

Substitute $p=\frac{s}{\sqrt{5}}$ and $\text{d}p=\frac{1}{\sqrt{5}}\space\text{d}s$:

---

$$\frac{1}{\sqrt{5}}\int\frac{1}{1-p^2}\space\text{d}p=\frac{\text{arctanh}\left(p\right)}{\sqrt{5}}+\text{C}=$$ $$\frac{\text{arctanh}\left(\frac{s}{\sqrt{5}}\right)}{\sqrt{5}}+\text{C}=\frac{\text{arctanh}\left(\frac{\sin\left(u\right)}{\sqrt{5}}\right)}{\sqrt{5}}+\text{C}=$$ $$\frac{\text{arctanh}\left(\frac{\sin\left(\text{arcsec}\left(\frac{x}{2}\right)\right)}{\sqrt{5}}\right)}{\sqrt{5}}+\text{C}=\frac{\text{arctanh}\left(\frac{\sqrt{x^2-4}}{x\sqrt{5}}\right)}{\sqrt{5}}+\text{C}$$

The curve $t^2=x^2-4$ is a hyperbola, which can be parametrized by a single value. Rewrite this equation as $(x+t)(x-t)=4$, and set $y=x+t$. Then $4/y=x-t$, and we have $$ x=\frac{y+\frac{4}{y}}{2},\;\;\;\;t=\frac{y-\frac{4}{y}}{2}. $$ Compute $dx=(1/2-2 y^{-2})dy$, and the integral becomes $$ \int \frac{dx}{(x^2+1)\sqrt{x^2-4}}=\int \frac{\frac{1}{2}-\frac{2}{y^2}}{\left(\left(\frac{y+\frac{4}{y}}{2}\right)^2+1\right)\left(\frac{y-\frac{4}{y}}{2}\right)}dy=\int \frac{4y\, dy}{y^4+12 y^2+16}. $$ This can be computed using partial fractions.