$I=\int_1^\infty \frac{dx}{x\sqrt{x^2-1}}$Integrating this

This can be done by a change of variables which simplifies the integrand \begin{eqnarray} y=\sqrt{x^2-1}\\ y^2=x^2-1, \ x=\sqrt{1+y^2}\\ ydy=xdx\\ y(1)=0,\ y(\infty)=\infty. \end{eqnarray}

We can now write $$ I=\int_0^\infty \frac{dy}{1+y^2}=\tan^{-1} y\big|^\infty_0=\frac{\pi}{2}. $$

I see 2 problems. The first is you didn't change the limits of integration to terms of $\theta$. The second is you have your trig identity wrong.

$$\sec^2\theta-1=\tan^2\theta$$

Therefore, your substitution should be

$$x=\sec\theta,dx=\sec\theta\tan\theta d\theta$$

$$\theta=\cos^{-1}\frac1x$$

Now when we substitute we get

$$\int_0^\frac\pi2\frac{\sec\theta\tan\theta d\theta}{\sec\theta\tan\theta}$$

From here, the solution should be trivial.