Trig substitution integration of $\int1/(x^2\sqrt{x^2 - 9}) dx$ - stuck on a problem
I'd suggest to use the substitution $x:=3\cosh t$ instead. This leads to $$\eqalign{I&=\int{1\over 9\cosh^2 t\ 3\sinh t}\ 3\sinh t\ dt=\int{1\over 9\cosh^2 t}\ dt\cr &={1\over9}\tanh t+C={\sqrt{\cosh^2 t -1}\over9\cosh t}+C={\sqrt{x^2-9}\over 9 x}+C\ .\cr}$$
Notice first that you dropped a square root in the denominator. Also, $dx = 3 \sec \theta \tan \theta d \theta$. Otherwise, everything looked fine so far:
$$ \int \frac{1}{x^2\sqrt{x^2 - 9}}dx = \int \frac{3 \sec \theta \tan \theta}{9 \sec^2 \theta \sqrt{9 \sec^2 \theta - 9}} d \theta. $$
Now, what is a simpler way to write $\sqrt{9 \sec^2 \theta - 9}$?
I think you made a mistake in substituting.
Added. Or rather, you made an algebra mistake. You seem to have gone from $$\sqrt{9\sec^2\theta - 9}$$ to $$3\sec\theta - 3.$$ That's incorrect. The square root does not distribute over sums and differences; that is, the square root of a difference is not the difference of the square roots (for example, $\sqrt{5} = \sqrt{9-4}$ is not equal to $\sqrt{9}-\sqrt{4} = 3-2=1$).
If $x=3\sec\theta$, then $x^2 - 9 = 9\sec^2\theta - 9 = 9(\sec^2\theta-1) = 9\tan^2\theta$, so that $\sqrt{x^2-9} = \sqrt{9\tan^2\theta} = 3|\tan\theta|$. For your substitution to work, though, you want to restrict $\theta$ to a nice interval where tangent is positive, so you can drop the absolute value bars.