Show convergence of $\int_1^\infty{\frac{dx}{\sqrt{x^3-x}}}$

In cases like these, it may be useful to look at the asymptotic behaviour. You say you couldn't find a bounding function, but if it were $\frac{1}{\sqrt{x^3+x}}$ I'm quite certain you could.

The fact is that you almost immediately want to compare with some kind of $\frac{1}{\sqrt{x^3}}$. So what you can do is show that this is asymptotically the same as $\frac{1}{\sqrt{x^3+x}}$, in the sense that $\frac{1/\sqrt{x^3+x}}{1/\sqrt{x^3}} \to 1$ as $x \to +\infty$. With this fact at hand, you now have that for $x$ greater than a certain $K$, it is true that $\frac{1/\sqrt{x^3+x}}{1/\sqrt{x^3}} < 2$, say. Therefore, $\frac{1}{\sqrt{x^3+x}}<\frac{2}{\sqrt{x^3}}$ for those $x>K$, which gives you enough to conclude the exercise. This strategy also is easily generalizable for other cases. (Note that it isn't necessary for the limit to actually be $1$. It suffices for it to exist as a real number.)

Note that for $x>1$, we have that $2<x(x+1)$ and $(x-1)^2<x(x+1)$ therefore $$\frac{1}{\sqrt{x^3-x}}=\frac{1}{((x-1)x(x+1))^{1/2}}<\frac{1}{(2(x-1))^{1/2}}$$ and $$\frac{1}{\sqrt{x^3-x}}=\frac{1}{((x-1)x(x+1))^{1/2}}<\frac{1}{((x-1)^3)^{1/2}}.$$ Can you take if from here?

You don't need a bounding function if you've found an equivalent function near $1$ and near $\infty$ (in general they're not the same) for which the integral functions $\int_1^a$ and $\int_a^\infty$ converge.

• Near $\infty$, $x^3-x\sim x^3$, so $\;\dfrac1{\sqrt{x^3-x}}\sim_\infty \dfrac1{x^{3/2}}$, and the integral to $\infty$ of this function converges.
• Near $1$, set $x=t-1$. The integral becomes $$\int_1^a\frac{\mathrm d x}{\sqrt{x^3-x}}=\int_0^{a+1}\!\!\!\frac{\mathrm dt}{\sqrt{2t-3t^2+t^3}}$$ and near $t=0$, we have $\;\dfrac{\mathrm 1}{\sqrt{2t-3t^2+t^3}}\sim\dfrac{\mathrm 1}{\sqrt{2t}}$, which has a convergent integral.