Trigonometic Substitution VS Hyperbolic substitution

This is a great question. I would add yet another option as well: Euler Substitution (which has various subtypes depending on parameters in the square root term).

In most cases, if one works, then all three will work. More generally, if you are performing an integral of the form $$ \int R(x, \sqrt{ax^2 + bx + c}) dx, $$ where $R(x,y)$ is a rational function of $x$ and $y$ (which means that it's a polynomial in $x$ and $y$ divided by another polynomial in $x$ and $y$), then all three techniques will generically work. Combining completing the square, partial fraction decomposition, and integration parts allows this to be stated more formally.

There is a problem in practice, though. Reducing a problem to the point where we need to perform partial fraction decompositions is great for numerics, but can be bad for exact solutions since we may need to decompose a polynomial whose factors we don't know exact forms for. (But we do know the partial fraction decomposition exists).

That's starting to get away from your question, though. A good starting rule is to always use trig instead of hyperbolic trig, since you are likely more familiar with rational integrals of trigonometric functions.

Some calculus instructors mention that if a trig sub yields integration of a power of $\sec x$ [which students often have trouble with because they often fall into the "tricky" double integration-by-parts category], you can instead do a hyperbolic trig sub and (often) get an "easier" integral. It is challenging to actually say it is "easier," since there is a strict algorithm for handling integrals of powers of $\sec x$ and for more general products of trig functions. So is it really easier to avoid it?

I once wrote a blog post on trig sub vs. hyperbolic trig sub vs. Euler sub. I perform one integral using all three techniques to see how the feel different, and talk a bit about motivation. If I were to boil everything down to a single heuristic, it's that there should be roughly a "Conservation of Difficulty" between these techniques.