Evaluate integral $\int (x^2-1)(x^3-3x)^{4/3} \mathop{dx}$

If you multiply and divide by $3$, you get $$ \int (x^2 -1)(x^3 - 3x)^{4/3}dx = \frac{1}{3}\int (3x^2-3)(x^3-3x)^{4/3} dx $$ changing variable to $u = x^3 - 3x$ you have $du = (3x^2 - 3x)dx$ so $$ \begin{split} \int (x^2 -1)(x^3 - 3x)^{4/3}dx &= \frac{1}{3}\int (3x^2-3)(x^3-3x)^{4/3} dx\cr &= \frac{1}{3} \int u^{4/3} du \cr &= \frac{1}{3} \times \frac{3u^{7/3}}{7} + C \cr &= \frac{1}{7} (x^3 - 3x)^{7/3} + C \cr \end{split} $$

I agree with the other answers. My response is long-winded so...

Often when attacking indefinite integrals, you will immediately suspect that a substitution [i.e. $u = g(x)$] is needed, but won't be sure which substitution to try.

I have to ask the OP:
Why did you think that $x = \sec \theta$ was the right substitution? Had you recently been exposed to problems that seemed similar where $x = \sec \theta$ was the right substitution?

The point of my response/rant is to develop the OP's intuition. Since the integral contains $(x^3 - 3x)^{(4/3)},$ my first guess as to the right substitution to try would be $u = (x^3 - 3x).$ This would convert this portion of the integral to $u^{(4/3)}.$

The idea is that (as a first guess for the right substitution), I would be hoping that (except for the $u^{(4/3)}$ factor), the remainder of the integral would be a polynomial in $u$, where each term has an integer exponent.

As I say, the point of my response is simply to expand the OP's intuition (and perspective).