How to integrate $ \int\frac{x-2}{(7x^2-36x+48)\sqrt{x^2-2x-1}}dx$?

$$I=\int \frac{x-2}{(7x^2-36x+48)\sqrt{x^2-2x-1}}\,dx$$ This can be simplifies using $$\frac{x-2}{7x^2-36x+48}=\frac 1{7(a-b)}\left(\frac{a-2 } {x-a }+\frac{2-b } {x-b } \right)$$ where $$a=\frac{2}{7} \left(9-i \sqrt{3}\right) \qquad \text{and} \qquad b=\frac{2}{7} \left(9+i \sqrt{3}\right) $$ which makes that we are facing two integrals $$I_c=\int \frac {dx} {(x-c)\sqrt{x^2-2x-1}}$$ Complete the square and let $x=1+\sqrt 2 \sec(t)$ which gives $$I_c=\int \frac{dt}{(1-c) \cos (t)+\sqrt{2}}$$ Now, using the tangent half-angle subtitution $$I_c=2\int\frac{du}{\left(c+\sqrt{2}-1\right) u^2-c+\sqrt{2}+1}=\frac{2 }{\sqrt{-c^2+2 c+1}}\tan ^{-1}\left(u\frac{\sqrt{c+\sqrt{2}-1} }{\sqrt{-c+\sqrt{2}+1}}\right)$$ and so on ....

Remember: For such questions, It is always better to differentiate the options in an MCQ format exam.

Since you'd like an approach to integrate this, here goes:)

$$\int\frac{x-2}{(7x^2-36x+48)\sqrt{x^2-2x-1}}\, dx = \int\frac{(x-1-1)\,dx}{(7x(x-1)-29(x-1)+19)\sqrt{(x-1)^2-2}}$$

Put $x-1 \rightarrow v$, $$ = \int\frac{v-1}{(7v^2-22v+19)\sqrt{v^2-2}}\, dv$$ Now put $v \rightarrow \sqrt2\sec y$, $dv = \sqrt 2 \sec y\tan y\,dy$ $$ = \int\frac{(\sqrt2\sec y-1)\sec y}{14\sec^2y - 22\sqrt2\sec y+ 19}\,dy$$ $$ = \frac{1}{7\sqrt2}\int\frac{\sec^2y - (11\sqrt2/7)\sec y + 19/14 + (15\sqrt2/7)\sec y-19/14}{\sec^2y - (11\sqrt2/7)\sec y+ 19/14}\,dy$$ $$ = \frac {y}{7\sqrt2} + \frac{15}{49}\int\frac{\sec y - 19/(30\sqrt2)}{\sec^2y - (11\sqrt2/7)\sec y+ 19/14}\, dy$$ Our denominator here doesn't have real roots. The way ahead would be to factorize the denominator as $(\sec y - a)(\sec y - b)$ where $a$ and $b$ are the complex and then use partial fractions.

By inspection and from the options too, it is clear that the antiderivative has to be of the following form with constants $a$, $b$ and $c$ remaining to be determined.

$$c\arctan\left(\frac{\sqrt{x^2-2x-1}}{ax+b}\right)$$

Differentiation yields the following expression, comparing it with the integrand gives equations in $a$, $b$ and $c$, solving which gives the antiderivative.

$$\frac{c\left(\left(b+a\right)x-b+a\right)}{\sqrt{x^2-2x-1}\left(\left(a^2+1\right)x^2+\left(2ab-2\right)x+b^2-1\right)}$$

But obviously the answer had to be something different from the ones given in the options, because the term $\sqrt{x^2-2x-1}$ has to remain put inside the $\arctan()$.