How to proceed $\int \frac{dx}{(x-2) \left(1+\sqrt{7 x-10-x^2}\right)} $

If you want to use integration by substitution, you'll need to determine what you're substituting for, so that you can actually follow through with the substitution.

Added: In this case, it seems that you're wanting to rewrite $\sqrt{7x-10-x^2}$ in the form $(x-2)t,$ which turns out to be workable, unlike what I thought you were trying to do, and turns out to be much simpler than what I ended up doing instead.

Since $7x-10-x^2=(5-x)(x-2)$ is nonnegative exactly when $x\in[2,5],$ then $$\sqrt{7x-10-x^2}=\sqrt{5-x}\sqrt{x-2}=\sqrt{\frac{5-x}{x-2}}(x-2)$$ except at the single point $x=2,$ which doesn't alter the antiderivative. Hence, we want $$t=\sqrt{\frac{5-x}{x-2}}.$$ Now, for $x\in(2,5],$ we have $$(x-2)t=\sqrt{5-x}\sqrt{x-2}\\(x-2)^2t^2=(5-x)(x-2)\\(x-2)t^2=5-x\\xt^2-2t^2=5-x\\x+xt^2=5+2t^2\\x\left(1+t^2\right)=5+2t^2\\x=\frac{5+2t^2}{1+t^2}.$$ From there, we can see that $$x-2=\frac{5+2t^2}{1+t^2}-\frac{2+2t^2}{1+t^2}=\frac{3}{1+t^2},$$ whence $$1+\sqrt{7x-10-x^2}=1+(x-2)t=1+\frac{3t}{1+t^2}=\frac{1+3t+t^2}{1+t^2}.$$ Finally, applying the quotient rule shows us that $$dx=\frac{4t\left(1+t^2\right)-2t\left(5+2t^2\right)}{\left(1+t^2\right)^2}\,dt=\frac{-6t}{\left(1+t^2\right)^2}\,dt.$$

At this point, we're ready to follow through on the substitution, finding that $$I=\int\frac1{x-2}\cdot\frac1{1+\sqrt{7x-10-x^2}}\,dx\\I=\int\frac{1+t^2}{3}\cdot\frac{1+t^2}{1+3t+t^2}\cdot\frac{-6t}{\left(1+t^2\right)^2}\,dt\\I=-\int\frac{2t}{1+3t+t^2}.$$ This is definitely nicer looking!

Since $$3t+t^2=(3+t)t=\left(\frac32+t+\frac32\right)\left(\frac32+t-\frac32\right)=\left(\frac32+t\right)^2-\left(\frac32\right)^2=\left(\frac32+t\right)^2-\frac94,$$ then $$1+3t+t^2=\left(\frac32+t\right)^2-\frac94=\left(\frac32+t\right)^2-\left(\frac{\sqrt 5}2\right)^2=\left(\frac32+t+\frac{\sqrt 5}2\right)\left(\frac32+t-\frac{\sqrt 5}2\right),$$ and so $$I=-\int\frac{2t}{\left(\frac32+t+\frac{\sqrt 5}2\right)\left(\frac32+t-\frac{\sqrt 5}2\right)}\\I=-\int\frac{2t}{\left(t+\frac12\left(3+\sqrt 5\right)\right)\left(t+\frac12\left(3-\sqrt 5\right)\right)}.$$

The next step I'd use is called partial fraction decomposition, in which we suppose that $$\frac{2t}{\left(t+\frac12\left(3+\sqrt 5\right)\right)\left(t+\frac12\left(3-\sqrt 5\right)\right)}=\frac{A}{t+\frac12\left(3+\sqrt 5\right)}+\frac{B}{t+\frac12\left(3-\sqrt 5\right)}\tag{$\diamondsuit$}$$ for some constants $A$ and $B,$ then determine what we can about those constants. Multiplying both sides by the left-hand side's denominator gives us $$2t=A\left(t+\frac12\left(3-\sqrt 5\right)\right)+B\left(t+\frac12\left(3+\sqrt 5\right)\right)\\2t+0=(A+B)t+\frac32(A+B)+\frac{\sqrt 5}2(B-A).$$ As both sides are polynomials, the only way for equality to hold is if the corresponding coefficients are equal. Hence, we need $A+B=2,$ meaning that $A=2-B.$ Substituting back in gives us $$2t+0=2t+3+\frac{\sqrt 5}2(2B-2)\\2t+0=2t+3+(B-1)\sqrt 5,$$ so we also need $$0=3+(B-1)\sqrt 5\\-3=(B-1)\sqrt 5\\-\frac3{\sqrt 5}=B-1\\1-\frac3{\sqrt 5}=B\\B=-\frac{3-\sqrt 5}{\sqrt 5},$$ so $$A=2-B\\A=2+\frac{3-\sqrt 5}{\sqrt 5}\\A=\frac{3+\sqrt 5}{\sqrt 5}.$$ Thus, by $(\diamondsuit),$ we find that $$I=-\frac{3+\sqrt 5}{\sqrt 5}\int\frac{1}{t+\frac12\left(3+\sqrt 5\right)}\,dt+\frac{3-\sqrt 5}{\sqrt 5}\int\frac{1}{t+\frac12\left(3-\sqrt 5\right)}\,dt\\I=-\frac{3+\sqrt 5}{\sqrt 5}\ln\left|t+\frac12\left(3+\sqrt 5\right)\right|+\frac{3-\sqrt 5}{\sqrt 5}\ln\left|t+\frac12\left(3-\sqrt 5\right)\right|+C.$$ Noting that $t$ is nonnegative on $(2,5]$ and that $3-\sqrt 5>0,$ we see then that $$I=-\frac{3+\sqrt 5}{\sqrt 5}\ln\left(t+\frac12\left(3+\sqrt 5\right)\right)+\frac{3-\sqrt 5}{\sqrt 5}\ln\left(t+\frac12\left(3-\sqrt 5\right)\right)+C.$$ Finally, we recall our logarithm rules to get $$I=-\ln\left(\left(t+\frac12\left(3+\sqrt 5\right)\right)\left(t+\frac12\left(3-\sqrt 5\right)\right)\right)+\frac{3}{\sqrt 5}\ln\left(\frac{t+\frac12\left(3-\sqrt 5\right)}{t+\frac12\left(3+\sqrt 5\right)}\right)+C\\I=-\ln\left(t^2+3t+1\right)+\frac{3}{\sqrt 5}\ln\left(\frac{2t+3-\sqrt 5}{2t+3+\sqrt 5}\right)+C\\I=-\ln\left(\frac{1}{t^2+3t+1}\right)+\frac{3}{\sqrt 5}\ln\left(\frac{2t+3-\sqrt 5}{2t+3+\sqrt 5}\right)+C\\I=-\ln\left(\frac13\cdot\frac{3}{1+t^2}\cdot\frac{1+t^2}{t^2+3t+1}\right)+\frac{3}{\sqrt 5}\ln\left(\frac{2t+3-\sqrt 5}{2t+3+\sqrt 5}\right)+C\\I=-\ln\left(\frac13\cdot\frac{x-2}{1+\sqrt{7x-10-x^2}}\right)+\frac{3}{\sqrt 5}\ln\left(\frac{2t+3-\sqrt 5}{2t+3+\sqrt 5}\right)+C\\I=-\ln\left(\frac{x-2}{3+3\sqrt{7x-10-x^2}}\right)+\frac{3}{\sqrt 5}\ln\left(\frac{2t+3-\sqrt 5}{2t+3+\sqrt 5}\right)+C.$$ One last substitution gets us $$I=\ln\left(\frac{x-2}{3+3\sqrt{7x-10-x^2}}\right)+\frac3{\sqrt 5}\ln\left(\frac{2\sqrt{\frac{5-x}{x-2}}+3-\sqrt 5}{2\sqrt{\frac{5-x}{x-2}}+3+\sqrt 5}\right)+C\\I=\ln\left(\frac{x-2}{3+3\sqrt{7x-10-x^2}}\right)+\frac3{\sqrt 5}\ln\left(\frac{2\sqrt{5-x}+(3-\sqrt 5)\sqrt{x-2}}{2\sqrt{5-x}+(3+\sqrt 5)\sqrt{x-2}}\right)+C.$$

While we might be able to massage that a bit to make it nicer, that's probably enough time spent on this one problem.


Everything after this point was based on my initial misunderstanding of the OP. While accurate (thanks to a few later corrections), it's very long, and a little embarassing, so feel free to ignore it.

For example, if you want the expression under the radical to have the form $(x-2)t,$ that turns out to be doable, since factoring shows that $$7x-10-x^2=(x-2)(5-x).$$ That means that you'll need $$t=5-x,$$ so that $$x=5-t$$ and $$dx=-dt.$$ Following through with this substitution will then yield $$I=\int\frac{-dt}{(5-t-2)\left(1+\sqrt{(5-t-2)t}\right)}=-\int\frac{dt}{(3-t)\left(1+\sqrt{(3-t)t}\right)}.$$

That's not looking much better, but let's see what we can do with it.

Multiplying the numerator and denominator by $1-\sqrt{(3-t)t}$ will give us $$I=-\int\frac{(1-\sqrt{(3-t)t})dt}{(3-t)\bigl(1-(3-t)t\bigr)},$$ but that isn't a lot better, either. We could try another substitution, say $u=3-t,$ so that $t=3-u$ and $dt=-du,$ which would give us $$I=-\int\frac{-(1-\sqrt{u(3-u)})du}{u\bigl(1-u(3-u)\bigr)}=\int\frac{(1-\sqrt{(3-u)u})du}{u\bigl(1-(3-u)u\bigr)},$$ but while we've made the denominator a little simpler, we've done nothing at all for the numerator, and it still isn't obvious how to proceed.


To go any further, we could be a bit clever, and rewrite the bit under the radical as a difference of two squares. This will allow us to make a special substitution called a trigonometric substitution.

Let's take it back to the original form $$I=\frac{dx}{(x-2)\left(1+\sqrt{7x-10-x^2}\right)}.$$ Now, observe that $$7x-x^2=-(x^2-7x)=-(x-7)x=-\left(x-\frac72-\frac72\right)\left(x-\frac72+\frac72\right).$$ Recalling that $(a-b)(a+b)=a^2-b^2,$ then letting $a=x-\frac72$ and $b=\frac72$ shows us that $$7x-x^2=-\left(\left(x-\frac72\right)^2-\left(\frac72\right)^2\right)=\left(\frac72\right)^2-\left(x-\frac72\right)^2=\frac{49}4-\left(x-\frac72\right)^2.$$ Consequently, $$7x-10-x^2=\frac{49}4-\left(x-\frac72\right)^2-10=\frac94-\left(x-\frac72\right)^2=\left(\frac32\right)^2-\left(x-\frac72\right)^2.$$

We're not quite there, yet, but we're getting closer. Note that $$x-\frac72=\frac32\left(\frac23x-\frac73\right),$$ so that $$7x-10-x^2=\left(\frac32\right)^2\left(1-\left(\frac23x-\frac73\right)^2\right).$$ Here's where we're ready for the clever bit: we want to replace $\frac23x-\frac73$ with $\sin t,$ giving us $$7x-10-x^2=\left(\frac32\right)^2(1-\sin^2t)=\left(\frac32\cos t\right)^2$$ by the Pythagorean Identity, and so $$\sqrt{7x-10-x^2}=\frac32|\cos t|.$$ That means we'll need $$2x-7=3\sin t\\2x=7+3\sin t\\x=\frac72+\frac32\sin t,$$ and so by the chain rule, $$dx=\frac32\cos t\,dt.$$ Also, $$x-2=\frac32+\frac32\sin t=\frac32(1+\sin t).$$ Following through with the substitution, we get $$I=\int\frac{\frac32\cos t\,dt}{\frac32(1+\sin t)\left(1+\frac32|\cos t|\right)}=\int\frac{2\cos t}{(1+\sin t)\bigl(2+3|\cos t|\bigr)}\,dt.$$

But why are we even allowed to do this? Couldn't this substitution make the bit under the radical a negative number? We'll, let's find out if that's possible. Since we require $7x-10-x^2\ge 0,$ then $(x-2)(5-x)\ge0,$ meaning either that $x=2,$ that $x=5,$ or that $x-2$ and $5-x$ have the same sign. Since $x-2$ and $5-x$ only have the same sign when $2<x<5,$ then we need $$2\le x\le 5\\2\le\frac72+\frac32\sin t\le 5\\4\le 7+3\sin t\le 10\\-3\le 3\sin t\le 3\\-1\le\sin t\le1,$$ which is always true! Excellent! Our substitution is safe, after all.

I see that another answer has appeared already, but it doesn't really offer much insight as to why it works or what steps were taken to get there (assuming it was worked out by hand and not just computed with software). With that in mind, I will go ahead and continue my answer.

Unfortunately, the absolute value seems to make things problematic, but let's see what we can do. We'd really like to make sure that $\cos t\ge 0$ with our initial substitution. One way to make it happen is to assume that $t\in[-\pi/2,\pi/2],$ which is fine, since in that interval, $\sin t$ can take on any of its possible values, so we aren't preventing our antiderivative from being valid. Good news! That lets us go with $$I=\int\frac{2\cos t}{(1+\sin t)\bigl(2+3\cos t\bigr)}\,dt.$$

Our next step involves what's known as partial fraction decomposition. In this case, that means trying to find constants $A,$ $B,$ $C,$ $D,$ $E,$ and $F$ so that $$\frac{2\cos t}{(1+\sin t)\bigl(2+3\cos t\bigr)}=\frac{A\cos t+B\sin t+C}{1+\sin t}+\frac{D\cos t+E\sin t+F}{2+3\cos t}.\tag{$\heartsuit$}$$ Multiplying both sides of $(\heartsuit)$ by $(1+\sin t)\bigl(2+3\cos t\bigr)$ gives us $$2\cos t=(A\cos t+B\sin t+C)(2+3\cos t)+(D\cos t+E\sin t+F)(1+\sin t),$$ or $$0=(A\cos t+B\sin t+C)(2+3\cos t)+(D\cos t+E\sin t+F)(1+\sin t)-2\cos t.\tag{$\star$}$$ Note that if we can find values of those constants to make $(\star)$ hold, then they will make $(\heartsuit)$ hold, as well. Unfortunately, that right-hand side is a mess, but expanding it out yields $$3A\cos^2t+E\sin^2t+(3B+D)\cos t\sin t+(2A+3C+D-2)\cos t+(2B+E+F)\sin t+2C+F,$$ which is a bit friendlier. Now we can try out some different values of $t$ to see if any of them simplify things for us. One nice one is $t=-\frac\pi 2,$ which makes $\sin t=-1$ and $\cos t=0,$ so that $(\star)$ becomes $$0=E-(2B+E+F)+2C+F$$ or $$0=2C-2B.$$ Thus, we need $C=B,$ yielding $$\frac{2\cos t}{(1+\sin t)\bigl(2+3\cos t\bigr)}=\frac{A\cos t}{1+\sin t}+\frac{D\cos t+E\sin t+F}{2+3\cos t}+B\tag{$\heartsuit_1$}$$ and making the right-hand side of $(\star)$ equal to $$3A\cos^2t+E\sin^2t+(3B+D)\cos t\sin t+(2A+3B+D-2)\cos t+(2B+E+F)\sin t+2B+F,$$ which can be factored somewhat, allowing us to rewrite $(\star)$ as $$0=\bigl((3\cos t+2)A-2\bigr)\cos t+\bigl((3B+D)\cos t+E\sin t+2B+F\bigr)(1+\sin t).\tag{$\star_1$}$$ Note that we could not have made the substitution $t=-\frac\pi2$ into $(\heartsuit),$ since it would have made two of the denominators equal to $0.$ To simplify $(\star_1)$ further, we actually want to find a way for the other denominator in $(\heartsuit)$ to be made equal to $0$--that is, we want some $t$ such that $\cos t=-\frac23.$ There are many such $t,$ and for such $t$ we have by Pythagorean identity that either $\sin t=\frac{\sqrt 5}3$ or $\sin t=-\frac{\sqrt 5}3.$ Making the substitutions $\cos t=-\frac23$ and $\sin t=\pm\frac{\sqrt 5}3$ into $(\star_1)$ gives us $$0=-\frac23\bigl(0A-2\bigr)+\left(-\frac23(3B+D)+2B+F\pm\frac{\sqrt5}3E\right)\left(1\pm\frac{\sqrt 5}3\right)\\0=\frac43+\left(-\frac23D+F\pm\frac{\sqrt5}3E\right)\left(1\pm\frac{\sqrt 5}3\right)\\0=\frac43+\frac13\left(-2D+3F\pm E\sqrt5\right)\left(1\pm\frac{\sqrt 5}3\right)\\0=4+\left(-2D+3F\pm E\sqrt5\right)\left(1\pm\frac{\sqrt 5}3\right)\\0=4+\frac13\left(-2D+3F\pm E\sqrt5\right)\left(3\pm\sqrt 5\right)\\0=12+\left(-2D+3F\pm E\sqrt5\right)\left(3\pm\sqrt 5\right)\\0=12-6D+9F+E(\pm\sqrt5)^2\pm(-2D+3F+3E)\sqrt 5\\0=12-6D+5E+9F\pm(-2D+3E+3F)\sqrt 5.$$ Now, this is technically not one equation, but two, depending on whether we substituted $\sin t=-\frac{\sqrt 5}3$ or $\sin t=\frac{\sqrt 5}3,$ meaning that we have both $$0=12-6D+5E+9F\pm(-2D+3E+3F)\sqrt 5\tag{1}$$ and $$0=12-6D+5E+9F\pm(-2D+3E+3F)\sqrt 5\tag{2}$$ but the only way for both $(1)$ and $(2)$ to be true is if $-2D+3E+3F=0$--meaning that $D=\frac32(E+F)$--which turns both $(1)$ and $(2)$ into $$0=12-9(E+F)+5E+9F\\0=12-4E.$$ Hence, we need $E=3$ and $D=\frac32(3+F),$ which yields $$\frac{2\cos t}{(1+\sin t)\bigl(2+3\cos t\bigr)}=\frac{A\cos t}{1+\sin t}+\frac{\frac92\cos t+3\sin t}{2+3\cos t}+B+\frac12F\tag{$\heartsuit_2$}$$ and lets us rewrite $(\star_1)$ as follows: $$0=\bigl((3\cos t+2)A-2\bigr)\cos t+\left(\left(3B+\frac92+\frac32F\right)\cos t+3\sin t+2B+F\right)(1+\sin t)$$ $$0=\bigl((6\cos t+4)A-4\bigr)\cos t+\bigl((6B+3F+9)\cos t+6\sin t+4B+2F\bigr)(1+\sin t).\tag{$\star_2$}$$ Next, we can try $t=\frac\pi2,$ which makes $\sin t=1$ and $\cos t=0,$ so that $$0=0+\bigl(0+6+4B+2F\bigr)(2)\\0=12+8B+4F\\F=-2B-3.$$ Thus, we have $$\frac{2\cos t}{(1+\sin t)\bigl(2+3\cos t\bigr)}=\frac{A\cos t}{1+\sin t}+\frac{\frac92\cos t+3\sin t}{2+3\cos t}-\frac32\tag{$\heartsuit_3$}$$ and we can rewrite $(\star_2)$ as follows: $$0=\bigl((6\cos t+4)A-4\bigr)\cos t+\bigl((6B+3(-2B-3)+9)\cos t+6\sin t+4B+2(-2B-3)\bigr)(1+\sin t)$$ $$0=\bigl((6\cos t+4)A-4\bigr)\cos t+(6\sin t-6)(1+\sin t)$$ $$0=\bigl((6\cos t+4)A-4\bigr)\cos t+6\sin^2 t-6$$ $$0=\bigl((6\cos t+4)A-4\bigr)\cos t-6(1-\sin^2 t)$$ $$0=\bigl((6\cos t+4)A-4\bigr)\cos t-6\cos^2 t$$ $$0=\bigl((6\cos t+4)A-6\cos t-4\bigr)\cos t$$ $$0=(6\cos t+4)(A-1)\cos t.$$ Substituting $t=0$ so that $\cos t=1,$ we can immediately conclude that $A=1,$ and so $$\frac{2\cos t}{(1+\sin t)\bigl(2+3\cos t\bigr)}=\frac{\cos t}{1+\sin t}+\frac{\frac92\cos t+3\sin t}{2+3\cos t}-\frac32\\\frac{2\cos t}{(1+\sin t)\bigl(2+3\cos t\bigr)}=\frac{\cos t}{1+\sin t}+\frac{\frac92\cos t+3\sin t}{2+3\cos t}-\frac{\frac32(2+3\cos t)}{2+3\cos t}\\\frac{2\cos t}{(1+\sin t)\bigl(2+3\cos t\bigr)}=\frac{\cos t}{1+\sin t}+\frac{\frac92\cos t+3\sin t}{2+3\cos t}-\frac{\frac92\cos t+3}{2+3\cos t}\\\frac{2\cos t}{(1+\sin t)\bigl(2+3\cos t\bigr)}=\frac{\cos t}{1+\sin t}+\frac{3\sin t}{2+3\cos t}-\frac{3}{2+3\cos t}.$$

At this point, an integral table would really be your friend, but let's see if we can get around it. By the work above, we know that $$I=\int\frac{\cos t}{1+\sin t}\,dt+\int\frac{3\sin t}{2+3\cos t}\,dt-\int\frac{3}{2+3\cos t}\,dt.$$ The first integral is easily found by noting that if $u=1+\sin t,$ then $du=\cos t,$ so we get $$I=\int\frac1u\,du+\int\frac{3\sin t}{2+3\cos t}\,dt-\int\frac{3}{2+3\cos t}\,dt\\I=\ln|u|+\int\frac{3\sin t}{2+3\cos t}\,dt-\int\frac{3}{2+3\cos t}\,dt\\I=\ln|1+\sin t|+\int\frac{3\sin t}{2+3\cos t}\,dt-\int\frac{3}{2+3\cos t}\,dt.$$ The second integral is easily found by noting that if $v=2+3\cos t,$ then $dv=-3\sin t\,dt$ so we similarly get $$I=\ln|1+\sin t|-\ln|2+3\cos t|-\int\frac{3}{2+3\cos t}\,dt.$$ The last integral is more of a pain. To prepare, we can use the double angle identity $\cos 2\theta=2\cos^2\theta-1$ with $\theta=\frac t2$ to show that $$I=\ln|1+\sin t|-\ln|2+3\cos t|-\int\frac{3}{-1+6\cos^2\left(\frac t2\right)}\,dt\\I=\ln|1+\sin t|-\ln|2+3\cos t|-\int\frac{\frac12\sec^2\left(\frac t2\right)}{1-\frac16\sec^2\left(\frac t2\right)}\,dt.$$ Next, we can use the Pythagorean identity $\sec^2\theta=1+\tan^2\theta$ to see that $$I=\ln|1+\sin t|-\ln|2+3\cos t|-\int\frac{\frac12\sec^2\left(\frac t2\right)}{\frac56-\frac16\tan^2\left(\frac t2\right)}\,dt.$$ Next, we make the substitution $w=\tan\frac t2,$ so that $dw=\frac12\sec^2\frac t2\,dt,$ and so $$I=\ln|1+\sin t|-\ln|2+3\cos t|-\int\frac1{\frac56-\frac16w^2}\,dw\\I=\ln|1+\sin t|-\ln|2+3\cos t|-\frac65\int\frac1{1-\frac15w^2}\,dw\\I=\ln|1+\sin t|-\ln|2+3\cos t|-\frac65\int\frac1{1-\left(\frac w{\sqrt 5}\right)^2}\,dw\\I=\ln|1+\sin t|-\ln|2+3\cos t|-\frac6{\sqrt 5}\int\frac{\frac1{\sqrt 5}}{1-\left(\frac w{\sqrt 5}\right)^2}\,dw.$$ A final substitution of $y=\frac w{\sqrt 5}$ then gives us $$I=\ln|1+\sin t|-\ln|2+3\cos t|-\frac6{\sqrt 5}\int\frac1{1-y^2}\,dy\\I=\ln|1+\sin t|-\ln|2+3\cos t|-\frac6{\sqrt 5}\tanh^{-1} y+C\\I=\ln|1+\sin t|-\ln|2+3\cos t|-\frac6{\sqrt 5}\tanh^{-1}\left(\frac{w}{\sqrt 5}\right)+C\\I=\ln|1+\sin t|-\ln|2+3\cos t|-\frac6{\sqrt 5}\tanh^{-1}\left(\frac{\tan\frac t2}{\sqrt 5}\right)+C.$$

At last, we let remember that by our earlier assumptions, $t=\sin^{-1}\left(\frac23 x-\frac 73\right).$ Immediately, $\sin t=\frac23 x-\frac 73,$ so that $$I=\ln\left|\frac23 x-\frac43\right|-\ln|2+3\cos t|-\frac6{\sqrt 5}\tanh^{-1}\left(\frac{\tan\frac t2}{\sqrt 5}\right)+C,$$ and recalling that $x\ge 2$ we see that $$I=\ln\left(\frac23 x-\frac43\right)-\ln|2+3\cos t|-\frac6{\sqrt 5}\tanh^{-1}\left(\frac{\tan\frac t2}{\sqrt 5}\right)+C.$$ Also, with a little Pythagorean work, we can show that $\cos\bigl(\sin^{-1}(z)\bigr)=\sqrt{1-z^2}$ for any $z\in[-1,1],$ so that $$\cos t=\sqrt{1-\left(\frac23 x-\frac73\right)^2}=\sqrt{-\frac49 x^2+\frac{28}9x-\frac{40}9}=\frac23\sqrt{7x-10-x^2},$$ and so $$I=\ln\left(\frac23 x-\frac43\right)-\ln\left|2+2\sqrt{7x-10-x^2}\right|-\frac6{\sqrt 5}\tanh^{-1}\left(\frac{\tan\frac t2}{\sqrt 5}\right)+C\\I=\ln\left(\frac23 x-\frac43\right)-\ln\left(2+2\sqrt{7x-10-x^2}\right)-\frac6{\sqrt 5}\tanh^{-1}\left(\frac{\tan\frac t2}{\sqrt 5}\right)+C.$$ With quite a bit more work, we can show that $\tan\frac{\sin^{-1}(a)}2=\frac{a}{1+\sqrt{1-a^2}}$ for any $a\in[-1,1],$ so that $$\tan\frac t2=\frac{\frac23 x-\frac73}{1+\sqrt{1-\left(\frac23 x-\frac73\right)^2}}=\frac{\frac13(2x-7)}{1+\frac23\sqrt{7x-10-x^2}}=\frac{2x-7}{3+2\sqrt{7x-10-x^2}},$$ and so $$I=\ln\left(\frac23 x-\frac43\right)-\ln\left(2+2\sqrt{7x-10-x^2}\right)-\frac6{\sqrt 5}\tanh^{-1}\left(\frac{\frac{2x-7}{3+2\sqrt{7x-10-x^2}}}{\sqrt 5}\right)+C\\I=\ln\left(\frac23 x-\frac43\right)-\ln\left(2+2\sqrt{7x-10-x^2}\right)-\frac6{\sqrt 5}\tanh^{-1}\left(\frac{(2x-7)\sqrt{5}}{15+10\sqrt{7x-10-x^2}}\right)+C.$$


The moral to the story is that sometimes it's not really worth doing it all out by hand. :-P


Note that $7x-10-x^2 = \frac94-(\frac72-x)^2$. So, substitute $\frac72-x=\frac32\sin t$ to integrate as follows \begin{align} & \int \frac{dx}{(x-2)(1+\sqrt{7x-10-x^2})}\\ = & - \int \frac{2\cos t}{(1-\sin t)(2+3\cos t)}dt\\ = & - \int \left ( \frac{\cos t}{1-\sin t}-\frac {3\sin t}{2+3\cos t} - \frac {3}{2+3\cos t} \right)dt\\ =& \>\ln(1-\sin t) -\ln(2+3\cos t)+\frac3{\sqrt5}\tanh^{-1}\frac{\tan\frac t2}{\sqrt5}+C \end{align}