Evaluating $\int \frac{\sec^2 x}{(\sec x + \tan x )^{{9}/{2}}}\,\mathrm dx$

You turn out correct that this integral can be tackled by using substitution $t=\tan x$. To make this more fun, let us rewrite the integral in general form as follows \begin{equation} I_n=\int\frac{\sec^2x}{\left(\sqrt{1+\tan^2x}+\tan x\right)^n}\,dx\qquad,\qquad n>1 \end{equation} Our original problem is for $n=\frac{9}{2}$. Now, let $t=\tan x$, we then have \begin{equation} I_n=\int\frac{dt}{\left(\sqrt{1+t^2}+t\right)^n} \end{equation} Make a substitution using hyperbolic functions either $t=\sinh y$ or $t=\cosh y$, we have \begin{align} I_n=\int\frac{\cosh y}{\left(\sqrt{1+\sinh^2y}+\sinh y\right)^n}\,dy&=\int\frac{\cosh y}{\left(\cosh y+\sinh y\right)^n}\,dy\\ &=\frac{1}{2}\int\frac{e^{y}+e^{-y}}{e^{ny}}\,dy\\ &=\frac{1}{2}\int\left[e^{-(n-1)y}+e^{-(n+1)y}\right]\,dy\\ &=-\frac{e^{-(n-1)y}}{2(n-1)}-\frac{e^{-(n+1)y}}{2(n+1)}+C \end{align} The rest part is trivially done by using identities $\cosh y\pm\sinh y=e^{\pm y}$ and $\cosh^2 y-\sinh^2 y=1$, hence \begin{equation} I_n=-\frac{(\sec x-\tan x)^{n-1}}{2(n-1)}-\frac{(\sec x-\tan x)^{n+1}}{2(n+1)}+C \end{equation} Interestingly, we have a nice closed-form for the following definite integral \begin{equation} I_n=\int_0^{\pi/2}\frac{\sec^2x}{\left(\sec x+\tan x\right)^n}\,dx=\frac{n}{n^2-1} \end{equation}

$\bf{My\; solution::}$ Given $$\displaystyle \int\frac{\sec^2 x}{(\sec x+\tan x)^{\frac{9}{2}}}dx$$

Let $$(\sec x+\tan x)= t\;,$$ Then $$\left(\sec x\cdot \tan x+\sec^2 x\right)dx = dt $$

So $$\displaystyle \left(\sec x+\tan x\right)dx = \frac{dt}{\sec x}\Rightarrow \displaystyle dx = \frac{dt}{\sec x}$$

Now Using $$\bullet \displaystyle (\sec^2 x-\tan^2 x) = 1\Rightarrow (\sec x+\tan x)\cdot (\sec x-\tan x) = 1$$

So $$\displaystyle (\sec x-\tan x) = \frac{1}{t}$$ and we above substitute $$\displaystyle (\sec x+\tan x)=t$$

So we get $$\displaystyle \sec x = \frac{1}{2}\left(t+\frac{1}{t}\right)$$

Now our Integral convert into $$\displaystyle \frac{1}{2}\int\frac{\sec x}{t^{\frac{9}{2}}}dt = \frac{1}{2}\int \frac{t+\frac{1}{t}}{t^{\frac{9}{2}}}dt = \frac{1}{2}\int t^{-\frac{7}{2}}dt+\frac{1}{2}\int t^{-\frac{11}{2}}dt$$

So we get $$\displaystyle \frac{1}{2}\cdot -\frac{2}{5}\cdot t^{-\frac{5}{2}}+\frac{1}{2}\cdot -\frac{2}{9}\cdot t^{-\frac{9}{2}}+\mathcal{C}$$

So $$\displaystyle \int\frac{\sec^2 x}{(\sec x+\tan x)^{\frac{9}{2}}}dx = -\frac{1}{5}\cdot (\sec x+\tan x)^{-\frac{5}{2}}-\frac{1}{9}\cdot (\sec x+\tan x)^{-\frac{9}{2}}+\mathcal{C}$$

Given $$\displaystyle \int\frac{\sec^2 x}{(\sec x+\tan x)^{\frac{9}{2}}}dx = \frac{1}{2}\int\frac{\sec x(\sec x+\tan x)+\sec x(\sec x-\tan x)}{(\sec x+\tan x)^{\frac{9}{2}}}dx$$

$$\displaystyle = \frac{1}{2}\int\frac{\sec x\cdot (\sec x+\tan x)}{(\sec x+\tan x)^{\frac{9}{2}}}dx+\frac{1}{2}\int\frac{\sec x\cdot (\sec x-\tan x)}{(\sec x+\tan x)^{\frac{9}{2}}}dx$$

Now Use $\sec^2x-\tan^2 x= 1$ For Second Integral, We Get

$$\displaystyle = \frac{1}{2}\int\frac{\sec x\cdot (\sec x+\tan x)}{(\sec x+\tan x)^{\frac{9}{2}}}dx+\frac{1}{2}\int\frac{\sec x\cdot (\sec x+\tan x)}{(\sec x+\tan x)^{\frac{13}{2}}}dx$$

Now For both Integral , Let $\sec x+\tan x=t\;,$ Then $\sec x(\sec x+\tan x)dx = dt$

We Get $$\displaystyle = \frac{1}{2}\int\frac{1}{t^{\frac{9}{2}}}dt+\frac{1}{2}\int\frac{1}{t^{\frac{13}{2}}}dt = -\frac{1}{2}\cdot \frac{2}{7}t^{-\frac{7}{2}}-\frac{1}{2}\cdot \frac{2}{11}t^{-\frac{11}{2}}+\mathcal{C}$$

So We Get $$\displaystyle \int\frac{\sec^2 x}{(\sec x+\tan x)^{\frac{9}{2}}}dx = -\frac{1}{7}(\sec x+\tan x)^{-\frac{9}{2}}-\frac{1}{11}(\sec x+\tan x)^{-\frac{11}{2}}+\mathcal{C}$$