Trig Subsitution When There's No Square Root

You are doing $(r\sec\theta)^3=r^6\sec^6\theta$. Oops! ;-)

---

There's a slicker way to do it.

Get rid of the $r$ with $x=ru$ to begin with, so your integral becomes $$ \frac{A}{r}\int_{a/r}^{\infty}\frac{1}{(1+u^2)^{3/2}}\,du $$ Now let's concentrate on the antiderivative $$ \int\frac{1}{(1+u^2)^{3/2}}\,du= \int\frac{1+u^2-u^2}{(1+u^2)^{3/2}}\,du= \int\frac{1}{(1+u^2)^{1/2}}\,du-\int\frac{u^2}{(1+u^2)^{3/2}}\,du $$ Do the second term by parts $$ \int u\frac{u}{(1+u^2)^{3/2}}\,du= -\frac{u}{(1+u^2)^{1/2}}+\int\frac{1}{(1+u^2)^{1/2}}\,du $$ See what happens? $$ \int\frac{1}{(1+u^2)^{3/2}}\,du=\frac{u}{(1+u^2)^{1/2}}+c $$ which we can verify by direct differentiation.

Now $$ \left[\frac{u}{(1+u^2)^{1/2}}\right]_{a/r}^{\infty}=1-\frac{a/r}{(1+(a/r)^2)^{1/2}} =1-\frac{a}{(r^2+a^2)^{1/2}} $$ and your integral is indeed $$ \frac{A}{r}\left(1-\frac{a}{\sqrt{r^2+a^2}}\right) $$

Firstly you made an error in the first line of working $$(r\sec{(\theta)})^3=r^3\sec^3{(\theta)}$$ Secondly, you need to change the range of integration after performing a substitution. If $\theta=\arctan{(\frac{x}{r})}$ then the limits should change as $x=a \implies \theta=\arctan{(\frac{a}{r})}$ also $x=\infty \implies \theta=\frac{\pi}2$.