Integral of $\int_0^{\pi/2} \ (\sin x)^7\ (\cos x)^5 \mathrm{d} x$

To integrate the function $$f(x)=\sin ^{n}x\cdot\cos ^{m}x,$$ when $n$ or $m$ are positive odd numbers, we can apply a general technique which consists of expanding $f(x)$ into a sum of terms of the form $$\sin ^{p}x\cdot \cos x,\qquad p=1,2,\ldots $$ or $$\cos ^{q}x\cdot \sin x,\qquad q=1,2,\ldots.$$ Using the identity $$\begin{equation*} \cos ^{2}x=1-\sin ^{2}x, \end{equation*}$$ in the form $$\begin{equation*} \cos ^{4}x=(1-\sin ^{2}x)^2 \end{equation*}=1-2\sin ^{2}x+\sin ^{4}x,$$

we rewrite our $$f(x)=\sin ^{7}x\cdot\cos ^{5}x=\sin ^{7}x\cdot\cos ^{4}x\cdot\cos x$$ as $$\begin{eqnarray*} f(x) &=&\sin ^{7}x\cdot \left( 1-2\sin ^{2}x+\sin ^{4}x\right) \cdot \cos x \\ &=&\sin ^{7}x\cdot \cos x-2\sin ^{9}x\cdot \cos x+\sin ^{11}x\cdot \cos x. \end{eqnarray*}$$

Each term is of the form $\sin ^{p}x\cdot \cos x$ and can easily be integrated by the substitution $u=\sin x$, $u^{\prime }=\cos x$, $du=\cos x\;dx=u'\;dx$: $$\begin{eqnarray*} \int \sin ^{p}x\cdot \cos x\;dx &=&\int u^{p}\;du=\frac{u^{p+1}}{p+1}=\frac{\sin ^{p+1}x}{p+1}+C, \\ \int_{0}^{\pi /2}\sin ^{p}x\cdot \cos x\;dx &=&\frac{1}{p+1}. \end{eqnarray*}$$

Added: detailed computation in view of OP's comment

$$\begin{eqnarray*} \int_{0}^{\pi /2}f(x)dx &=&\int_{0}^{\pi /2}\sin ^{7}x\cos ^{5}xdx \\ &=&\int_{0}^{\pi /2}\sin ^{7}x\cdot \cos xdx-2\int_{0}^{\pi /2}\sin ^{9}x\cdot \cos xdx \\ &&+\int_{0}^{\pi /2}\sin ^{11}x\cdot \cos xdx \\ &=&\frac{1}{8}-2\cdot \frac{1}{10}+\frac{1}{12}=\frac{1}{120}. \end{eqnarray*}$$

Integration by parts is not the method to use here. $u$-substitution is the method to use. This is perhaps not so obvious, except that as you noticed, IBP doesn't seem to get you very far right off.

You have $\displaystyle \int \sin x (1 - \cos^3 x) \cos ^5 x dx = \int \sin x \cos ^5 x dx - \int \sin x \cos ^8 x dx$

For the first, note that $-\sin x$ is the dervative of $\cos x$, and use $u$-substitution. Do the same for the second. Does that make sense?

EDIT

I see now that the rewritten formula in the OP is not actually correct. $\sin^6 x = (\sin^2 x)^3 = (1 - \cos ^2 x)^3$. So you must substitute this (instead of $1 - \cos ^3$) into the integral and proceed. My work above still gives the way to the answer. You might also work from $\sin x \sin^4 x$ instead - and it might even save you some time.

Further Edited for an example

I see some confusion. But let's suppose we had $\int \sin x \cos^2 x dx$. I know that $-\sin x = \frac{d}{dx} \cos x$, so if I let $u = \cos x$, $du = -\sin x dx$, then we have that

$$\int \sin x \cos ^2 x dx = -\int u^2 du$$

And we know how to calculate this.

$$-\int u^2 du = -\frac{u^3}{3} + C$$

As $u = \cos x$, ths actually says that

$$\int \sin x \cos ^2 x dx = -\frac{\cos^3 x}{3} + C$$

So in this way, there is not a string of $u$-substitutions, but just one. Your integrals can be handled similarly. I encourage you to update us with your work.

If ...you actually meant $\,\,\displaystyle{\int_0^{\pi/2}\sin^7x\cos^5x\,\, dx}\,\, $ then, putting $\,s:= \sin x\,\,,\,c:=\cos x\,$ , we get:$$\int s^7c^5=\frac{1}{2}\int 2sc(1-s^2)^3c^4=\frac{1}{8}c^4(1-c^2)^4+\frac{1}{2}\int c^3s(1-c^2)^4 --\text{ int. by parts, with}$$$$\,u=c^4\,,\,u'=-4c^3s\,\,,\,\,v'=2sc(1-c^2)^3\,,\,v=\frac{(1-c^2)^4}{4}$$ Here we can again put things as before within the integral: $$c^3s(1-c^2)^4=\frac{1}{2}\left[2cs(1-c^2)^4c^2\right]$$and do integration by parts again. I'll leave this for you.