Why can't $\int x(x-3)^8\ dx$ be integrated by parts?

Yes, it can be done by parts, treating $(x-3)^8$ as the expression to integrate and $x$ as the expression to differentiate: $$\int x(x-3)^8\,dx=x\cdot\frac19(x-3)^9-\int\frac19(x-3)^9\,dx=\frac19x(x-3)^9-\frac1{90}(x-3)^{10}+K=\frac1{30}(3x+1)(x-3)^9+K$$ To get to the book answer: $$\frac1{30}(3x+1)(x-3)^9+K=\frac{3x-9+10}{30}(x-3)^9+K=\left(\frac{x-3}{10}+\frac13\right)(x-3)^9+K=\frac{(x-3)^{10}}{10}+\frac{(x-3)^9}3+K$$

Note that the substitution $x \mapsto x \! + \! 3$ moves the $8^\text{th}$ power from the binomial onto the monomial $x$, making the integrand much easier to attack directly:

$$\int x(x-3)^8 \ \text{d}x \ = \ \int (x+3)x^8 \ \text{d}x \ = \ \int x^9 + 3x^8 \ \text{d}x$$

Relative to this, IBP involves more work. Despite inefficient choices for $u$ and $\text{d}v$ relative to Parcly Taxel (+1), the following demonstrates that IBP is not only possible, but can be carried out in more than one way.

$\textbf{Warning: }$Unnecessarily hard, but posting for pedagogical value. If one were asked on a test to solve this integral by parts, we advise—in no uncertain terms—to use Parcly Taxel's choices for $u$ and $\text{d}v$. For illustrative purposes only!

Letting $u = (x-3)^8$ and $\text{d}v = x$, we get:

$$\int x(x-3)^8 \ \text{d}x \ = \ \frac{1}{2}x^2(x-3)^8 - \int4x^2(x-3)^7 \ \text{d}x$$

To solve the newly-created integral, you can proceed similarly with $u = (x-3)^7$ and $\text{d}v = 4x^2$. Continue as such until the binomial disappears with $\text{d}u = \text{d}x$; at this point the final integral of the form $\displaystyle \int v \ \text{d}u$ should look something like $\displaystyle \int cx^8 \ \text{d}x$ for some $c \in \mathbb{R}$. Solve this integral and, lastly, add all of the terms together.