Integration using "method of judicious guessing"
Of course one can say: The expertise of the author led him to this "Ansatz". Note that you cannot make a mistake by trying it. If the true solution is not in the bag of proposed functions the computation will reveal this.
Your own efforts resulted in the recursion formula $$\int x^m\log^n x\>dx=\langle{1\over m+1}x^{m+1}\log^n x\rangle-{n\over m+1}\int x^m \log^{n-1} x\>dx\qquad(n\geq1)\ .$$ Here $m$ is fixed, and $n$ is the recursion variable, and the $\langle\ldots\rangle$ notation means "up to an additive constant". It is then pretty obvious, and can be verified by a simple induction proof, that the complete solution will be of the form $$\int x^m\log^n x\>dx=\bigl\langle x^{m+1}\sum_{k=0}^n c_k\log^{n-k}x\bigr\rangle\tag{1}$$ with certain rational coefficients $c_k$, e.g., $c_0={1\over m+1}$. Knowing this it is perhaps simpler to start right away with the "Ansatz" $(1)$ with indetermined coefficients $c_k$ than going through an $n$ steps of the recursion with all its integral signs.
But there is also an " abstract nonsense" proof that the proposed approach works: The result shows that it is not sufficient to consider "monomials" of the form $x^m\log^n x$ and integrate them in a single step. For a general theory we have to deal with the full space of functions $f$ of the form $$f(x)=x^m p(\log x)\qquad(x>0)\ ,\tag{2}$$ where $p(t)$ is a polynomial in an "indeterminate" $t$.
Denote the vector space of all polynomials $p(t)$ of degree $\leq n$ by $P_n$, and denote the vector space of all functions $(2)$ with $p\in P_n$ by $V_{m,n}$. For an $f\in V_{m,n}$ one computes $$f'(x)=x^{m-1}\bigl(m p(\log x)+p'(\log x)\bigr)\ ,\tag{3}$$ which implies that $f'\in V_{m-1,\, n}\>$.
This shows that the derivation $D:\>f\mapsto f'$ maps $V_{m,n}$ to $V_{m-1,\, n}$. The problem at hand is about the converse direction: We are given a $g\in V_{m-1,\,n}$ and have to show that $g$ is the $G'$ of some $G\in V_{m,n}$, in other words: that $$D:\>V_{m,n}\to V_{m-1,\, n}\ ,\qquad f\mapsto f'\tag{4}$$ is surjective.
In $(3)$ we see a linear map $\psi:\>P_n\to P_n$ at work, namely $$\psi:\quad P_n\to P_n,\qquad p\mapsto mp +p'\ .$$ When $m>0$ this map is injective, because $\psi(p)=0$, i.e., $p'=-mp$, implies $p=0$, or $p$ would be an exponential function. As $P_n$ is finite dimensional we can conclude that $\psi$ is also surjective, hence $(4)$ is surjective as well. This is saying that any $g\in V_{m-1, \,n}$ is the derivative of some $G\in V_{m,\,n}$. In particular this is the case for the function $g(x):=x^{m-1}\,(\log x)^n$.
More or less, when the author of the textbook makes their guess of the form of the integral, they are eliding exactly the work that you did to calculate the integral.
Essentially, there is an inductive argument at work here. So, the critical observation that the textbook asks you to make is the following one: $$f_{m+1,n}'=m\cdot f_{m,n}+n\cdot f_{m,n-1}.$$ The idea is then that if you want to integrate $f_{m,n}$, what you need to do is the following:
Calculate an anti-derivative for $f_{m,n-1}$.
Subtract $n$ times that anti-derivative from $f_{m+1,n}$ and divide by $m$.
The idea then is that, if you repeat this argument until you get to calculating an anti-derivative for $f_{m,0}$ - which is just some multiple of $f_{m+1,0}$ - you see that the answer should be some weighted sum of the following functions $$f_{m+1,n},\,f_{m+1,n-1},\,f_{m+1,n-2},\ldots,f_{m+1,2},\,f_{m+1,1},\,f_{m+1,0}.$$ This is just what happens when you "unroll" the process to figure out the antiderivative of $f_{m,n}$ by using the antiderivative for $f_{m,n-1}$, which can be determined by knowing the antiderivative for $f_{m,n-2}$ and so on.
Basically, the textbook runs this argument by figuring out what the terms will be, ignoring the factors of $n$ and $m$ that appear, then figuring out what the coefficients have to be at the end. Your method does the same idea, but you explicitly keep track of the constants at each step.
The clever step here is in seeing that $f_{m+1,n}'=m\cdot f_{m,n} + n\cdot f_{m,n-1}$. Once you have that, you see that you can relate your desired anti-derivative to a simpler one, and then proceed. I would say that the "judicious guess" is this step. As you discovered, there are multiple ways to proceed once you get to this step - and whatever you do, it's a somewhat straightforward calculation from that point on.
Let me try to give a little insight into my thinking process when I see problems like that.
Objective: Compute explicitly integral $\,I=\int x^3\log^2 x\,dx$ by "trial and error".
Interpretation: Authors want you to guess "general form" of solution (meaning they want you to plug in some expression that you think might be solution, but without necessarily knowing exact coefficients, which can be adjusted later).
Hint: find the derivative of $\,f_{mn}\left(x\right)=x^m\log^nx$ and guess general solution of integral $\,I=\int x^3\log^2 x\,dx$ from there.
Interpretation: We need to guess a function whose derivative is $\, x^3\log^2 x$. Since the result is product of powers of $x$ and $\log x$, it is reasonable to assume that the integrand has similar form, with maybe more terms and higher powers, but it should still have product of powers of $x$ and $\log x$ in order for its derivative to have required form.
Approach to Solution
Following the hint, we write down derivative of $\,f_{mn}\left(x\right)$ as $f'_{mn}(x)=mx^{m-1}\log^n{x}+nx^{m-1}\log^{n-1}{x}$. Thus if we want the derivative of function to be $x^3\log^2 x$ we have $ m-1 = 3$ as well as $n = 2$ or $n-1=2$.
Moreover, the derivative of $f_{mn}(x)$ has more than one term, thus if we seek for expression of whose derivative is $x^3\log^2 x$ we need to take linear combination of functions of the form $f_{mn}$ hoping that redundunt terms will cancel out.
More specifically, we see that in the integrand power of $\log x$ is smaller by $1$ than the power of $ x$, thus we conclude that we want to get rid of the term $x^{m-1}\log^{n-1}{x}$ since it has the same powers.
From here on you can use trial and error to try to match and cancel redundant terms.