Methods for choosing $u$ and $dv$ when integrating by parts?

My general principle is "Which bit gets nicer to work with when differentiated than when integrated?", which roughly lines up with the LIATE approach:

• The derivative of $\ln x$ is $\frac{1}{x}$, which will interact nicely with polynomial terms, whereas the integral is some weird rubbish on the order of $x \ln x$ which looks terrible.
• The derivatives of the inverse trig functions tend to be rational functions or square roots of rational functions, which again tend to interact with polynomial terms decently, whereas the integrals are horrible things with logarithms in them.
• Algebraic stuff differentiates into other algebraic stuff, and if you're lucky it will get simpler as it goes (but not always).
• Trig and exponential functions tend to sit at about the same complexity no matter how much you differentiate or integrate them. So if you're left picking them as your $u$ it probably means you're going to be setting up a recurrence relation of some kind.

There is an acronym called "LIATE":

Set $u$ to be the first function you see on this list (ordered):

1. logarithm
2. inverse trigonometric function
3. algebraic function
4. trigonometric function
5. exponential

Doesn't always work perfectly, but it's your best bet.

In your first integral, the algebraic function $x^2$ takes precedence.

In the second, the logarithm $\ln x$ takes precedence.

Forget cute (or weird) acronyms or tables. You are focusing on the wrong thing: differentiation is easy, while integration is hard, so you should ask not what to pick as $u$, but what to pick as $dv$. The answer is: choose as $dv$ the most complicated expression in the integrand that you currently know how to integrate.

For example, you asked about integrating $x^2e^x$. Between $x^2$ and $e^x$ the factor $e^x$ is more sophisticated and you can integrate it, so let $dv = e^x dx$ and then $u = x^2$. You also asked about integrating $\sqrt{x}\ln x$. For students the antiderivative of $\sqrt{x}$ is known but the antiderivative of $\ln x$ is not, so let $dv = \sqrt{x} dx$ and then $u = \ln x$.

When this tip of how to pick $dv$ rather than $u$ was passed on to me, I never had a problem applying integration by parts afterwards. This method requires no memorization of rules, but just experience integrating to recognize one factor as being more complicated than another.