How is an operator applied to a wavefunction in quantum mechanics?

It is the same as matrix mathematics. In general quantum mechanics is linear algebra in funny hats.

That is, suppose I want to compute $\frac 12 \langle \hat X \hat P + \hat P \hat X\rangle,$ the closest Hermitian observable to the moment $\langle x p \rangle$ in classical mechanics. The relation that $[\hat X, \hat P] = i\hbar$ is usually taken to mean that in the position basis, $\hat X = (x\cdot)$ while $\hat P = -i\hbar \frac{\partial}{\partial x},$ so the first part of this integral is:$$\langle \hat X \hat P\rangle = -i\hbar \int_{-\infty}^\infty dx~\Psi^*(x)\cdot x\cdot\frac{\partial\Psi}{\partial x},$$ while the second part is $$\langle \hat P \hat X\rangle = -i\hbar\int_{-\infty}^\infty dx~\Psi^*(x)\cdot \frac{\partial}{\partial x}\big(x \cdot \Psi(x)\big).$$ The juxtaposition $\hat X~\hat P$ is really a sort of operator composition, just like how the matrix multiplication between two matrices forms a composition of the transforms that they describe.