Distinction between 'adjoint' and 'formal adjoint'

Are you talking about differential operators on functions over a domain in $\mathbb{R}^d$? (This is the context in which the phrase "formal adjoint" usually comes up.)

The idea is that working with, say, smooth functions with compact support, we have the integration by parts formula

$$ \int Du\cdot v ~dx + \int u\cdot Dv~ dx = 0 $$

So if a linear partial differential operator is defined as $P = \sum A_\alpha D^\alpha$ where $\alpha$ are multi-indices, you can write $P'$ as a linear partial differential operator $P'\phi = \sum (-1)^{|\alpha|} D^\alpha(A_\alpha \phi)$ and generalize the integration by parts formula

$$ \int Pu \cdot v~dx = \int u \cdot P'v~ dx $$

which looks, in form, suspiciously like the adjoint with respect to the $L^2$ inner product. That is, writing $\langle,\rangle$ for the $L^2$ inner product of real valued functions,

$$ \langle Pu,v\rangle = \langle u, P'v\rangle $$

The reason that we call this a formal adjoint is because, technically, to take an adjoint (in the Hilbert space sense, there is also a different notion for Banach spaces) of an operator, you need to specify which Hilbert space you are working over. In the case of the formal adjoint, it is left unspecified: indeed, the formula only really hold for sufficiently smooth function decaying sufficiently fast at infinity, and not in general for arbitrary functions $u,v\in L^2$.

In general for differential operators, the operator itself will not be bounded on an $L^2$ Hilbert space, and so the operator is only densely defined on your Hilbert space. Therefore the adjoint can only be defined on another subset of the Hilbert space, the domain of the adjoint. (In the most general cases, the domain of the adjoint can be a much, much smaller set [even finite dimensional], so does not make much sense as an operator on the original Hilbert space. For differential operators, the adjoint is still densely defined using the density of $C^\infty_0$ in $L^2$.) (Note that also if the spatial domain has a boundary, the integration by parts formula picks up a boundary term in general, so you pick up a further problem with the notion of adjoints, related to the fact that $C^\infty_0(\Omega)$ is not dense in the Sobolev space $W^{1,2}(\Omega)$ when $\Omega$ has boundary.)

While the word "formal" is, I think, not mentioned explicitly, a lot of the problems that can arise when you deal with unbounded operators are discussed in chapter 8 of Reed-Simon, "Methods of mathematical physics".