How does an outer product really work?

The appeal of the bra-ket notation is exactly that "it just fits that way" - everything works out like it should if you just "match" bras and kets.

But as a beginner, it is good to keep in mind what actually happens in the background. Let us go through your example:

• A "ket" $|\psi\rangle$ is just a vector in the Hilbert space. In the following, I will not use bra-ket notation and just write $\psi \in \mathcal H$.

• There is a canonical isomorphism between a Hilbert space and its dual. This is the mathsy way of saying the following: Given a vector $\phi \in \mathcal H$, the function $\phi^\ast: \mathcal H \to \mathbb C$ defined as $$ \phi^\ast(\psi) = \langle \phi, \psi \rangle \tag 1 $$ is a linear map. In physics, we write the bra $\langle \phi |$ for $\phi^\ast$.

• An operator $\hat B$ is defined by how it acts on a vector of the Hilbert space. In your case, that is $$ \hat B(\mu) = \phi^\ast(\mu)\, \psi = \psi\, \phi^\ast(\mu) . \tag 2 $$ This is a linear map $\mathcal H \to \mathcal H$ by definition.

Writing (2) as $\hat B = \psi\, \phi^\ast$ is now a quite obvious abbreviation. A physicist would write $\hat B = |\psi \rangle\!\langle \phi|$ and it "fits".