Definition of an operator in quantum mechanics

This is what computer scientists would call (ad-hoc) polymorphic or “overloaded” functions: basically, an operator $X$ on the Hilbert space $\mathcal{H}$ is not just one function $X: \mathcal{H}\to\mathcal{H}$, but a family of two functions $$ X = \{ X_{\mathcal{H}}, \quad X_{\mathcal{H}^\ast} \} $$ with $$ X_{\mathcal{H}}: \mathcal{H}\to\mathcal{H}, \quad X_{\mathcal{H}^\ast}: \mathcal{H}^\ast\to\mathcal{H}^\ast $$ Since a Hilbert space's dual space (the “space of bra-vectors”) is a different space from $\mathcal{H}$ itself, it's always unambiguous which of the two functions you mean when applying $X$ to either a ket or a bra.

And the definition of $X_{\mathcal{H}^\ast}$ follows directly from the one of $X_{\mathcal{H}}$ vice versa. This is easy to see in one direction: $$ \bigl(X_{\mathcal{H}^\ast}(\langle f|)\bigr)(|v\rangle) = \langle f| \bigl(X_\mathcal{H}(|v\rangle)\bigr) $$ In the other direction, we need to invoke the Riesz representation theorem: for any $|v\rangle \in \mathcal{H}$, let $$\begin{align} \langle X_v^\ast| & \in\mathcal{H}^\ast \\ \langle X_v^\ast| &:= X_{\mathcal{H}^\ast}\bigl(\langle . , v\rangle\bigr). \end{align}$$ where by $\langle . , v\rangle$ I mean the function $$\begin{align} v^\ast & \in \mathcal{H}^\ast \\ v^\ast(w) & := \langle w,v\rangle \end{align}$$ (This is now not an application of dual-vectors, but the actual scalar product that comes with the Hilbert space!)

Then Riesz tells us that this corresponds to one unique element $X_v\in\mathcal{H}$, so we can define $$ X_{\mathcal H}(|v\rangle) := X_v. $$ And because physicists are lazy (good trait, although some overdo it), they avoid all the “obvious” parentheses etc., and just assume the reader will know how the operator needs to be applied, which space it lives in, etc.. Swapping between a bra- and a ket-version of a state implicitly always invokes the Riesz representation theorem, but this is seldom talked about.

The action (to the right) on the ket space naturally induces an action (to the left) on the bra space. The bra space is the dual space of the ket space (that is the space of linear functionals over the kets). We can simply define $\left< \psi \right| X$ by its action on kets (or, since it's linear, on a basis of ket space and linear extension): $$ \big(\left< \psi \right| X\big) \left| \phi \right> := \left< \psi \right| \big( X \left| \phi \right>\big).$$ Such definitions are also common in pure mathematics. If you want to be pedantic, you can use some notation, e.g. $\iota(X)$, for die induced operator on the dual space.

Just think of the operator as a matrix (as per your "matrix-elements" tag). Then it operates to the right on a column-vector $\left|\mbox{ket}\right>$, and to the left on a row-vector $\left<\mbox{bra}\right|$