Applying an operator to a function vs. a (ket) vector
I really appreciate this question. You are perfectly right and your confusion is understandable. Sadly, the physics world is somewhat sloppy in their use of notation at times.
Of course, when writing $P\psi(x)$ one does not intent to apply the operator to a scalar, but the $P$ is applied to the ket vector $\psi$.
But now comes the major source of confusion, IMO. While to you the wave function really is just
the coefficients of a state vector when the vector is written in a particular basis
and the state vector is element of some abstract Hilbert space, many people will argue that the Hilbert space is actually a function space. Which one precisely depends on the system under consideration but specifically for a free particle it'll be $L^2(\mathbb R,\mathbb C)$, i.e. the space of square-integrable function from $\mathbb R$ to $\mathbb C$. So in this space, the actual function $\psi$ is the vector. That space is linear (a.k.a. a vector space) and it also has a scalar product, defined via the integral. It is also complete which means that one can insert identities as you did above. That's actually what a Hilbert space is (to a mathematician): a complete vector space with a scalar product.
When taking on this view, it is legitimate to apply an operator to a function. Indeed, only now does the identification $P=-i\hbar\partial_x$ make any sense. It is a differential operator which can be applied to functions.
The advantage of the Dirac bra-ket-notation is that it allows you to step back from any concrete realizations of the underlying Hilbert space. This has some advantages:
- The analogy to linear algebra is more pronounced.
- Many general ideas which arise in quantum theory can be formulated independent of the system under investigation. In particular, the formalism is the same for infinite-dimensional Hilbert space like the $L^2$ and finite-dimensional Hilbert spaces like $\mathbb C^2$, the spin-1/2 space.
- Change of basis is very transparent. Specifically the connection of position and momentum space via the Fourier transform comes about naturally.
So why should we bother working in function spaces?
- It leads to differential equations which have a well-understood theory.
- There are some mathematical subtleties$^1$ involved when dealing with infinite-dimensional spaces. Those can best be understood in function spaces.
- Often we need to leave the blessed world of Hilbert spaces and go beyond that. The space is then no longer complete. Delta functions are such an example. They are certainly not square integrable, hell, the square of a delta function is not even defined. In won't go into depth here. The phrase you need to search for is Gelfand space triplet.
$^1$ For an overview of what can go terribly wrong in quantum mechanics upon getting to comfortable with Dirac's notation, I recommend this excellent article: F. Gieres, Mathematical surprises and Dirac's formalism in quantum mechanics, arXiv:quant-ph/9907069
Edit in response to @Jyothi's comment:
The action of the momentum operator as a derivative may be written as $\langle x|\hat P|\psi\rangle = -i\hbar\frac{\partial}{\partial x}\langle x|\psi\rangle$.
Now, consider the following calculation of the action of the operator product $\hat P\hat X$:
$$ \langle x|\hat P\hat X|\psi\rangle = \int\mathrm dx'\,\langle x|\hat P\hat X|x'\rangle\langle x'|\psi\rangle = \int\mathrm dx'\,x'\langle x|\hat P|x'\rangle\langle x'|\psi\rangle\\ = -i\hbar\int\mathrm dx'\,x'\psi(x')\frac{\partial}{\partial x}\delta(x-x')\\ = +i\hbar\int\mathrm dx'\,x'\psi(x')\frac{\partial}{\partial x'}\delta(x-x')\\ =-i\hbar\int\mathrm dx'\,\delta(x-x')\frac{\partial}{\partial x'}(x'\psi(x'))\\ =-i\hbar\frac{\partial}{\partial x}(x\psi(x))\\ = -i\hbar\Bigl(\psi(x)+x\frac{\partial}{\partial x}\psi(x)\Bigr)\\ =-i\hbar\langle x|\psi\rangle + x\langle x|\hat P|\psi\rangle\\ =-i\hbar\langle x|\psi\rangle + \langle x|\hat X\hat P|\psi\rangle\\ =\langle x|(-i\hbar+\hat X\hat P)|\psi\rangle. $$ In the third line, I rewrote a derivative for $x$ as an derivative for $x'$, yielding a minus sign. In the step after, integration by parts was used, arguing that the boundary terms vanish at $\pm\infty$. By assuming this holds for general $|\psi\rangle$, one can equate the operators on the LHS and RHS, giving $\hat P\hat X=-i\hbar+\hat X\hat P$ or $[\hat X,\hat P]=i\hbar$.
This "calculation" is severely handwavy, none of it is mathematically rigorous. For example, I'm taking derivatives of $\delta$ which is not defined. But I believe the gist of this calculation could be made rigorous in a distributional sense. Also, in my mind the commutator is actually more fundamental and this, if anything, shows that the wave function representation is compatible with that.
Your confusion stems from the fast that this is a mixture of abuse of notation and having different Hilbert spaces to work with:
Generally the abstract operator $A$ acts on some abstract vector $|\psi\rangle$, for example the momentum operator $\hat p$ acts on its eigenstates $|p\rangle$ by $\hat p |p \rangle = p |p \rangle$.
In the wavefunction formalism, we say that for a Hilbert space that has some position basis $|x\rangle$, we can switch to the space of square-integrable functions $L^2(\mathbb{R})$ without loss of information by defining $\psi(x) = \langle x | \psi \rangle $. This $\psi(x)$ now is a scalar function, and it is a vector as an element of the Hilbert space $L^2(\mathbb{R})$. Now, operators on this Hilbert space must be operators on the functions, and it just so hpapens that the explicit form of the momentum operator is $\hat p = \mathrm{i}\hbar \partial_x $. This can be seen by considering the momentum operator as the generator of translations as per $T(\delta x) = 1 - \frac{\mathrm{i}}{\hbar}\hat p \delta x $ which must act as $T(\delta x) |x\rangle = |x + \delta x\rangle$ and then appying this to some $|\psi\rangle$. Let me know if you wish me to carry out that (not overly long) calculation.
Your comments have made it clearer to me what you actually want, the following is intended to address that:
In general, the kets $|\psi\rangle$ are elements of an abstract Hilbert space $\mathcal{H}$, we know nothing more about them.
In the context we are discussing, $\mathcal{H}$ is the space of states of a particle moving in one dimension, which we will call $\mathcal{H}_{1D}$. It is constructed by saying that there is an operator $\hat x$ on $\mathcal{H}_{1D}$ and a set of states $X := \{|x\rangle | x \in \mathbb{R} \}$ which are the eigenvectors of $\hat x$, i.e $\hat x | x \rangle = x | x \rangle\forall |x\rangle \in X$, and demanding that $X$ is a basis of $\mathcal{H}_{1D}$, where "basis" means a "rigged basis" as also explained in this answer of mine. In the following, none of the integrals and manipulations involving $\lvert x \rangle$ are fully mathematically rigorous.
Now, consider the Hilbert space of square integrable functions $ f : \mathbb{R} \rightarrow \mathbb{C} $, denoted $L^2(\mathbb{R},\mathbb{C})$. To every function $ \psi :\mathbb{R} \rightarrow \mathbb{C} $ with $\psi \in L^2(\mathbb{R},\mathbb{C})$, we define the map $$ \mathrm{Ket}: L^2(\mathbb{R},\mathbb{C})\rightarrow \mathcal{H}_{1D}, \psi \mapsto|\psi\rangle := \int_{-\infty}^\infty\psi(x)|x\rangle\mathrm{d}x,$$ where the integral is supposed to be a generalization of the usual basis decomposition $\sum_n c_n \vert n\rangle$ for a countable basis and should probably be some sort of Bochner integral, yet the space the $\lvert x\rangle$ live in is unfortunately not a Banach space, so this does not work straightforwardly.
Certainly, different functions $\psi,\psi'$ cannot generate the same ket this way, since for $\psi \neq \psi'$, we must have $\psi(x_0) \neq \psi'(x_0)$ at some $x_0 \in \mathbb{R}$, so the generated kets will differ, since the coefficient of at least one basis ket differs between them. Thus, this map is injective, and no information is lost.
Conversely, for a given ket $|\psi\rangle$ define the map
$$ \mathrm{Func} : \mathcal{H}_{1D} \rightarrow L^2(\mathbb{R},\mathbb{C}), |\psi\rangle\mapsto (\psi : \mathbb{R} \rightarrow \mathbb{C}, x \mapsto \langle x | \psi \rangle)$$
Ignoring that fact that the $\psi$ here is not always really a function, but belongs to the larger space of tempered distributions containing $L^2(\mathbb{R},\mathbb{C})$, we can again see that different kets cannot produce the same function $\psi$, since $\langle x | \psi\rangle$ gives the coefficents in the basis $X$, and two vectors with identical basis coefficents are identical. Thus, this map is also injective, and so also does not forget information.
Again, if you ignore the mathematical subtleties (which you shouldn't after you have become comfortable with the basic concepts!), these maps are actually inverses of each other, and thus show that there is no difference in the information encoded in $\mathcal{H}_{1D}$ and $L^2(\mathbb{R},\mathbb{C})$.
What about operators? Let me just exemplify this for the position operator $\hat x$. The scalar product of $L^2(\mathbb{R},\mathbb{C})$ is given by
$$ (\psi,\phi) = \int_{-\infty}^\infty \phi(x)\bar\psi(x)\mathrm{d}x\forall \phi, \psi \in L^2(\mathbb{R},\mathbb{C})$$
and the defining property of the position operator is that $\hat x |x\rangle = x|x\rangle$, and so applying the map $\mathrm{Func}$ to it yields $\langle \psi | \hat x | x\rangle = x \langle \psi | x \rangle$, which immediately yields $\mathrm{Func}(\hat x | x \rangle) = x \mathrm{Func}(|x\rangle)$ and means that representing $\hat x$ on $L^2(\mathbb{R},\mathbb{C})$ is nothing more that multiplying a function $\psi$ by the identity function $id_\mathbb{R}(x) = x$.
To make it completely foolproof1, you'd need to define the operators via their eigenbasis: $$ P|\psi\rangle = \int_\mathbb{R}\!\!\mathrm{d}p\ (p\cdot|p\rangle\langle p|\psi\rangle) $$ where (proportionality $\propto$ means, you'd still need to normalise the Fourier transform) $$ |p\rangle \propto \int_\mathbb{R}\!\!\mathrm{d}x\ e^{-i(p/\hslash)x} |x\rangle. $$
But because $P$ (like most anything you'll find operating on the Hilbert spaces of quantum-mechanics) is a linear operator, it isn't really necessary to give such a "proper definition" of the function $P : \mathcal{H} \to \mathcal{H}$. Instead, you can just define the result by giving all the possible scalar products2: $$ \langle x|P|\psi\rangle = -i\hslash \frac{\partial \psi(x)}{\partial x} = -i\hslash \frac{\partial \langle x|\psi\rangle}{\partial x}. $$ This is, by means of partial integration3, equivalent to the other definition: $$\begin{align} \int_\mathbb{R}\!\!\mathrm{d}p\ (p\cdot\langle x|p\rangle\langle p|\psi\rangle) \propto& \int_\mathbb{R}\!\!\mathrm{d}p \int_\mathbb{R}\!\!\mathrm{d}x'\! \int_\mathbb{R}\!\!\mathrm{d}x'' \ p\cdot e^{-i(p/\hslash)(x'-x'')} \langle x|x''\rangle \langle x'|\psi\rangle \\=& \int_\mathbb{R}\!\!\mathrm{d}p \int_\mathbb{R}\!\!\mathrm{d}x' \ p\cdot e^{-i(p/\hslash)(x'-x)} \langle x'|\psi\rangle \\=& \int_\mathbb{R}\!\!\mathrm{d}p\ (i\hslash) \int_\mathbb{R}\!\!\mathrm{d}x' \ (\tfrac{\partial}{\partial x'} e^{-i(p/\hslash)(x'-x)}) \langle x'|\psi\rangle \\=& -i\hslash\int_\mathbb{R}\!\!\mathrm{d}p \int_\mathbb{R}\!\!\mathrm{d}x' \ e^{-i(p/\hslash)(x'-x)} (\tfrac{\partial}{\partial x'} \langle x'|\psi\rangle) \\\propto& -i\hslash \frac{\partial \langle x | \psi \rangle}{\partial x}. \end{align}$$
Simply writing $P = -i\hslash \frac{\partial}{\partial x}$ is a pretty natural shorthand for the scalar product definition.
1Actually, I've still been using sloppy notation there: $|x\rangle$ and $|p\rangle$ mean different things though both $x$ and $p$ are merely arbitrary integration-variable names... but alas, that's what physicist do all the time. Really strictly speaking, we'd need to write something like $|\mathrm{loc}_x\rangle$ and $|\mathrm{mom}_p\rangle$ for these eigenstates.
2I.e. you're basically defining the dyadic function $P: \mathcal{H}\to\mathcal{H}^\ast \to \mathbb{C}$.
3When working in an unbounded space (often also when we aren't...) we ignore boundary terms.