Quantum mechanics formalism and C*-algebras
In addition to what has already been said I would like to add some more comments. I completely understand your suspicion that the passage from unbounded operators to bounded ones is at least tricky. For the canonical commutation relations of position and momentum operators this can be solved in a reasonable and also physically acceptable way by passing to the Weyl algebra (OK; there are zillions of Weyl algebras around in math, but I'm refering here to the $C^*$-algebra generated by the exponentials of $Q$'s and $P$'s subject to the heuristic commutation relations arising from $[Q, P] = \mathrm{i} \hbar$).
However, there are other situations in physics where this is much more complicated: when one tries to quantize a classical mechanical system which has a more complicated phase space than just $\mathbb{R}^{2n}$ then one can not hope to get some easy commutation relations which allow for a Weyl algebra like construction. To be more specific, for general symplectic of Poisson manifolds the beast quantization scheme one can get in this generality is probably formal deformation quantization. Here the classical observable algebra (a Poisson algebra) is deformed into a noncommutative algebra in a $\hbar$-dependent way such that the new product, the so-called star product, depends on $\hbar$ in such a way that for $\hbar = 0$ one recovers the classical mutliplication and in first order of $\hbar$ one gets the Poisson bracket in the commutator.
Now two difficulties arise: the most severe one is that in this generality one only can hope for formal power series in $\hbar$. Thus one has even changed the underlying ring of scalars from $\mathbb{C}$ to $\mathbb{C}[[\hbar]]$. So there is of course no notion of a $C^\ast$-algebra whatsoever on this ring. Nevertheless, there is a good notion of states in the sense of positive functionals already at this stage. Second, even if one succeeds to find a convergent subalgebra one does usually not end up with a $C^\ast$-algebra on the nose. Worse: in most of the explicit exampes one knows (and there are not really many of them...) the commutation relations one obtains are very complicated. In particular, it is not clear at all how one can affiliate a $C^\ast$-algebra to them. Moreover, it is not clear which of the previous states survive this condition of convergence and yield reasonable representations by the GNS construction.
It takes quite some effort to first represent the observables by typically very unbounded operators in a reasonable way and then show that they give rise to some self-adjoint operators still obying the relevant commutation relations. This is far from being obvious. To get a flavour for the difficulties it is quite illustrative to take a look at the book of Klimyk and Schmüdgen on Quantum Groups adn their Representations. Note that in these examples one still has a lot of structure around which helps to understand the analysis.
However, physics usually requires still much more general situations. Most important here are systems with gauge degrees of freedoms where one has to pass to a reduced phase space. Even if one starts with a geometrically nice phase space the reduced one can be horribly complicated. This problem is present in any contemporary QFT really relevant to physics :(
For other quantization schemes things are similar, even though I'm not quite the expert to say something more substantial :)
So one may ask the question why one should actually insist on $C^\ast$-algebras and this strong analytic background. There is indeed a physical reason and this is that quantum physics predicts not only expectation values of observables in given states (here the notion of a ${}^\ast$-algebra and a positive functional is sufficient) but also the possible outcomes of a measurement: they are given by particular numbers called the physical spectrum of the observable. To get a good description with predictive power the (as far as I know) only way to achieve this is to say that the physical spectrum is givebn by the mathematical spectrum of a self-adjoint operator in a Hilbert space. If one is here at this point, then the passage from a (unbounded) self-adjoint operator to a $C^\ast$-algebra is comparably easy: one has the spectral projections and takes e.g. the von Neumann algebra generated by them...
So the point I would like to make is that it is very desirable from a physical point of view to have the strong analytic framework for observables as either self-adjoint operators or Hermitian elements in a $C^\ast$-algebra. But the quantum theory of many nontrivial systems requires a long long and nontrivial way before one ends up in this nice heavenly situation.
The book mentioned by Bora,
- F. Strocchi: "An Introduction to the Mathematical Structur of Quantum Mechanics"
is indeed a good reference for an axiomatic approach to quantum mechanics where the observables of a given system are assumed to form a $C^*$-algebra. The physical motivation for this approach is that every detector is representable by an observable (a self-adjoint operator) that is bounded, because every detector has an upper bound of values it can measure. A detector is therefore a bounded function of an unbounded, essentially self-adjoint observable as the impuls operator in, say, the usual (space or impuls) representation of the Heisenberg comutation relations.
This is the physical motivation of the basic assumption of an axiomatic approach to quantum physics via $C^*$- algebras: The assumption is that a quantum system, whose observables are essentially self-adjoint - maybe unbounded - operators, can equivalently be described by an operator algebra containing all bounded functions of the original observables. (The function $e^{i s A}$ for a real number s and an essentially self-adjoint operator A is of course an example, as mentioned in the comment by Pieter).
For an explanation of how and why this works in quantum mechanics, see the book by Strocci. (The short answer is that the $C^*$ - algebra of a massive particle described by position and momentum as observables is the Weyl algebra which is generated by the bounded operators occuring in the Weyl communtation relations mentioned by Pieter.)
In axiomatic quantum field theory, the difference of bounded versus unbounded observables manifests itself in two different sets of axioms:
the Wightman axioms use unbounded operators,
the Haag-Kastler axioms use bounded operators.
In this setting it is not completely clear that both sets of axioms are equivalent, although the relations Wightman => Haag-Kastler and vice versa have been proven with certain additional "technical" assumptions, see for example:
- H.J. Borchers, Jakob Yngvason: “From quantum fields to local von Neumann algebras”, Rev.Math.Phys. Special issue, 1992, p.15-47.
("Quantum fields" refer to the Wightman axioms and "local von Neumann algebras" to the Haag-Kastler axioms.)
When you compare the situation in quantum field theory to the situation in quantum mechancis, you'll see that the situation is more complicated because in quantum field theory the von Neumann uniqueness theorem cannot be applied.
I wrote a paper about this at an REU I attended two years ago. At the time, I had the goal of making the transition from classical mechanics to quantum mechanics as natural as possible, motivating the axioms of quantum mechanics from those of classical mechanics, which are a lot more intuitive. Unfortunately, given the time constraints of the REU (just a couple of months) and being relatively new to the subject, I found myself mostly following other sources (in particular, Strocchi), which made use of the $C^*$-algebraic formalism. I have to admit, at the time, I did feel a bit uncomfortable throwing away all the unbounded operators, because, having studied "physicist's" quantum mechanics before, these are some things I naturally wanted to include.
Come two years later at the same REU, I tried to tackle the same problem, but this time I wanted to develop the axioms of quantum mechanics so to as allow unbounded operators in the theory. This led me to develop the notion of what I call an $F^*$-algebra ($F$ for Fréchet). I am actually still working on the paper at the moment, but, after reading your post, I decided to upload a preliminary copy to my academia.edu account. You should read it, check it out, and see what you think. I'd be very happy to hear any comments you have.
Be warned though, I am still in the process of editing and revising it, so dare I say, there may be some errors. Read it with a skeptical eye and let me know if you catch anything.
Cheers! Jonny Gleason
Here's the link: https://drive.google.com/open?id=0B6xfgYpCM4U3UXRNVkhPTGc5RDA
EDIT: I actually did find a couple of errors in the paper at some point, but I had since lost the .tex file and never got around to just rewriting the entire thing with corrections in place. I don't remember exactly where they are at this point (it's been a couple of years), but if I recall they're not subtle, so if you just make sure to check the proofs you should be okay.