What is Quantization ?
As I'm sure you'll see from the many answers you'll get, there are lots of notions of "quantization". Here's another perspective.
Recall the primary motivation of, say, algebraic geometry: a geometric space is determined by its algebra of functions. Well, actually, this isn't quite true --- a complex manifold, for example, tends to have very few entire functions (any bounded entire function on C is constant, and so there are no nonconstant entire functions on a torus, say), so in algebraic geometry, they use "sheaves", which are a way of talking about local functions. In real geometry, though (e.g. topology, or differential geometry), there are partitions of unity, and it is more-or-less true that a space is determined by its algebra of total functions. Some examples: two smooth manifolds are diffeomorphic if and only if the algebras of smooth real-valued functions on them are isomorphic. Two locally compact Hausdorff spaces are homeomorphic if and only if their algebras of continuous real-valued functions that vanish at infinity (i.e. for any epsilon there is a compact set so that the function is less than epsilon outside the compact set) are isomorphic.
(From a physics point of view, it should be taken as a definition of "space" that it depends only on its algebra of functions. Said functions are the possible "observables" or "measurements" --- if you can't measure the difference between two systems, you have no right to treat them as different.)
So anyway, it can be useful to recast geometric ideas into algebraic language. Algebra is somehow more "finite" or "computable" than geometry.
But not every algebra arises as the algebra of functions on a geometric space. In particular, by definition the multiplication in the algebra is "pointwise multiplication", which is necessarily commutative (the functions are valued in R or C, usually).
So from this point of view, "quantum mathematics" is when you try to take geometric facts, written algebraically, and interpret them in a noncommutative algebra. For example, a space is locally compact Hausdorff iff its algebra of continuous functions is commutative c-star algebra, and any commutative c-star algebra is the algebra of continuous functions on some space (in fact, on its spectrum). So a "quantum locally compact Hausdorff space" is a non-commutative c-star algebra. Similarly, "quantum algebraic space" is a non-commutative polynomial algebra.
Anyway, I've explained "quantum", but not "quantization". That's because so far there's just geometry ("kinetics"), and no physics ("dynamics").
Well, a noncommutative algebra has, along with addition and multiplication, an important operation called the "commutator", defined by $[a,b]=ab-ba$. Noncommutativity says precisely that this operation is nontrivial. Let's pick a distinguished function H, and consider the operation $[H,-]$. This is necessarily a differential operator on the algebra, in the sense that it is linear and satisfies the Leibniz product rule. If the algebra were commutative, then differential operators would be the same as vector fields on the corresponding geometric space, and thus are the same as differential equations on the space. In fact, that's still true for noncommutative algebras: we define the "time evolution" by saying that for any function (=algebra element) f, it changes in time with differential [H,f]. (Using this rule on coordinate functions defines the geometric differential equation; in noncommutative land, there does not exist a complete set of coordinate functions, as any set of coordinate functions would define a commutative algebra.)
Ok, so it might happen that for the functions you care about, $[a,b]$ is very small. To make this mathematically precise, let's say that (for the subalgebra of functions that do not have very large values) there is some central algebra element $\hbar$, such that $[a,b]$ is always divisible by $\hbar$. Let $A$ be the algebra, and consider the $A/\hbar A$. If $\hbar$ is supposed to be a "very small number", then taking this quotient should only throw away fine-grained information, but some sort of "classical" geometry should still survive (notice that since $[a,b]$ is divisible by $\hbar$, it goes to $0$ in the quotient, so the quotient is commutative and corresponds to a classical geometric space). We can make this precise by demanding that there is a vector-space lift $(A/\hbar A) \to A$, and that $A$ is generated by the image of this lift along with the element $\hbar$.
Anyway, so with this whole set up, the quotient $A/\hbar A$ actually has a little more structure than just being a commutative algebra. In particular, since $[a,b]$ is divisible by $\hbar$, let's consider the element $\{a,b\} = \hbar^{-1} [a,b]$. (Let's suppose that $\hbar$ is not a zero-divisor, so that this element is well-defined.) Probably, $\{a,b\}$ is not small, because we have divided a small thing by a small thing, so that it does have a nonzero image in the quotient.
This defines on the quotient the structure of a Poisson algebra. In particular, you can check that $\{H,-\}$ is a differential operator for any (distinguished) element $H$, and so still defines a "mechanics", now on a classical space.
Then quantization is the process of reversing the above quotient. In particular, lots of spaces that we care about come with canonical Poisson structures. For example, for any manifold, the algebra of functions on its cotangent bundle has a Possion bracket. "Quantizing a manifold" normally means finding a noncommutative algebra so that some quotient (like the one above) gives the original algebra of functions on the cotangent bundle. The standard way to do this is to use Hilbert spaces and bounded operators, as I think another answerer described.
I don't know what it means for a mathematician to quantize something, but I can give you a rough description, and a few specific examples, from a physicist's point of view.
Motivational fluff
When quantum mechanics was first discovered, people tended to think of it as a modified version of classical mechanics [1]. In those days, very few quantum systems were known, so people would create quantum systems by "quantizing" classical ones. To quantize a classical system is to come up with a quantum system that "behaves similarly" in some sense. For example, you generally want there to be an intuitive correspondence between the observables of a classical system and the observables of its quantization, and you generally want the expectation values of the quantized observables to obey the same equations of motion as their classical counterparts.
Because the goal of quantization is to find a quantum system that's "analogous" in some way to a given classical system, it's not a mathematically well-defined procedure, and there's no unique way of doing it. How you attempt to quantize a system, and how you decide whether or not you've succeeded, depends entirely on your motivation and goals.
The harder stuff
I've been using the phrase "quantum system" a lot---what do I really mean? In my opinion, one of the best ways to find out is to read Section 16.5 of Probability via Expectation, by Peter Whittle.
Roughly speaking, a quantum system has two basic parts:
A complex inner product space $H$, called the state space [2]. Each ray of $H$ represents a possible "pure state" of the system. A pure state is somewhat analogous to a probability distribution, in that it tells you how to assign expectation values to "observables"; in particular, it tells you how to assign probabilities to propositions.
A collection of self-adjoint linear maps from $H$ to itself, called observables. An observable is somewhat analogous to a random variable; it represents a property of the system that can be measured and found to have a certain value. The values that an observable can take are given by its eigenvalues (or, in the infinite-dimensional case, its spectrum). Say $A$ is an observable, $a$ is an eigenvalue of $A$, and $v_1, \ldots, v_n \in H$ form an orthonormal basis for the eigenspace of $a$. If the state of the system is the ray generated by the unit vector $\psi \in H$, the probability that the observable $A$ will be found to have the value $a$ is $\langle v_1, \psi \rangle + \ldots + \langle v_n, \psi \rangle$, where $\langle \cdot, \cdot \rangle$ is the inner product. You can then easily show that the expectation value of the observable $A$ is $\langle \psi A \psi \rangle$. Observables whose only eigenvalues are $1$ and $0$—that is, projection operators on $H$—play a special role, because they correspond to logical propositions about the system. The expectation value of a projection operator is just the probability of the proposition.
Most interesting quantum systems have another part, which is often very important:
- A set of unitary maps from $H$ to itself, which might be called transformations. These represent "automorphisms" of the system. In physics, many quantum systems have a one-parameter group of transformations, often denoted $U(t)$, that represent time evolution; the idea is that if the state of the system is currently (the ray generated by) $\psi$, the state will be $U(t)\psi$ after $t$ units of time have passed. Physical systems often have other transformation groups as well; for example, a quantum system that's supposed to have a "spatial orientation" will generally have a group of transformations that form a representation of $SO(3)$.
A few examples
Quantum random walks are, as the name suggests, quantized random walks. More generally, you can quantize the idea of a Markov chain. For a great introduction, see the paper "Quantum walks and their algorithmic applications", by Andris Ambainis.
In Sections 2 and 3 of the notes "A Short Introduction to Noncommutative Geometry", Peter Bongaarts describes quantized versions of compact topological spaces and classical mechanical systems.
In Section 4 of the book Noncommutative Geometry (caution---big PDF), Alain Connes introduces a quantized version of calculus. Here, the observables representing complex variables are non-self-adjoint because complex variables can take on complex values. An observable representing a complex variable must therefore be allowed to have complex eigenvalues.
I hope this helps!
[1] Today, in contrast, most physicists think of classical mechanics as an approximation to quantum mechanics.
[2] If $H$ is infinite-dimensional, it's typically a separable Hilbert space. You may even need $H$ to be something fancier, like a rigged Hilbert space.
Just to restate some facts already stated in other answers, quantization can mean a few different things. In deformation quantization, we start with a classical theory given by a Poisson manifold. Then, (by definition) the algebra of functions forms a Poisson algebra. A quantization of this algebra is a noncommutative algebra with operators $X_f$ for $f$ a function. There is also a formal parameter $\hbar$. This algebra satifies $$ X_f\ X_g = X_{fg} + \mathcal{O}(\hbar)\ . $$
The idea of quantization is that the Poisson bracket becomes a commutator, or
$$ [X_f,X_g] = \hbar X_{\lbrace f,g \rbrace} + \mathcal{O}(\hbar^2)\ . $$
Thus, we have a noncommutative version of classical mechanics. The existence of such an algebra is a theorem of Kontsevich (the case of a symplectic manifold was solved much earlier, but I forget by whom).
In mathematics, there are plenty of interesting analogous situations where you have a noncommutative thingie which is, in some sense, a formal deformation of a commutative thingie. You can see the other direction of the above as an example of the following general fact. Given a filtered algebra whose associated graded is commutative, there is a natural Poisson structure on the associated graded.
In physics, however, it's not enough to just deform the algebra of functions; we have to now represent things on a Hilbert space. This introduces a whole host of other problems. In geometric quantization, this is split into two steps. Let's say we have a symplectic manifold whose symplectic form is integral. Then we can construct a line bundle with connection whose curvature is that symplectic form. The Hilbert space is the space of $L^2$ sections of this bundle. This is much too large, however, so you have to cut it down (which is step 2). In various cases, well-defined procedures exist, but I don't believe this is well-understood in general. For example, I'm not sure it's possible to represent every function as an operator.
It's probably worth pointing out that, from the point of view of physics, quantization is backwards. It is the quantum theory that is fundamental, and the classical theory should arise as some limit of the quantum theory. There's some interesting mathematics there, and also a whole lot of philosophy too.