What is a deformation of a category?

Here is a definition that I heard: a family of additive categories over a scheme $X$ (say, defined over a field $k$) is an $k$-linear additive category ${\mathcal C}$ equipped with a structure of a module category over the monoidal category of (quasi)coherent sheaves on $X$. Apparently, this setting is rather general and flexible, e.g. allows one to make sense of the notion of flatness of such a family, etc.

I also want to mention that there is now a new paper of van den Bergh on this, arXiv:1002.0259.

Certainly not my field, but you might want to check the paper by Lowen and Van den Bergh Deformation theory of abelian categories. I think that's where the first notion of deformation of a category appeared.

Maybe it is helpful to look at these lectures by Yekutieli: arXiv:0801.3233. There he discusses twisted deformations of algebraic varieties $X$ suggested by Kontsevich, i.e. deformations of the category of coherent sheaves on $X$. These deformations are classified by the second Hochschild cohomology of $X$, which by the Hochschild-Kostant-Rosenberg theorem is $H^0(\wedge^2T)\oplus H^1(T)\oplus H^2({\mathcal O})$, where ${\mathcal O}$ is the structure sheaf and $T$ is the tangent sheaf of $X$.

Here, roughly speaking, the first summand corresponds to noncommutative deformations of the structure sheaf (global Poisson bivectors), the second summand corresponds to commutative deformations of this sheaf (i.e., formal deformations of X as a variety), and the third term corresponds to deformations of the category of coherent sheaves which do not arise from deformations of the structure sheaf as a sheaf of algebras (i.e., the algebra deformations exist only on local charts but do not glue into a sheaf; only their categories of modules glue into a sheaf of categories, called a gerbe).

Another helpful reference on this may be van den Bergh's paper arXiv:math/0603200.