What notions are used but not clearly defined in modern mathematics?

Surprised nobody mentioned fractal yet. (Chaos has been mentioned but the connection is tenuous.)

No satisfactory definition of fractal exists. Mandelbrot tentatively defined a fractal as a set whose Hausdorff dimension is strictly larger than its topological dimension. But this leaves out many sets that most people agree are fractals, and it's hard to extend to other objects (like measures) that one also wants to consider as fractals.

Taylor defined a fractal as a set with coinciding Hausdorff and packing dimensions. His goal was to leave out too irregular objects (for which different concepts of fractal dimension may differ), but according to his definition any smooth object is a fractal, and clearly fractal sets such as Bedford-McMullen carpets are left out.

In applied fields, a fractal is often defined as a set having some kind of similarity: small parts are similar to the whole set, perhaps in a statistical or approximate sense. While many fractals arising in practice do enjoy this feature, this is still a very vague definition.

Some authors consider any set or measure in Euclidean space to be a fractal, when the goal is to study properties typically associated with fractal sets, such as Hausdorff dimension.

At the end of the day, there is agreement that giving a universal definition of fractal is impossible, yet it is a useful concept to have around, and people know a fractal when they see it.

The field with one element, $F_1$.

Georges Elencwajg in http://mathoverflow.tqft.net/discussion/968/notions-used-but-not-rigorously-defined/#Item_0

One of the most important contemporary mathematical concepts without a rigorous definition is quantum field theory (and related concepts, such as Feynman path integrals).

Note: As noted in the comments below, there is a branch of pure mathematics --- constructive field theory --- devoted to making rigorous sense of this problem via analytic methods. I should add that there is also a lot of research devoted to understanding various aspects of field theory via (higher) categorical points of view. But (as far as I understand), there remain important and interesting computations that physicists can make using quantum field theoretic methods which can't yet be put on a rigorous mathematical basis.