What does the 'displacement' refer to in the definition of work?

The distance referred to in the definition of work is, specifically, the distance that the object moves while the force is being applied. This is because the actual definition of work is a line integral, where the work $W$ along a path $C$ with tangent vector $d\vec{s}$ is defined as:

$$W=\int_C \vec{F}\cdot d\vec{s}$$

In general, calculating the work done requires that you specify a path along which the object moves first, which is why the notion of "force applied over a distance" makes sense; it's really more like "force applied along a specific path." For a special class of forces (conservative forces), the work done does not depend on the path taken, but only on the starting and ending points. These forces turn out to be exactly the kinds of forces for which you can define a potential energy, since the work done between any two points by that force is just the difference in the potential energy between those points.

It's true that terminology can sometimes be confusing. Let's go to the clear math:

If the body moves in straight line, you have to take the component along the force's direction.

For example, if you push a block during a 3m path in front of you, then the distance is 3m.

However, if the block moves 3m to North-west but the force is north-directed, then you must take only the component along the direction of force, i.e. the northern component. You'll have $3m\cdot \cos(45º)$

In sum, if the trajectory $s$ and the force $F$ make an angle $\alpha$, then

$$W=F\cdot s \cdot \cos(\alpha)$$

This applies only to straight lines. If the movement is curve, you must divide the curve into very small segments of small distance $ds$, and the small amount of work in each segment is:

$$dW= F \cdot ds \cdot \cos(\alpha) $$

In the definition, distance refers to the distance along which the force is applied.

In the example you have given, you must distinguish between an instaneous push and continuous pushing on the mass. If the force is applied continuously, the object will accelerate to arbitrarily close to the speed of light, and the work done approaches infinity.

Imagine pushing a box with your hand in a perfect vacuum, if your hand remains in contact with the box for a distance $d$, then that is the distance used in the definition of work.