Does the displacement in the definition of work correspond to the displacement of the object or the point of application of the force?

Here's a way to argue it isn't the displacement of the (center of mass) of the body.

Take a spring that's at rest. Apply two forces on either end so it compresses. You can do this in such a way that the center of mass of the spring doesn't move. However, the energy of the spring system has changed (the potential energy increased). In order to satisfy the work energy theorem $$W_\text{net, external} = \Delta E_\text{total},$$ it would seem that the work must be non-zero. If you used the center of mass displacement, then $W=0$, which is inconsistent. The solution is to use the displacement of the point of application of the (two) force(s): $$W=W_1^\text{app} + W_2^\text{app}.$$

First of all, I appreciate your doubt. It is conceptual and involves critical thinking regarding the topic.

In order to understand work, you need to understand the 2 physical quantities namely - Force and Displacement .

Note that using only one single word Displacement in order to define work is not appropriate. (Same is for Force ) . As, Displacement depends on the direction of Force . Now, again, it will not be appropriate to just write Force in order to define Work/Displacement. I will better write it as : As, Displacement depends on the direction of Force applied on the point of application.

Whenever a body is moving, there are many forces acting upon it. While you are walking, the body is applying force on you, the ground is applying force on you, the air is resisting you from moving etc. So, basically it is much more complicated than we think.

Coming to the main point, displacement! Let us just take a point object in order to understand the situation well. Now, before I go on, I will first define what is point-object.

The concept of a point object can be understood here : Point Particle

So, as you're clear with the basic concepts, I can go on. Each object is made up of millions of particles...Uncountable! Now, when you push an object, suppose a car, then the car is moving but the force has acted upon on a specific point. That point may be referred to point of application of the force. When the car moves (if the other opposing forces become lesser than the force applied by you) , then it surely displaces itself. Thus, it does some Work. While, you have also done the work to push the object, but we usually calculate the work done by the object experiencing the force.

The displacement here, will be based on the point particle though many simply refer it to the complete object. It is not wrong either, but just a little mis-conceptional to refer the displacement to a whole object. You must keep this in mind that basically, the force(external) is acting upon the point particle which later on affects the nature of the whole object, that's why the object displaces itself.

This can be explained in 1-2 para also, but, what I'm trying to explain is critically-thinking a topic. Whenever you try to follow a question or topic, first think about it. Try to compare the conditions with nature, notice the similarities and differences. This will make your concepts much stronger!

It is the displacement of the point on which the force is applied. This is it. That is the real definition, period. The books which do it otherwise are wrong. I would like you to read Resnick Halliday Krane , physics Vol. 1, it has a separate chapter devoted to this problem with most of the books available.

As an example to show that this is what the definition should be, you write an infinitesimal work done as, you could not use things like $m\frac{d^2x}{dt^2}dx$, the $d^2x$ and $dx$ would then be for different points and the integration wouldn't proceed the way it does.

Other simplifications arise from constraints, newton's 3rd law and viewing from different reference frame when applied to complex bodies.

I would still advise you to at least once go through that marvellous book. It is the best book on introductory physics that exists.