If an earthquake can destroy buildings why it cant kill us according to physics?

There are several differences between humans and buildings:

1. Anchoring Suppose I were to push you backward. Just on reflex, one of your feet will move backward to catch yourself. A building, on the other hand, is anchored in place. When the ground moves, the bottom of the building has to move with it. But a human body doesn't have to move with the ground. There's only so much force that a shaking ground can apply to the human body.

2. Flexibility: The human body is designed to move (anthropomorphizing evolution a bit here). Other than bones, our organs are flexible, and the bones are connected by multiple joints and that can bend without harm to the body (that what a joint is: a part of the body that's supposed to move). A building doesn't have that level of flexibility. Nowadays buildings, especially ones in earthquake-prone regions, are often designed with some flexibility, but that can't match that of a human body.

3. Scaling: This is probably the largest factor. Things work differently at different scales. If you double every dimension, you multiply the cross-section by four, but you multiply the volume by eight. If a building is a hundred times as large as a human in each direction, then it's going to have a hundred times the volume per unit of the cross-section. And torque is proportional to both mass and distance, so you can get ten thousand times as much torque per area.

The maximum recorded earthquake peak ground acceleration is less than 5g, which is typically survivable, but buildings typically are not designed for such (not very short-term) acceleration.

You are right, that you need more force to break a building than to break a human body.

But an earthquake acts by acceleration (not by force). It suddenly accelerates a large part of the ground (and hence all buildings and humans there) by the same value.

The human body is quite soft and flexible, and can therefore withstand accelerations of $10\ g$ for a few seconds (see G-force - Human tolerance).
Also quoted from there:

The human body is flexible and deformable, particularly the softer tissues. A hard slap on the face may briefly impose hundreds of $g$ locally but not produce any real damage;

But hard buildings can only withstand much smaller accelerations. And they are even more sensitive for horizontal accelerations than for vertical accelerations.
According to Earthquake effects on buildings (chapter 4):

Poorly constructed buildings begin to suffer damage at about 10 percent $g$ (or $0.1\ g$).