D'Alembert's Principle: Necessity of virtual displacements
Let us consider a non-relativistic Newtonian problem of $N$ point particles with positions
$$ {\bf r}_i(q,t), \qquad i\in\{1, \ldots, N\},\tag{1}$$
with generalized coordinates $q^1, \ldots, q^n$, and $m=3N-n$ holonomic constraints.
Let us for simplicity assume that the applied force of the system has generalized (possibly velocity-dependent) potential $U$. (This e.g. rules out friction forces proportional to the velocity.)
It is then possible to derive the following key identity
$$\begin{align}\sum_{i=1}^N \left({\bf F}_i-\dot{\bf p}_i\right)\cdot \left(\delta {\bf r}_i - \frac{\partial {\bf r}_i}{\partial t}\delta t\right) ~=~& \sum_{i=1}^N \left({\bf F}_i-\dot{\bf p}_i\right)\cdot \sum_{j=1}^n\frac{\partial {\bf r}_i}{\partial q^j}\delta q^j\cr ~=~& \sum_{j=1}^n \left(\frac{\partial L}{\partial q^j}-\frac{\mathrm d}{\mathrm dt} \frac{\partial L}{\partial \dot{q}^j} \right) \delta q^j,\end{align} \tag{2} $$
where
$$ {\bf p}_i~=~m{\bf v}_i, \qquad {\bf v}_i~=~\dot{\bf r}_i, \qquad L~=~T-U,\qquad T~=~\frac{1}{2}\sum_{i=1}^Nm_i {\bf v}_i^2. \tag{3} $$
Here $\delta$ denotes an arbitrary infinitesimal$^1$ displacement in $q$s and $t$, which is consistent with the constraints. There are infinitely many such displacements $\delta$.
The actual displacement (i.e the one which is actually being realized) is just one of those with $\delta t >0$.
In contrast, a virtual displacement $\delta$ has by definition
$$\delta t~=~0. \tag{4} $$
It is customary to refer to the time axis as horizontal, and the $q^j$ directions as vertical. Then we may say that a virtual displacement is vertical (4), while an actual displacement never is.
Note that both the lhs. and the rhs. of eq. (2) do effectively not depend on $\delta t$.
We can chose between the following first principles:
$$ \text{D'Alembert's principle } \Leftrightarrow \text{ Lagrange equations }\Leftrightarrow\text{ Stationary action principle}. \tag{5} $$
I) On one hand, d'Alembert's principle says that
$$ \sum_{i=1}^N \left({\bf F}_i-\dot{\bf p}_i\right)\cdot \delta {\bf r}_i~=~0 \tag{6} $$
for all virtual displacements $\delta$ satisfying eq. (4). This is equivalent to saying that the lhs. of eq. (2) vanishes for arbitrary (not necessarily vertical) displacements. Then Lagrange equations
$$ \frac{\partial L}{\partial q^j}-\frac{\mathrm d}{\mathrm dt} \frac{\partial L}{\partial \dot{q}^j}~=~0\tag{7} $$
follows via eq. (2) from the fact that the virtual displacements $\delta q^j$ in the generalized coordinates are un-constrained and arbitrary.
Conversely, when the Lagrange eqs. (7) are satisfied, then the lhs. of eq. (2) vanishes. This leads to d'Alembert's principle (6) for vertical displacements. It does not lead to d'Alembert's principle (6) for non-vertical displacements.
II) On the other hand, if we integrate the rhs. of eq. (2) over time $t$, we get (after discarding boundary terms) the infinitesimal virtual/vertical variation
$$ \delta S ~=~ \int \! \mathrm dt \sum_{j=1}^n \left(\frac{\partial L}{\partial q^j}-\frac{\mathrm d}{\mathrm dt} \frac{\partial L}{\partial \dot{q}^j} \right) \delta q^j\tag{8}$$
of the action $S= \int \!\mathrm dt~L$. The principle of stationary action then yields Euler-Lagrange equations (7).
III) Finally let us stress the following points:
Note in both case (I) and (II) that the freedom to perform arbitrary virtual displacements or virtual variations is what allows us to deduce the Lagrange eqs. (7).
Note in both case (I) and (II) that the displacements are vertical (4), i.e. no horizontal variation $\delta t$.
References:
- H. Goldstein, Classical Mechanics, Chapter 1 and 2.
--
$^1$ All displacements and variations in this answer are implicitly assumed to be infinitesimal.
Terms virtual displacements, as well as corresponding virtual works, are used to ensure that during these displacement all acting forces remain the same. Real displacements are, usually, supplemented by changes in forces.
The virtual displacement is collinear with the resulting force and acceleration of a particle. Now imaging, what if the REAL displacement is perpendicular to the force (it is possible if forces are changing). In this case the direction of the acceleration can not be defined. The virtual displacement is a vector value and it is not arbitrary.
Answering your points:
Virtual displacements are used because without them, the theorem would be useless in deriving useful equations of motion. With them, we can derive $N-m$ independent differential equations of motion where $N$ are the number of unconstrained degrees of freedom, $m$ the number of constraints.
A virtual displacement is a displacement that doesn't necessarily take place in the problem, but is imagined to take place, while remaining compatible with the constraints. Even in a statics problem with a marble at the bottom of a spherical well, any imagined displacement is virtual with time frozen because in reality, it's fixed at the bottom.
The value of the expression with displacements that really happen will be zero, since virtual displacements can take place in any direction compatible with the constraints of the problem