Are vacuum fluctuations really happening all the time?

I think it’s possible to give a beginners guide to what is meant by vacuum fluctuations, but it necessarily involves taking a few liberties so bear that in mind in what follows.

Before we start let’s remind ourselves of the following key point about superpositions. Suppose we have an operator $\hat{n}$ with eigenfunctions $\psi_i$ and we place it in a superposition:

$$ \Psi = a_0\psi_0 + a_1\psi_1 + \, … $$

Then when we do a measurement of the system using our operator $\hat{n}$ the suprposition will collapse and we will find it on one of the eigenstates $\psi_i$. The probability of finding it in that state is $a_i^2$.

Now suppose we do a measurement, then put the system back into the same superposition and do a second measurement, and keep repeating this. Our measurements will return different results depending on which of the eigenstates the superposition collapses into, so it looks as if our system is fluctuating i.e. changing with time. But of course it isn’t - this is just how quantum measurement works, and we’ll see that something similar to this is responsible for the apparent vacuum fluctuations.

Now let’s turn to quantum field theory, and as usual we’ll start with a non-interacting scalar field as that’s the simplest case. When we quantise the field we find it has an infinite number of states. These states are called Fock states and these Fock states are vectors in a Fock space, just as the states for regular QM are vectors in a Hilbert space. Each Fock state has a well defined number of particles, and there is a number operator $\hat{n}$ that returns the number of particles for a state. There is a vacuum state $\vert 0 \rangle$ that has no particles i.e. $\hat{n}\vert 0\rangle = 0$.

Suppose we consider a state of the scalar field that is a superposition of Fock states with different numbers of particles:

$$ \vert X\rangle = a_0\vert 0\rangle + a_1\vert 1\rangle +\, … $$

If we apply the number operator it will randomly collapse the superposition to one of the Fock states and return the number of particles in that state. But because this is a random process, if we repeat the experiment we will get a different number of particles each time and it looks as if the number of particles in the state is fluctuating. But there is nothing fluctuating about our state $\vert X\rangle$ and the apparent fluctuations are just a consequence of the random collapse of a superposition.

And by now you’ve probably guessed where I’m going with this, though we need to be clear about a few points. The free field is a convenient mathematical object that doesn’t exist in reality - all real fields are interacting. The states of interacting fields are not Fock states and don’t live in a Fock space. In fact we know very little about these states. However we can attempt to represent the vacuum of an interacting field $\vert \Omega\rangle$ as a sum of free field Fock states, and if we do this then applying the number operator to $\vert \Omega\rangle$ will return an effectively random value, just as it would do for a superposition of free field states.

And this is what we mean by vacuum fluctuations for an interacting field. There is nothing fluctuating about the vacuum state, however measurements we make of it will return random values giving the appearance of a time dependent fluctuation. It is the measurement that is fluctuating not the state.

I’ve used the example of the number operator here, but it’s hard to see how the number operator corresponds to any physical measurement so take this just as a conceptual example. However the process I’ve described affects real physical measurements and happens whenever the vacuum is not an eigenstate of the observable measured. For an example of this have a look at Observation of Zero-Point Fluctuations in a Resistively Shunted Josephson Tunnel Junction, Roger H. Koch, D. J. Van Harlingen, and John Clarke, Phys. Rev. Lett. 47, 1216 available as a PDF here.

Particles do not constantly appear out of nothing and disappear shortly after that. This is simply a picture that emerged from taking Feynman diagrams literally. Calculating the energy of the ground state of the field, i.e. the vacuum, involves calculating its so-called vacuum expectation value. In perturbation theory, you achieve this by adding up Feynman diagrams. The Feynman diagrams involved in this process contain internal lines, which are often referred to as "virtual particles". This however does not mean that one should view this as an actual picture of reality. See my answer to this question for a discussion of the nature of virtual particles in general.

It's true that the vacuum ought to be an eigenstate of the full interacting Hamiltonian. But as seen from the perspective of the Hamiltonian of the free theory (all interactions being treated as perturbations around this free theory) the actual ground state is "dressed" by many vacuum fluctuations on top of the free ground state.