What really goes on in a vacuum?

I don't think the particle-anti-particle picture is a very good one to grasp what's going on. Essentially, it's a consequence of zero-point energy. In classical physics, the lowest energy state of a system, its ground state, is zero. In quantum mechanics, it's a non-zero (but very small) value. The easiest way to see how this zero point energy arises is through an elementary problem is quantum mechanics, the quantum harmonic oscillator. The classical harmonic oscillator is a system in which there is a restorative force proportional to the displacement. For example, a spring — the further you pull the end of a spring, the more force the spring resists your pull. Modeling this system in classical physics is very easy. Things are a bit different in quantum mechanics — the state of a particle is specified by its wavefunction, which encodes the probabilities of finding the particle in certain positions. Another property of quantum systems is that their energies come in discrete energy levels. If you're interested in how it is worked out, you can see here. You can derive the following result for the energy levels of the particle $$E=\hbar \omega\left( n+\frac {1}{2}\right).$$ Since $n$ specifies the energy level, setting $n$ to zero will give us the ground state. However, we can see this isn't zero — so the lowest possible state of a quantum system still contains some energy.

In a practical example, liquid helium does not freeze under atmospheric pressure at any temperature because of its zero-point energy. One very important thing to note is the following: zero-point energy does not violate the conservation of energy. A common explanation is that the uncertainty principle allows particles to violate it 'if they're quick enough!'. This simply isn't true. From the Wiki page on conservation of energy:

In quantum mechanics, energy of a quantum system is described by a self-adjoint (Hermite) operator called Hamiltonian, which acts on the Hilbert space (or a space of wave functions ) of the system. If the Hamiltonian is a time independent operator, emergence probability of the measurement result does not change in time over the evolution of the system. Thus the expectation value of energy is also time independent. The local energy conservation in quantum field theory is ensured by the quantum Noether's theorem for energy-momentum tensor operator. Note that due to the lack of the (universal) time operator in quantum theory, the uncertainty relations for time and energy are not fundamental in contrast to the position momentum uncertainty principle, and merely holds in specific cases (See Uncertainty principle). Energy at each fixed time can be precisely measured in principle without any problem caused by the time energy uncertainty relations. Thus the conservation of energy in time is a well defined concept even in quantum mechanics.

Now, on to your question — in quantum field theory, all particle are modeled as excitations of fields. That is, every particle has an associated field. For the particles that carry forces, these are the familiar force fields — such as the electromagnetic field. Fields take a value everywhere in space. Now, in classical mechanics, this value would be zero in most places. However, as we saw above, the ground state of a quantum field is non-zero. So, even in empty space (or 'free space') these fields have a a very small value. So, empty space has vacuum energy.

Nothing goes on; the vacuum is completely inert.

In quantum field theory, the vacuum is the state containing exactly zero particles anywhere in space and at all times. Since it is an eigenstate of the number operator, there is no uncertainty at all about this.

Virtual particles don't exist in time, except in a (literally) figurative sense. They don't have associated states, hence no expectations, probabilities, uncertainties. See https://www.physicsforums.com/insights/misconceptions-virtual-particles/

Your question has been addressed in two physics.stackexchange articles: are-elementary-particles-actually-more-elementary-than-quasiparticles and what-is-the-relationship-between-string-net-theory-and-string-m-theory

In short, vacuum is not inert but a dynamical medium. Casimir effect has experiemntally demonstarted that vacuum is indeed a dynamical medium. As a dynamical medium, vacuum can have motions which is the wave in the vacuum. Those waves are colloective excitations which correspond the elementary particles in the vacuum. The order in such vacumm-medium determine the nature of the elementary particles. For example, if vacuum-medium is a quantum string liquid (with topological order), its wave will satisfy Maxwell equation which correspond to photons. The ends of strings will correspond to electrons/quarks.