Phonons and heat conduction
Thermal conductivity is defined as a ratio of energy flux and temperature gradient (up to a factor). If phonons move freely, arbitrary energy flux can exist without temperature gradient (so the finite phonon velocity does not make the thermal conductivity final). See details on the role of scattering, say, at http://www.physics.iisc.ernet.in/~aveek_bid/PH208/Lecture%208%20phonons-thermal%20properties.pdf
Ballistic propagation can be observed but that needs special conditions. Normally, transport is diffusive. At low temperatures, scattering is dominated by defects in the lattice. Even isotopes have an effect, diamond with a reduced content of $^{13}$C has higher thermal conductivity than diamond with the natural isotope composition. This scattering determines the mean free path. While $\lambda_{free}$ may be approximately independent of temperature at low temperatures, the conductivity increases with temperature because the phonons carry more energy (proportional to $C_v$).
At high temperatures, thermal conductivity decreases because the shorter mean free path caused by phonon-phonon interaction. When the variations in atomic bond lengths get larger, anharmonic terms become important. The wave equation is not linear anymore, waves do not always pass through each other anymore, there is a probability that new waves (phonons) will be created.
As a starting point, I suggest that you take a look at the Wikipedia page on heat transfer physics, phonon scattering and Umklapp scattering.
In general the thermal conductivity $\kappa$ associated to some carrier satisfies
$$\kappa = \frac 1 3 n c_v u \lambda \tag{1}\label{1}$$
where $n$ is the carrier number density, $c_v$ is the heat capacity per carrier, $u$ is the carrier speed and $\lambda$ is the mean free path. The mean free path is related to the scattering relaxation time of the carrier $\tau$ by
$$\lambda = u \tau \tag{2}\label{2}$$
You can interpret $\tau$ as the average time between two successive "collisions" (scattering events) of the carrier. Therefore, from \ref{1} and \ref{2}, you can immediately see that in an infinite perfect harmonic crystal, in which no scattering takes place, $\kappa$ would be infinite, since there would be no scattering and therefore $\tau = \infty$ (in a finite crystal, you would still have scattering from the boundaries).
In a realistic physical systems, phonons are scattered by other phonons, electrons, defects (impurities) and boundaries. To take these effects into account, you have to leave the harmonic approximation and consider also anharmonic terms in the Hamiltonian. It was shown by Pierls [a] that it is the anharmonicity, together with the discrete nature of the crystal lattice, that generates thermal resistance.
In particular, when talking about phonon-phonon and phonon-electron scattering, a distinction is usually made between "normal" processes, which conserve the total wave vector, and "Umklapp" processes (or U-processes), where the total wave vector is changed by a reciprocal lattice vector. To know more about this distinction and its (debatable) utility, you can take a look here.
All these events contribute to the relaxation time $\tau$, and their contributions are taken into account through Matthiessen's rule, which states that the total relaxation time $\tau$ can be calculated as
$$\frac 1 \tau = \sum_i \frac 1 {\tau_i}$$
where $\tau_i$ are the relaxation times associated to the possible different scattering events. So this is how, "mathematically", scattering events influence the thermal conductivity of a material.
How does this phonon picture explain the fact that when we heat a bad conductor the heat propagates gradually from the hotter to the cooler end? If they are delocalized collective excitations, shouldn't they heat up all parts of the substance at the same time?
I think that here the answer is simply that you need some time for the "delocalized collective excitation" you are talking about to set in. This must be true even for a perfect crystal, even if its infinite thermal conductivity would seem to suggest the opposite, otherwise we would have instant propagation of a signal (the vibration of the atoms). You think about the phonons as "delocalized collective excitation", but in reality they are much more similar to wave packets arising from a superposition of these collective excitation (the normal modes of the crystal). Maybe I am not being 100% precise in my terminology here, but I hope that I managed to convey the general meaning of what I have in mind.
References
[a] R. Peierls, “Zur kinetischen Theorie der Wärmeleitung in Kristallen” ("On the Kinetic Theory of Thermal Conduction in Crystals") Ann. Phys. 395, 1055–1101 (1929)