Dirac delta function and correlation functions
Saying that $\delta(0) = 0$ is completely non-sensical since the Dirac delta function is not a function to begin with. When we physicists write $$ \int \delta(x)f(x) \mathrm{d}x = f(0) \tag{1}$$ when that's all the "definition" of the delta "function" you actually need. Formally, the $\delta$ function is a tempered distribution, something that assigns numbers to test functions. The "integral notation" eq. (1) is just a mnemonic because in many respects this assignment "behaves like an integral", e.g. it obeys a variant of integration by parts. The formal definition of the delta "function" is just $f\mapsto \delta[f] = f(0)$ where you are already notationally prohibited from trying to feed a position like $x=0$ as $\delta(0)$ to it.
While it is true that one can represent the $\delta$ function as the limit of certain other functions (these are called nascent delta functions), this limit is not taken in the space of functions, but in the space of distributions, so the result is not a function. I do not have access to the specific article you are reading, but in general, manipulating the value of the delta function at specific points does not make any rigorous sense. As so often in physics, this does not necessarily mean the result obtained is wrong, but it should be proven by other means in order to trust it.
In addition to the correct mathematical interpretation appearing in the other answer by ACuriousMind, perhaps a good physically minded viewpoint is to observe that objects like $\delta(t)$ have always to be interpreted in the sense of the average value, using some smearing or averaging function (in QFT we use the same interpretation regarding field operators).
If you would like to know how $\delta(t)$ is made in a neighborhood of a point $t_0$, you should use a function $f$ concentrated in that neighborhood of $t_0$. The only information you may obtain is the integral $\int \delta(t)f(t) dt$. This way you see that the integral is non zero only if the neighborhood where $f$ does not vanish includes $t=0$, in this sense $\delta(t)$ is concentrated around $t=0$. A more precise shape of $\delta(t)$ is unaccessible.
As a matter of fact this is the way several practical instruments work, computing some sort of average values around a precise (though directly unaccessible) value $t_0$.
The white noise correlation has to be interpreted within the same framework. To check the correlation you should use an averaging function $f(t,t')$ describing the sensibility of your instrument. What you can see/measure is $$\int \langle \zeta(t) \zeta(t^\prime )\rangle f(t,t') dtdt'$$ If the correlation is the white noise one, the integral vanishes as soon as $f(t,t) = 0$.
Adopting this approach, every question regarding the locally precise shape of $\delta$ is meaningless.