Interesting phenomenon with the $\zeta(3)$ series

I was hoping someone else more knowledgeable would give a more detailed answer, but: the answer is the Euler-Maclaurin formula (applied to the difference between the two sums). The points at which the digits don't match correspond to the part of the formula involving a sum over Bernoulli numbers. The values of $N$ which most have this property are powers of $10$.

The behaviour you discovered is very similar to the one exhibited by the Madhava–Leibniz sum for $\frac{\pi}{4}$:

$$ \frac{\pi}4 = \sum_{k=0}^\infty\frac{(-1)^k}{2k+1} $$

The decimal expansion of $\pi$ obtained through this formula will contain digits that disagree with $\pi$ at digits predictable by a corresponding sum over Euler numbers. In particular, calculating the first five million digits of the decimal expansion will yield

$$ 3.141592\underline{4}5358979323846\underline{4}643383279502\underline{7}841971693993\underline{873}058... $$

where the underlined digits disagree with the decimal expansion of $\pi$.

The disagreeing digits can be found using

$$ \frac{\pi}{2} - 2 \sum_{k=1}^\frac{N}{2} \frac{(-1)^{k-1}}{2k-1} \sim \sum_{m=0}^\infty \frac{E_{2m}}{N^{2m+1}} $$

In your case, these digits can can be found in a similar way using a sum over Bernoulli numbers.