Closed form for $\sum_{k=1}^\infty(\zeta(4k+1)-1)$
I don't know if this will be useful, but thought that is might help. So, here we go ...
We can write the sum of interest as
$$\begin{align} \sum_{k=1}^{\infty}\left(\zeta(4k+1)-1\right)&=\sum_{k=1}^{\infty}\sum_{n=2}^{\infty}\frac{1}{n^{4k+1}} \tag 1\\\\ &=\sum_{n=2}^{\infty}\left(\frac{n}{2(n^2+1)}+\frac{1}{4(n-1)}+\frac{1}{4(n+1)}-\frac{1}{n}\right) \tag 2\\\\ &=\sum_{n=2}^{\infty}\left(\frac{n}{2(n^2+1)}-\frac{1}{2n}\right) \tag 3\\\\ &+\sum_{n=2}^{\infty}\left(\frac{1}{4(n-1)}-\frac{1}{4n}\right) \\\\ &+\sum_{n=2}^{\infty}\left(\frac{1}{4(n+1)}-\frac{1}{4n}\right) \\\\ &=\frac18-\frac12\sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}\tag 4 \end{align}$$
In arriving at $(1)$, we used the series definition of the Riemann-Zeta function.
In going from $(1)$ to $(2)$, we changed the order of the series, summed the interior geometric series, and expanded in partial fractions.
In going from $(2)$ to $(3)$, we merely split the sum of three convergent series.
In arriving at $(4)$, we noted that the latter two series in $(3)$ are telescoping series whose sum is $\frac18$.
Now, this last series can be written in terms of the Digamma Function as
$$\sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}=-1+\gamma+\frac12\psi^0(2-i)+\frac12\psi^0(2+i)\approx 0.17186598552401 \tag 5$$
Therefore, we have
$$\sum_{k=1}^{\infty}\left(\zeta(4k+1)-1\right)=\frac58 -\frac12 \gamma-\frac14\psi^0(2-i)-\frac14\psi^0(2+i)\approx 0.039067007237995$$
NOTE:
Here we show that the series in $(5)$ is indeed given by
$$\sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}=-1+\gamma+\frac12\psi^0(2-i)+\frac12\psi^0(2+i)$$
Expanding in partial fractions yields
$$\sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}=\frac12\sum_{n=2}^{\infty}\left(\frac1n-\frac{1}{n+i}\right)+\frac12\sum_{n=2}^{\infty}\left(\frac1n-\frac{1}{n-i}\right) \tag 6$$
Next, using the series definition of the digamma function, we find
$$\begin{align} \psi^0(z+1)&=-\gamma+\sum_{n=1}^{\infty}\frac{z}{n(n+z)}\\\\ &=-\gamma+\sum_{n=1}^{\infty}\left(\frac1n-\frac{1}{n+z}\right)\\\\ &-\gamma+\sum_{n=1}^{\infty}\left(\frac1n-\frac{1}{n+1}+\frac{1}{n+1}-\frac{1}{n+1+(z-1)}\right)\\\\ &=-\gamma+1+\sum_{n=2}^{\infty}\left(\frac{1}{n}-\frac{1}{n+(z-1)}\right) \tag 7\\\\ \end{align}$$
Using $(7)$ in $(6)$, we can write
$$\begin{align} \sum_{n=2}^{\infty}\frac{1}{n(n^2+1)}&=\frac12 \left(-2\gamma+2+\psi^0(2+i)+\psi^0(2-i)\right)\\\\ &=-1+\gamma +\frac12\psi^0(2+i)+\frac12\psi^0(2-i) \end{align}$$
as was to be shown!
Just expanding nospoon's comment, it is well-known that $$ g(x)=\sum_{n\geq 2}\left(\zeta(n)-1\right) x^{n-1} = \frac{x}{x-1}-\psi(1-x)-\gamma\tag{1}$$ so, in order to compute series like $\sum_{n\geq 2}\left(\zeta(an+b)-1\right)$, it is enough to apply a discrete Fourier transform to the middle series and to the RHS of $(1)$. So $\sum_{n\geq 2}\left(\zeta(an+b)-1\right)$ just depends on the RHS of $(1)$ evaluated over the $a$-th roots of unity. The value of $b$ chooses if there is some simplification in the sum of digamma values or not.
By expressing the infinite sum as a double sum and then switching the order of summation, we get
$$\begin{align} \sum_{n=1}^{\infty} \left[\zeta(4n+1) - 1 \right] &= \sum_{n=1}^{\infty} \sum_{m=2}^{\infty} \frac{1}{m^{4n+1}} = \sum_{m=2}^{\infty} \frac{1}{m}\sum_{n=1}^{\infty} \left( \frac{1}{m^{4}} \right)^{n} \\ &= \sum_{m=2}^{\infty} \frac{1}{m} \frac{\frac{1}{m^{4}}}{1-\frac{1}{m^{4}}} = \sum_{m=2}^{\infty} \frac{1}{m} \frac{1}{m^{4}-1}. \end{align}$$
One can use the series representation $$\psi(2+z) = - \gamma +1 + \sum_{n=2}^{\infty} \frac{z}{n(n+z)}, \tag{1}$$ where $\psi(x)$ is the digamma function, to show
$$\psi(2+z) + \psi(2-z) + \psi(2+iz) + \psi(2-iz) = - 4 \gamma +4 -4z^{4} \sum_{n=2}^{\infty} \frac{1}{n(n^{4}-z^{4})}.$$
Letting $z=1$, we find
$$ \begin{align} \sum_{m=2}^{\infty} \frac{1}{m(m^{4}-1)} &= 1-\gamma - \frac{1}{4} \Big(\psi(3) + \psi(1) + \psi(2+i) + \psi(2-i) \Big) \\ &= 1-\gamma - \frac{1}{4} \left(\frac{3}{2} - \gamma - \gamma + \psi(2+i)+\psi(2-i) \right) \\ &= \frac{5}{8}-\frac{\gamma}{2} - \frac{1}{4} \Big(\psi(2-i)+\psi(2+i) \Big) \\ &\approx 0.0390670072. \end{align}$$
EDIT:
$(1)$ I find it easier to prove that series representation for $\psi(2+z)$ using the difference equation $$\psi(N+1+z) - \psi(z) = \sum_{n=0}^{N} \frac{1}{n+z} $$ as opposed to manipulating the series representation for $\psi(1+z)$.
$$ \begin{align} \sum_{n=2}^{N} \frac{z}{n(n+z)} &= \sum_{n=2}^{N} \frac{1}{n} - \sum_{n=2}^{N} \frac{1}{n+z} \\ &= \Big( H_{N}-1 \Big) - \Big(\psi(N+1+z) - \psi(z) - \frac{1}{z} - \frac{1}{1+z} \Big)\\ &= \Big( \psi(N+1) + \gamma -1 \Big) - \Big(\psi(N+1+z) - \psi(2+z) \Big) \end{align}$$
Letting $N \to \infty$ leads to the result since $\psi(x+a) = \ln(x) + \mathcal{O} \left(\frac{1}{x} \right)$ as $x \to \infty$.