Closed form for the harmonic approximation sum $\sum _{k=1}^{\infty } \left(H_k^{(2)}-\zeta (2)\right){}^2$
By summation by parts $$\begin{eqnarray*} -\sum_{k\geq 1}1\cdot\left(\zeta(2)-H_k^{(2)}\right)^2 &=& \sum_{n\geq 1}\frac{n}{(n+1)^4}+\frac{2n (H_n^{(2)}-\zeta(2))}{(n+1)^2}\\&=&\zeta(3)-\zeta(4)+2\sum_{n\geq 1}\frac{H_n^{(2)}-\zeta(2)}{n+1}-2\sum_{n\geq 1}\frac{H_n^{(2)}-\zeta(2)}{(n+1)^2}\\&=&\zeta(3)-\zeta(4)+2\zeta(2)-4\zeta(3)+\frac{7\pi^4-60\pi^2}{180}\end{eqnarray*}$$ (the last identity follows from standard Euler sums) and by simplifying we get $$\boxed{ \sum_{n\geq 1}\left(\zeta(2)-H_k^{(2)}\right)^2 = \color{red}{3\,\zeta(3)-\zeta(2)^2.}}$$ Similarly, $$\begin{eqnarray*}\sum_{k\geq 1}\left(\zeta(m)-H_k^{(m)}\right)^2 &=& \sum_{k\geq 1}\left[2\,\zeta(m)-H_{k}^{(m)}-H_{k+1}^{(m)}\right]\frac{k}{(k+1)^m}\\&=&\sum_{k\geq 1}\left[2\,\zeta(m)-2H_{k}^{(m)}-\frac{1}{(k+1)^m}\right]\cdot\left[\frac{1}{(k+1)^{m-1}}-\frac{1}{(k+1)^m}\right]\end{eqnarray*}$$ boils down to computing few values of $\zeta$, recalling $\sum_{k\geq 1}\frac{H_k^{(m)}}{k^m}=\frac{\zeta(2m)+\zeta(m)^2}{2} $ (symmetry) and tackling $\sum_{k\geq 1}\frac{H_k^{(m)}}{k^{m-1}}$ or $\sum_{k\geq 1}\frac{H_k^{(m-1)}}{k^m}$. Flajolet and Salvy outline efficient ways through residues. The first cases are $$ \sum_{k\geq 1}\frac{H_k}{k^2}=2\,\zeta(3),\qquad \sum_{k\geq 1}\frac{H_k^{(2)}}{k^3} = 3\,\zeta(2)\,\zeta(3)-\frac{9}{2}\,\zeta(5) $$ $$ \sum_{k\geq 1}\frac{H_k^{(3)}}{k^4} = -10\,\zeta(2)\,\zeta(5)+18\,\zeta(7). $$
EDIT 03.01.18
alternating sum $s_{m,2}(-1)$ calculated up to the specific basic alternating sum $S^{+-}_{m,m} = \sum_{k=1}^\infty (-1)^k \frac{H_k^{(m)}}{k^m}$
brief discussion of $S^{+-}_{m,m}$. Closed forms are available for $m=1$ and $m=2$ in terms of $\zeta$-values, $\log(2)$, and Li. For $m \ge 3$ reducibility remains to be clarified.
asymptotic behaviour of $s_{m,2}(1)$ numerically determined
Original post
Inspired by the first part of the solution of Jack D'Aurizio I show here that the sum
$$s_{m}=s_{m,2}(1) =\sum _{k=1}^{\infty } \left(\zeta (m)-H_k^{(m)}\right){}^2\tag{1}$$
has a closed form not only for $m=2$ but generally for $m\ge 3$. Furthermore, this closed form is composed solely of zeta-values, i.e. of values of Riemann's $\zeta$ function at positive integers $\ge 2$.
Result
Here are the first few expressions in the format $\{m, s_m\}$
$$ \begin{array}{c} \left\{2,- \zeta (2)^2+3\zeta (3)\right\} \\ \left\{3,-\zeta (3)^2+\pi ^2 \zeta (3)-10 \zeta (5)\right\} \\ \left\{4,-\frac{1}{3} 10 \pi ^2 \zeta (5)+35 \zeta (7)-\frac{\pi ^8}{8100}\right\} \\ \left\{5,-\zeta (5)^2+\frac{\pi ^4 \zeta (5)}{9}+\frac{35 \pi ^2 \zeta (7)}{3}-126 \zeta (9)\right\} \\ \left\{6,-\frac{1}{15} 7 \pi ^4 \zeta (7)-42 \pi ^2 \zeta (9)+462 \zeta (11)-\frac{\pi ^{12}}{893025}\right\} \\ \left\{7,-\zeta (7)^2+\frac{2 \pi ^6 \zeta (7)}{135}+\frac{28 \pi ^4 \zeta (9)}{15}+154 \pi ^2 \zeta (11)-1716 \zeta (13)\right\} \\ \left\{8,-\frac{1}{105} 8 \pi ^6 \zeta (9)-\frac{22 \pi ^4 \zeta (11)}{3}-572 \pi ^2 \zeta (13)+6435 \zeta (15)-\frac{\pi ^{16}}{89302500}\right\} \\ \left\{9,-\zeta (9)^2+\frac{\pi ^8 \zeta (9)}{525}+\frac{22 \pi ^6 \zeta (11)}{63}+\frac{143 \pi ^4 \zeta (13)}{5}+2145 \pi ^2 \zeta (15)-24310 \zeta (17)\right\} \\ \left\{10,-\frac{1}{945} 11 \pi ^8 \zeta (11)-\frac{286 \pi ^6 \zeta (13)}{189}-\frac{1001 \pi ^4 \zeta (15)}{9}-\frac{24310 \pi ^2 \zeta (17)}{3}+92378 \zeta (19)-\frac{\pi ^{20}}{8752538025}\right\} \\ \end{array}\tag{2a} $$
Or, in a "canonical" form, in which all powers of $\pi$ are replaced by corresponding $\zeta$-functions
$$ \begin{array}{cc} 2 & - \zeta (2)^2+3\zeta (3)\\ 3 & -\zeta (3)^2+6 \zeta (2) \zeta (3)-10 \zeta (5) \\ 4 & -\zeta (4)^2-20 \zeta (2) \zeta (5)+35 \zeta (7) \\ 5 & -\zeta (5)^2+10 \zeta (4) \zeta (5)+70 \zeta (2) \zeta (7)-126 \zeta (9) \\ 6 & -\zeta (6)^2-42 \zeta (4) \zeta (7)-252 \zeta (2) \zeta (9)+462 \zeta (11) \\ 7 & -\zeta (7)^2+14 \zeta (6) \zeta (7)+168 \zeta (4) \zeta (9)+924 \zeta (2) \zeta (11)-1716 \zeta (13) \\ 8 & -\zeta (8)^2-72 \zeta (6) \zeta (9)-660 \zeta (4) \zeta (11)-3432 \zeta (2) \zeta (13)+6435 \zeta (15) \\ 9 & -\zeta (9)^2+18 \zeta (8) \zeta (9)+330 \zeta (6) \zeta (11)+2574 \zeta (4) \zeta (13)+12870 \zeta (2) \zeta (15)-24310 \zeta (17) \\ 10 & -\zeta (10)^2-110 \zeta (8) \zeta (11)-1430 \zeta (6) \zeta (13)-10010 \zeta (4) \zeta (15)-48620 \zeta (2) \zeta (17)+92378 \zeta (19) \\ \end{array}\tag{2b} $$
Asymptotic behaviour
Numerically we find for large $m$ a straight exponential decay:
$$N(s_m)=1.01021 e^{-1.38663\; m}$$
Discussion
Notice that for odd $m$ the structure is simpler, at least the fraction is missing.
The coefficient series are not in OEIS.
Derivation
We consider the partial sum up to $n$ and in then take the limit $n\to\infty$.
We shall use partial summation (PS) as was shown by Jack D'Aurizio in his solution to be the method of choice.
Remember that partial summation (PS) allows to transform a sum similar to the well known partial integration (PI)
$$\int_1^n A'(x) b(x) \, dx=A(x) b(x)|_1^n -\int_1^n A(x) b'(x) \, dx$$
namely
$$\sum _{k=1}^n a_k b_k=A_n b_n- \sum _{k=1}^{n-1} A_k (b_{k+1}-b_k)\tag{3}$$
where $A_k=\sum _{j=1}^k a_j$ is the "integral" of $a_k$, and $b_{k+1}-b_k$ is the "derivative" of $b_k$.
For a given summand the choice of the factors $a_k$ and $b_k$ is to a certain degreee arbitrary and one choice can be more favorable than the other. In any case the choice determines the rest of the calculations and hence should be mentioned in the beginning.
Here we take
$$a_k = b_k = \zeta (m)-H_k^{(m)}$$
Then we find
For the "derivative"
$$b_{k+1}-b_k=H_k^{(m)}-H_{k+1}^{(m)}=-\frac{1}{(1+k)^m}\tag{4a}$$
and for the "integral"
$$A_k=\sum _{j=1}^k a(j)=H_{k+1}^{(m-1)}-(k+1) H_{k+1}^{(m)}+k \zeta (m)\tag{4b}$$
Here we have used the summation formula
$$\sum _{k=1}^n H_k^{(m)}=(n+1) H_{n+1}^{(m)}-H_{n+1}^{(m-1)}\tag{5}$$
Plugging (4) into (3) the r.h.s. of (3) is given by the sum of four terms which are written down here already in the limiting form for $n \to \infty$
$$R_1=A_n b_n \to 0\tag{6a}$$ $$R_2 = \sum _{k=1}^{\infty} \frac{k \zeta (m)}{(k+1)^m} \to \zeta (m) (\zeta (m-1)-\zeta (m))\tag{6b}$$ $$R_3 = -\sum _{k=1}^{\infty} \frac{H_k^{(m)}}{k^{m-1}}= - S_{m,m-1}\tag{6c}$$ $$R_4 = \sum _{k=1}^{\infty} \frac{H_k^{(m-1)}}{k^m}= S_{m-1,m}\tag{6d}$$
Here we have used the notation of [1] for $R_3$ and $R_4$.
These authors provide a formula of Borwein for $S_{p,q}$ which is valid for the case of odd "weight" $w=p+q$.
We are lucky here, as we have always odd weight (p+q=2m-1).
Putting the parts together we have finally
$$s_{m,2}(1) = \zeta (m) (\zeta (m-1)-\zeta (m)) - S_{m,m-1} + S_{m-1,m}\tag{7}$$
Some results have been shown above. They were checked numerically to be valid.
In order to have everything in one place here is the function $S$
$$S_{p,q}=(-1)^p \sum _{k=1}^{\left\lfloor \frac{q}{2}\right\rfloor } \zeta (2 k) \binom{-2 k+p+q-1}{p-1} \zeta (-2 k+p+q)+(-1)^p \sum _{k=1}^{\left\lfloor \frac{p}{2}\right\rfloor } \zeta (2 k) \binom{-2 k+p+q-1}{q-1} \zeta (-2 k+p+q)+\left(-\frac{1}{2} (-1)^p \binom{p+q-1}{p}-\frac{1}{2} (-1)^p \binom{p+q-1}{q}+\frac{1}{2}\right) \zeta (p+q)+\frac{1}{2} \left(1-(-1)^p\right) \zeta (p) \zeta (q)\text{/;}\text{OddQ}[p+q]$$
For the convenience of some users, here is also the Mathematica code for copying (not for reading)
S[p_, q_] :=
(* = Sum[HarmonicNumber[k,p]/k^q,{k,1,\[Infinity]}], if p+q odd *) (Zeta[p + q] (1/2 - (-1)^p/2 Binomial[p + q - 1, p] - (-1)^p/2 Binomial[p + q - 1, q]) + (1 - (-1)^p)/2 Zeta[p] Zeta[q] + (-1)^p Sum[Binomial[p + q - 2 k - 1, q - 1] Zeta[2 k] Zeta[p + q - 2 k], {k, 1, Floor[p/2]}] + (-1)^p Sum[Binomial[p + q - 2 k - 1, p - 1] Zeta[2 k] Zeta[p + q - 2 k], {k, 1, Floor[q/2]}]) /; OddQ[p + q]
The alternating sum
I have calculated the alternating sum
$$s_a = s_{m,2}(-1) = \sum_{k\ge1} (-1)^k (\zeta(m) - H_k^{(m)})^2$$
using partial summation with the distribution $a_k = (-1)^k, b_k = (\zeta(m)-H_k^{(m)})^2$ with the (preliminary?) result
$$s_{m,2}(-1)=S^{+-}_{m,m}-\left(-2^{1-m} \zeta (m)^2-2^{-(2 m)} \zeta (2 m)+\frac{1}{2} \left(\zeta (m)^2+\zeta (2 m)\right)\right)\tag{8}$$
Where (using the definition of [1])
$$S^{+-}_{p,q} = \sum_{k=1}^\infty (-1)^k \frac{H_k^{(p)}}{k^q}$$
I have checked numerically that (8) is correct (to a high degree).
The sum $S^{+-}_{p,q}$ has been studied extensively in [1], but alas, closed forms are given (and seem to exist) only for odd weight $w = p + q$, and we have even weigth $2m$.
Also in [2] our case, designated $\alpha_h(m,n)$ there, is circumnavigated.
For $m=1$ the closed form result is known [3]:
$$S^{+-}_{1,1} = \frac{\log ^2(2)}{2}-\frac{\zeta(2)}{2}$$
For $m=2$ the question was asked in [4] and the answer was given by Przemo. Quoting also his notation it is
$$S^{+-}_{2,2}={\bf H}^{(2)}_2(-1) = -4 \text{Li}_4\left(\frac{1}{2}\right)-\frac{7}{2} \zeta (3) \log (2)+\frac{17 \pi ^4}{480}-\frac{\log ^4(2)}{6}+\frac{1}{6} \pi ^2 \log ^2(2)$$
It is not clear (to me) if his results really lead to closed form expressions for $m \ge 3$.
Conclusion
The question of a closed form was answered here affirmative for the non-alternating sum. For the alternating sum it could be traced back here to the reducibiity of the basic sum $S^{+-}_{m,m}$.
It remains to be seen if the confirmed reducibility for $m=1$ and $m=2$ can be extended to higher $m$.
References
[1] http://algo.inria.fr/flajolet/Publications/FlSa98.pdf, Euler Sums and contour integral representations, Philippe Flajolet and Bruno Salvy, 1995
[2] http://www.davidhbailey.com/dhbpapers/eulsum-em.pdf, Experimental Evaluation of Euler Sums, David H. Bailey, Jonathan M. Borwein and Roland Girgensohn, 1994
[3] $S^{+-}_{1,1} = \frac{1}{12} \left(6 \log ^2(2)-\pi ^2\right)$, Proving an alternating Euler sum: $\sum_{k=1}^{\infty} \frac{(-1)^{k+1} H_k}{k} = \frac{1}{2} \zeta(2) - \frac{1}{2} \log^2 2$, asked by Spivey, answered by many
[4] $S^{+-}_{2,2}$, Calculating alternating Euler sums of odd powers, asked by Zaid Alyafeai, answers were given by Przemo
I am adding a separate answer for showing a closed form for $s_{2,3}(1)$. By summmation by parts
$$ \sum_{k\geq 1}\left(\zeta(2)-H_m^{(2)}\right)^3=S_1+S_2+S_3 $$ where $$ S_1=\sum_{k\geq 1}\left[-\frac{1}{k^6}+\frac{1}{k^5}-\frac{\pi ^2}{2 k^4}+\frac{\pi ^2}{2 k^3}-\frac{\pi ^4}{12 k^2}+\frac{3 H_k^{(2)}}{k^4}-\frac{3 H_k^{(2)}}{k^3}+\frac{\pi ^2 H_k^{(2)}}{k^2}\right] $$ $$ S_2=\sum_{k\geq 1}\left[\frac{\pi ^4}{12 k}-\frac{\pi ^2 H_{k}^{(2)}}{k}+\frac{3\left(H_k^{(2)}\right)^2}{k}\right]=3\sum_{k\geq 1}\frac{\left(\zeta(2)-H_k^{(2)}\right)^2}{k}$$ $$ S_3 = -3\sum_{k\geq 1}\frac{\left(H_k^{(2)}\right)^2}{k^2}. $$ $S_1$ can be computed through standard Euler sums and $S_3$ can be computed through the previous ones and summation by parts: $$ S_1 = -2\zeta(6)-6\zeta(2)\zeta(3)+3\zeta(3)^2+\frac{29}{2}\zeta(5) $$ $$ S_3 = 2\zeta(6)-3\zeta(3)^2-\zeta(2)^3 $$ About $S_2$, summation by parts and Euler sums give $$ S_2 = 21\zeta(2)\zeta(3)-21\zeta(5)-6\sum_{k\geq 1}\frac{H_k H_k^{(2)}}{k^2} $$ and by Zaid's result outlined here we have $$\sum_{k\geq 1}\frac{H_k H_k^{(2)}}{k^2}=\zeta(2)\,\zeta(3)+\zeta(5). $$
It follows that also $s_{2,3}$ has a closed form in terms of values of the $\zeta$ function: $$\boxed{ \sum_{k\geq 1}\left(\zeta(2)-H_{k}^{(2)}\right)^3 = \color{red}{9\,\zeta(2)\,\zeta(3)-\zeta(2)^3-\frac{25}{2}\zeta(5)}.} $$