Prove $\int_0^{\pi/2}x^2\sqrt{\cot x} \ dx=\frac{\sqrt2}{8}\left( \frac{5\pi^{3}}{12}-\pi^2\ln2-\pi \ln^22 \right)$
A slightly easier form for $I(a, \frac{1}{2})$ is
$$I(a, \frac{1}{2}) = 2^{a-\frac{3}{2}} \beta\left(\frac{1+2a}{4}\right) (1 + \sin \pi a + \cos \pi a), $$
where $\beta(p) = \Gamma(p)^2/\Gamma(2p)$ is the central beta function. (This can be easily obtained by applying the Euler reflection formula and the Legendre duplication formula to OP's representation.) Now from the relation
$$ \frac{d}{dp} \log \beta(p) \bigg|_{p=\frac{1}{2}} = 2\psi(1/2) - 2\psi(1) = 2 \sum_{n=0}^{\infty} \left( \frac{1}{n+1} - \frac{1}{n+\frac{1}{2}} \right) = -4 \log 2 $$
and
$$ \frac{d^2}{dp^2} \log \beta(p) \bigg|_{p=\frac{1}{2}} = 2\psi^{(1)}(1/2) - 4\psi^{(1)}(1) = \frac{\pi^2}{3}, $$
we can compute $\frac{\partial^2 I}{\partial a^2}(\frac{1}{2}, \frac{1}{2})$ as follows:
\begin{align*} &\frac{\partial^2 I}{\partial a^2} (1/2, 1/2) \\ &= \frac{1}{4} \beta''(1/2) - \left(\frac{\pi}{2}-\log 2\right) \beta'(1/2) - \left(\frac{\pi^2}{2} + \pi\log 2-\log^2 2\right) \beta(1/2) \\ &= \frac{1}{4} \pi \left( 16\log^2 2 + \frac{\pi^2}{3} \right) \\ &\qquad - \left(\frac{\pi}{2}-\log 2\right) (-4\pi \log 2) - \left(\frac{\pi^2}{2} + \pi\log 2-\log^2 2\right) \pi \\ &= -\left(\frac{5\pi^3}{12} - \pi^2\log2 - \pi \log^2 2 \right). \end{align*}
I am also trying a more direct approach. Using contour integral, we can check that
$$ \int_{0}^{\frac{\pi}{2}} x^2 \sqrt{\cot x} \, \mathrm{d}x = \frac{1}{3\sqrt{2}} \left( \frac{\pi^3}{4} - \frac{3\pi}{2} \int_{0}^{1} \frac{\operatorname{artanh}^2 t}{\sqrt{t}} \, \mathrm{d}t \right). $$
So I am tackling the last integral, but have no good news at this point.
$\newcommand{\angles}[1]{\left\langle\,{#1}\,\right\rangle} \newcommand{\braces}[1]{\left\lbrace\,{#1}\,\right\rbrace} \newcommand{\bracks}[1]{\left\lbrack\,{#1}\,\right\rbrack} \newcommand{\dd}{\mathrm{d}} \newcommand{\ds}[1]{\displaystyle{#1}} \newcommand{\expo}[1]{\,\mathrm{e}^{#1}\,} \newcommand{\half}{{1 \over 2}} \newcommand{\ic}{\mathrm{i}} \newcommand{\iff}{\Longleftrightarrow} \newcommand{\imp}{\Longrightarrow} \newcommand{\Li}[1]{\,\mathrm{Li}_{#1}} \newcommand{\mc}[1]{\mathcal{#1}} \newcommand{\mrm}[1]{\mathrm{#1}} \newcommand{\ol}[1]{\overline{#1}} \newcommand{\pars}[1]{\left(\,{#1}\,\right)} \newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}} \newcommand{\root}[2][]{\,\sqrt[#1]{\,{#2}\,}\,} \newcommand{\totald}[3][]{\frac{\mathrm{d}^{#1} #2}{\mathrm{d} #3^{#1}}} \newcommand{\ul}[1]{\underline{#1}} \newcommand{\verts}[1]{\left\vert\,{#1}\,\right\vert}$ It's quite clear the OP already evaluated the integral $\ds{\color{#f00}{\int_{0}^{\pi/2}x^{2}\root{\cot\pars{x}}\,\dd x}}$. However, the OP is $\texttt{still looking for a simple evaluation}$ which start from $\ds{\,\mrm{I}\pars{a,b}\ \pars{~\mbox{see below}~}}$. Hereafter we presente a 'relatively simple' evaluation of it.
$\ds{\mrm{I}\pars{a,b} = {\pi\,2^{b}\,\cos\pars{\pi a/2}\Gamma(1 - b) \over \Gamma\pars{a/2 - b/2 + 1}\Gamma\pars{-a/2 - b/2 + 1}}\,,\qquad\Gamma\pars{\half} = \root{\pi}}$
\begin{align} \mrm{I}\pars{a,\half} & = \root{2}\pi^{3/2}\,\, {\cos\pars{\pi a/2} \over \Gamma\pars{3/4 + a/2}\Gamma\pars{3/4 - a/2}} \end{align}
Note that $$ \color{#f00}{\int_{0}^{\pi/2}x^{2}\root{\cot\pars{x}}\,\dd x} = -\,{\root{2} \over 8}\braces{2\bracks{\epsilon^{2}}\mrm{I}\pars{\half + \epsilon,\half}} $$ where \begin{align} \mrm{I}\pars{\half + \epsilon,\half} & = \pi^{3/2}\,\,{\cos\pars{\pi\epsilon/2} - \sin\pars{\pi\epsilon/2} \over \Gamma\pars{1 + \epsilon/2}\Gamma\pars{1/2 - \epsilon/2}} = \pi^{3/2}\,\,{\cos\pars{\pi\epsilon/2} - \sin\pars{\pi\epsilon/2} \over \pars{\epsilon/2}!\pars{-1/2 - \epsilon/2}!} \\[5mm] & = \pi^{3/2}\,{1 \over \pars{-1/2}!}{-1/2 \choose \epsilon/2} \bracks{\cos\pars{\pi\epsilon/2} - \sin\pars{\pi\epsilon/2}} \\[5mm] & = \pi\,{-1/2 \choose \epsilon/2} \bracks{\cos\pars{\pi\epsilon \over 2} - \sin\pars{\pi\epsilon \over 2}} \end{align}
We just need the binomial and 'the inside brackets term' expansion up to $\ds{\epsilon^{2}}$. The binomial expansion, up to order $\ds{\epsilon^{2}}$, is simple but laborious: It is simplified by using the Digamma value $\ds{\Psi\pars{1/2} = -\gamma - 2\ln\pars{2}}$. $\ds{\gamma}$: Euler-Mascheroni Constant.
$$ \left\{\begin{array}{rcl} \ds{-1/2 \choose \epsilon/2} & \ds{=} & \ds{1 - \ln\pars{2}\,\epsilon + \bracks{\half\,\ln^{2}\pars{2} - {\pi^{2} \over 12}}\epsilon^{2} + \,\mrm{O}\pars{\epsilon^{3}}} \\[3mm] \ds{\cos\pars{\pi\epsilon \over 2} - \sin\pars{\pi\epsilon \over 2}} & \ds{=} & \ds{1 - {\pi \over 2}\,\epsilon - {\pi^{2} \over 8}\,\epsilon^{2} + \,\mrm{O}\pars{\epsilon^{3}}} \end{array}\right. $$
\begin{align} &\color{#f00}{\int_{0}^{\pi/2}x^{2}\root{\cot\pars{x}}\,\dd x} \\[5mm] = &\ -\,{\root{2} \over 8}\,\times 2\times \pi\braces{% 1\times\pars{-\,{\pi^{2} \over 8}} + \bracks{-\ln\pars{2}}\pars{-\,{\pi \over 2}} + \bracks{\half\,\ln^{2}\pars{2} - {\pi^{2} \over 12}}\times 1} \\[5mm] & = \color{#f00}{{\root{2} \over 8}\bracks{{5\pi^{3} \over 12} - \pi^{2}\ln\pars{2} - \pi\ln^{2}\pars{2}}} \approx 0.8077 \end{align}