Minimum of the given expression

\begin{eqnarray*} (a-b)^2+(2-a-b)^2+(2a-3b)^2=6a^2-12ab+11b^2-4(a+b)+4 \\ =6\left(a-b-\frac{1}{3}\right)^2+5\left(b-\frac{4}{5}\right)^2+\color{red}{\frac{2}{15}}. \end{eqnarray*}

Let $a=\frac{17}{15}$ and $b=\frac{4}{5}$.

Hence, we get a value $\frac{2}{15}$.

Thus, it remains to prove that $$(a-b)^2 + (2-a-b)^2 + (2a-3b)^2\geq\frac{2}{15}$$ or $$10(3a-3b-1)^2+3(5b-4)^2\geq0$$ Done!

I got my solution by the following way.

We need to find a maximal $k$ for which the following inequality is true for all reals $a$, $b$ and $c$. $$(a-b)^2 + (2-a-b)^2 + (2a-3b)^2\geq k$$ or $$6a^2-4(3b+1)a+11b^2-4b+4-k\geq0,$$ for which we need $$4(3b+1)^2-6(11b^2-4b+4-k)\leq0$$ or $$15b^2-24b+10-3k\geq0,$$ for which we need $$12^2-15(10-3k)\leq0$$ or $$k\leq\frac{2}{15}.$$ The equality occurs for $k=\frac{2}{15}$, $b=\frac{24}{2\cdot15}$, which is $b=\frac{4}{5}$ and for these values we obtain $$(a-b)^2 + (2-a-b)^2 + (2a-3b)^2\geq \frac{2}{15}$$ it's $$6a^2-4(3b+1)a+11b^2-4b+4-\frac{2}{15}\geq0$$ or $$90a^2-60(3b+1)a+165b^2-60b+58\geq0$$ or $$10(9a^2-6(3b+1)a+(3b+1)^2)-10(3b+1)^2+165b^2-60b+58\geq0$$ or $$10(3a-3b-1)^2+75b^2-120b+48\geq0$$ or $$10(3a-3b-1)^2+3(5a-4)^2\geq0.$$

$$(a-b)^2 + (2-a-b)^2 + (2a-3b)^2 = \left\| \,\, \begin{bmatrix} 1 & -1\\ 1 & 1\\ 2 & -3\end{bmatrix} \begin{bmatrix} a\\ b\end{bmatrix} - \begin{bmatrix} 0\\ 2\\ 0\end{bmatrix} \,\, \right\|_2^2$$

This is a least-squares problem. Since the matrix has full column rank, the minimum is

$$\left\| \,\, \begin{bmatrix} 1 & -1\\ 1 & 1\\ 2 & -3\end{bmatrix} \left( \begin{bmatrix} 1 & -1\\ 1 & 1\\ 2 & -3\end{bmatrix}^\top \begin{bmatrix} 1 & -1\\ 1 & 1\\ 2 & -3\end{bmatrix} \right)^{-1} \begin{bmatrix} 1 & -1\\ 1 & 1\\ 2 & -3\end{bmatrix}^\top \begin{bmatrix} 0\\ 2\\ 0\end{bmatrix} - \begin{bmatrix} 0\\ 2\\ 0\end{bmatrix} \,\, \right\|_2^2 = \color{blue}{\frac{2}{15}}$$

---

SymPy code

>>> from sympy import *
>>> A = Matrix([[ 1,-1],
                [ 1, 1],
                [ 2,-3]])
>>> b = Matrix([0,2,0])
>>> error = A * (A.T * A)**-1 * A.T * b - b

The squared Euclidean norm of the error vector is

>>> error.T * error
Matrix([[2/15]])