Squaring a Vector?
$$ {\bf r} \cdot {\bf r} = \left|{\bf r}\right|^2 = \sum_{i,j} r_i r_j {\bf e}_i \cdot {\bf e}_j = \sum_{i,j} r_i r_j \delta_{ij} = \sum_i r_i r_i $$ is often written as ${\bf r}^2$, when it's clear that you don't mean $$ {\bf r}{\bf r} = \sum_{i,j} r_i r_j {\bf e}_i {\bf e}_j $$ which is a perfectly valid quantity [1].
As for ${\bf r} = {\bf x} + {\bf y} + {\bf z}$, $$ \begin{eqnarray} {\bf r} \cdot {\bf r} &=& {\bf x} \cdot {\bf x} + {\bf x} \cdot {\bf y} + {\bf x} \cdot {\bf z} + {\bf y} \cdot {\bf x} + {\bf y} \cdot {\bf y} + {\bf y} \cdot {\bf z} + {\bf z} \cdot {\bf x} + {\bf z} \cdot {\bf y} + {\bf z} \cdot {\bf z} \\ &=& {\bf x} \cdot {\bf x} + {\bf y} \cdot {\bf y} + {\bf z} \cdot {\bf z} + 2\left( {\bf x} \cdot {\bf y} + {\bf x} \cdot {\bf z} + {\bf y} \cdot {\bf z}\right) \end{eqnarray} $$ equals ${\bf x} \cdot {\bf x} + {\bf y} \cdot {\bf y} + {\bf z} \cdot {\bf z}$ for all $\bf x$, $\bf y$, $\bf z$ iff ${\bf x} \cdot {\bf y} + {\bf x} \cdot {\bf z} + {\bf y} \cdot {\bf z} = 0$ -- a special case is when ${\bf x} \cdot {\bf y} = {\bf x} \cdot {\bf z} = {\bf y} \cdot {\bf z} = 0$.
- For example, both the dot product and regular product are used in the expression for the inertia tensor: $$ \int d^3{\bf r} \ \rho\left({\bf r}\right) \left[\left({\bf r} \cdot {\bf r}\right) {\bf I} - {\bf r}{\bf r}\right] $$ where $$ {\bf I} = \sum_{i,j} \delta_{ij} {\bf e}_i {\bf e}_j $$
Why don't you work out the expansion yourself.
$$ \vec{r}\cdot \vec{r} = \left( \vec{x}+\vec{y}+\vec{z} \right) \cdot \left( \vec{x}+\vec{y}+\vec{z} \right) =\\= (\vec{x} \cdot \vec{x})+(\vec{y}\cdot\vec{y}) + (\vec{z}\cdot\vec{z}) + 2(\vec{x} \cdot\vec{z}) +2(\vec{y}\cdot\vec{z}) + 2(\vec{x} \cdot\vec{y}) = \\ = \|\vec{x}\|^2 + \|\vec{y}\|^2+\|\vec{z}\|^2 + 2 \left((\vec{x} \cdot\vec{z}) +(\vec{y}\cdot\vec{z}) + (\vec{x} \cdot\vec{y}) \right)$$
IF the basis vectors are orthogonal then all the inner products are zero between vectors and you have $$|\vec{r}|^2 = \|\vec{x}\|^2 + \|\vec{y}\|^2+\|\vec{z}\|^2$$
otherwise no.
First if nothing is said you can assume that $\vec{x}\vec{y}$ is a dot product.
What you said ${(\vec{x}+\vec{y}+\vec{z})}^{2}=x^2+y^2+z^2$ is only true if the vectors $\vec{x},\vec{y},\vec{z}$ are perpendicular to each other but in the general case:
$$ {\vec{r}}^{2}={(\vec{x}+\vec{y}+\vec{z})}^{2}\\ =x^2+y^2+z^2+2(xy\cos{\alpha}+yz\cos{\beta}+xz\cos{\gamma}) $$
Where : $$ x=||\vec{x}|| $$