Analytical solution to system of linear ODE's

First, let’s deal with the errors in the way you applied (13) to this problem. You need to be more careful about reordering terms: matrix multiplication is in general not commutative. You’ve switched the order of $x^{0,j}$, which is a $3\times1$ vector, and a $3\times3$ matrix, so the product in your expression doesn’t make sense.

Also, it looks like you’ve somehow combined the contributions of the two eigenvalues into one term. The inner sum in particular is different for different multiplicities. Taking first the eigenvalue $3$ with algebraic multiplicity $1$, the inner sum for this eigenvalue is simply $I$. Its contribution to the solution is therefore just $e^{3t}x^{0,3}$. (I’ve changed the notation a bit—instead of using an index $j$, I’m labeling the components of $\xi$ with the eigenvalues.) The other eigenvalue $0$ has algebraic multiplicity $2$, so the inner sum is $I+tA$, and its contribution to the solution is $(I+tA)x^{0,0}$, therefore the solution to this initial value problem is $$e^{3t}x^{0,3}+(I+tA)x^{0,0}.$$

To find the components $x^{0,j}$ of $\xi$, you basically need to apply the same change of basis to $\xi$ that you would to $A$ to convert it into its Jordan normal form. So, let $$P=\begin{bmatrix}1&1&4\\-2&-1&4\\1&0&1\end{bmatrix}$$ so that $$P^{-1}AP=\begin{bmatrix}0&1&0\\0&0&0\\0&0&3\end{bmatrix}.$$ The elements of $P^{-1}\xi$ are the coordinates of $\xi$ relative to this basis, and the $x^{0,j}$ are constructed via corresponding linear combinations of the three generalized eigenvectors. Now, since the product of a matrix and vector can be interpreted as a linear combination of the columns of the matrix with coefficients given by the elements of the vector, we can write $$x^{0,0}=P\begin{bmatrix}1&0&0\\0&1&0\\0&0&0\end{bmatrix}P^{-1}\xi = P_0\xi\\ x^{0,3}=P\begin{bmatrix}0&0&0\\0&0&0\\0&0&1\end{bmatrix}P^{-1}\xi = P_3\xi.$$ The matrices $P_0$ and $P_3$ are projectors onto the respective generalized eigenspaces with the property that $P_0P_3=P_3P_0=0$.

---

Addendum: If $\xi = C_1v_1+C_2v_1^{(1)}+C_3v_2$ then formula (13) gives $$x(t) = (I+tA)\left(C_1v_1+C_2v_1^{(1)}\right)+C_3e^{3t}v_2$$ as the solution to this IVP, but we can simplify this. Recall that $v_1$ is an eigenvector with eigenvalue $0$, so $Av_1=0$. Expanding the first term of $x(t)$, we therefore have $$\begin{align} x(T) &= C_1(I+tA)v_1+C_2(I+tA)v_1^{(1)}+C_3e^{3t}v_2 \\ &= C_1v_1+C_2(I+tA)v_1^{(1)}+C_3e^{3t}v_2. \end{align}$$