Raising $e$ to the power of a matrix

Yes, the exponential of a matrix can be defined by that Taylor series, and it is a very useful thing. See e.g. Wikipedia The proof that it converges is not difficult, using any sub-multiplicative matrix norm. If $\| \cdot \|$ is such a norm, then any power series $\sum_n c_n A^n$ converges whenever the real series $\sum_n |c_n| \|A\|^n$ converges.

My favourite definition - as I teach quite often ODEs - is using differential equations.

Let us first note that $x(t)=\mathrm{e}^{at}$ is the unique solution of the initial value problem: $$ x'=ax, \quad x(0)=1. $$ Generalise that a little bit: Let $A$ be a square matrix ($A\in\mathbb R^{n\times n}$). Define as $\mathrm{e}^{tA}$ the unique, and globally defined, solution of $$ X'=AX, \quad X(0)=I, $$ where $X$ is an $n\times n$ matrix, and $I$ the identity matrix. Through this definition a lot of properties of the exponential of a matrix, (which in order to prove require messy calculations) are proved very elegantly if we use the above definition.