Derivative of matrix exponential wrt to each element of Matrix

Considering the expression $x = \exp(tA)$ I can think of two derivatives.

First, the derivative with respect to the real variable $t$ of the matrix-valued function $t \mapsto \exp(tA)$. Here the result is easily derived from direct calculation of the series definition of the matrix exponential: \begin{align} \frac{d}{dt} \exp(tA) &= \frac{d}{dt} \left[ I+tA+\frac{1}{2}t^2A^2+\frac{1}{3!}t^3A^3+ \cdots\right] \\ &= A+tA^2+\frac{1}{2}t^2A^3+ \cdots \\ &= A\exp(tA) \end{align} Thus, $\frac{d}{dt} \exp(tA) = A\exp(tA)$. (Edited to fix typo)

Second, we can differentiate with respect to the component $A_{ij}$ of $A = \sum A_{ij} E_{ij}$ where $E_{ij}=e_ie_j^T$ is the matrix which is zero except in the $ij$-th spot where there is a $1$. In other words, $(E_{ij})_{kl} = \delta_{ik}\delta_{jl}$. I'll look at the derivative as a directional derivative essentially: calculate the difference along the $E_{ij}$ direction: considering $f(t,A)=\exp(tA)$ $$ \frac{\partial f}{\partial A_{ij}}= \lim_{h \rightarrow 0}\frac{1}{h} \left[\exp(tA)-\exp(t(A+hE_{ij}))\right]$$ I expect this can be simplified.

Ok, the matrix exponential satisfies the Baker-Campbell-Hausdorf relation: $$ \exp(A)\exp(B) = \exp\left(A+B+ \frac{1}{2}[A,B] + \cdots\right)$$ From this we derive the Zassenhaus formula, $$ \exp(A+B) = \exp(A)\exp(B)\exp\left(-\frac{1}{2}[A,B] + \cdots\right) $$ I'll use this to simplify $\exp( t(A+hE_{ij})) = \exp\left(tA+ thE_{ij}\right)$ $$ \exp\left(tA+ thE_{ij}\right) = \exp(tA)\exp\left( thE_{ij}\right) \exp\left( -\frac{1}{2}[tA,thE_{ij}]+ \cdots\right)$$ hence $$ \exp\left(tA+ thE_{ij}\right) = \exp(tA)\exp\left( thE_{ij} -\frac{1}{2}[tA,thE_{ij}]+ \cdots\right)$$ where I am omitting terms with $h^2,h^3,\dots$ as those vanish in the limit and I am also omitting terms with nested commutators of $A$ so the answer below is just the first couple terms in an infinite series flowing from the BCH relation. \begin{align} \frac{\partial f}{\partial A_{ij}}&= \lim_{h \rightarrow 0}\frac{1}{h} \left[\exp(tA)-\exp(t(A+hE_{ij}))\right] \\ &=\lim_{h \rightarrow 0}\frac{1}{h} \left[\exp(tA)-\exp(tA)\exp\left( thE_{ij} -\frac{1}{2}[tA,thE_{ij}]+ \cdots\right) \right] \\ &=\exp(tA)\lim_{h \rightarrow 0}\frac{1}{h} \left[I-\exp\left( thE_{ij} -\frac{1}{2}[tA,thE_{ij}]+ \cdots\right) \right] \\ &=\exp(tA)\lim_{h \rightarrow 0}\frac{1}{h} \left[I-I-thE_{ij} +\frac{1}{2}[tA,thE_{ij}]+ \cdots \right] \\ &=\exp(tA)\left[-tE_{ij} +\frac{t^2}{2}[A,E_{ij}]+ \cdots \right]. \end{align} Note the terms linear in $h$ do survive the limit and there are such terms (indicated by the $+ \cdots$) stemming from $[tA,[tA,hE_{ij}]]$ and $[tA,[tA,[tA,hE_{ij}]]]$ etc. Now, you can calculate: $[A,E_{ij}] = \sum_{k=1}^n \left(A_{ki}E_{kj}-A_{jk}E_{ik} \right)$ so, $$ \frac{\partial f}{\partial A_{ij}} = \exp(tA) \left[-tE_{ij}+ \frac{t^2}{2}\left(A_{ki}E_{kj}-A_{jk}E_{ik} +\cdots \right)\right]$$ For what it's worth, you can simplify the nested commutator: $$ [A,[A,E_{ij}]] = \sum_{k,l=1}^n \left( A_{lk}A_{ki}E_{lj}-2A_{ki}A_{jl}E_{kl}+A_{jl}A_{lk}E_{ik} \right)$$ Or, in Einstein notation, $$ [A,[A,E_{ij}]] = (A^2)_{li}E_{lj}-2A_{ki}A_{jl}E_{kl}+(A^2)_{jk}E_{ik}.$$ Anyway, I hope this helps. Notice if $i=j$ and $A$ is diagonal or if simply $[A,E_{ij}]=0$ then we obtain: $$ \frac{\partial }{\partial A_{ij}} \exp(tA) = -t\exp(tA)E_{ij}.$$ which makes me think I've made a sign-error somewhere*. If someone sees it and messages me about it I would be happy.

*The sign-error lies in the definition equation of $\frac{\partial f}{\partial A_{ij}}$. It should be $$ \frac{\partial f}{\partial A_{ij}}= \lim_{h \rightarrow 0}\frac{1}{h} \left[\exp(t(A+hE_{ij})) - \exp(tA)\right].$$

If $A\in{\mathbb R}^{n\times n}$, then you can use Higham's "Complex Step Approximation" to calculate each component $$ \frac {\partial f} {\partial A_{jk}} = {\rm Im}\bigg(\frac{f(A+ihE_{jk})}{h}\bigg) $$ where $f(A)={\rm exp}(tA)$ and $h=10^{-20}$.