How to calculate the gradient of log det matrix inverse?

I assume that you are asking for the derivative with respect to the elements of the matrix. In this cases first notice that

$$\log \det X^{-1} = \log (\det X)^{-1} = -\log \det X$$

and thus

$$\frac{\partial}{\partial X_{ij}} \log \det X^{-1} = -\frac{\partial}{\partial X_{ij}} \log \det X = - \frac{1}{\det X} \frac{\partial \det X}{\partial X_{ij}} = - \frac{1}{\det X} \mathrm{adj}(X)_{ji} = - (X^{-1})_{ji}$$

since $\mathrm{adj}(X) = \det(X) X^{-1}$ for invertible matrices (where $\mathrm{adj}(X)$ is the adjugate of $X$, see http://en.wikipedia.org/wiki/Adjugate).

Or you can check section A.4.1 of the book Stephen Boyd, Lieven Vandenberghe, Convex Optimization for an alternative solution, where they compute the gradient without using the adjugate.

The simplest is probably to observe that $$-\log\det (X+tH) = -\log\det X -\log\det(I+tX^{-1}H) \\= -\log\det X - t \textrm{Tr}(X^{-1}H) + o(t),$$

where is used the "obvious" fact that $\det(I+A) = 1+\textrm{Tr}(A)+o(|A|)$ (all the other terms are quadratic expressions of the coefficients of $A$).

Notice that $\textrm{Tr}(X^{-1}H)=(X^{-T},H)$ in the Frobenius scalar product, hence $\nabla [-\log\det(X)] = -X^{-T}$ in this scalar product. (This gives another proof that $\nabla\det (X) = cof(X)$.)

Of course if $X$ is symmetric positive definite then $-X^{-1}$ is also a valid expression. Moreover, one has in this case, for $X,Y$ positive definite, $(-X^{-1}+Y^{-1},X-Y)\ge 0$.