A proof of: The derivative of the determinant is the trace
Since it seems you want to understand all the various identifications involved, let me introduce some notation to try and clarify what is going on.
Let $V$ be a finite dimensional real vector space (endowed with the natural smooth structure) and let $U \subseteq V$ be an open subset. Assume we are given a smooth function $F \colon U \rightarrow \mathbb{R}$. Then we can calculate three a priori distinct things:
- We can calculate the standard multivariable directional derivative of $F$ at a point $p \in U$ in the direction $v \in V$. I'll denote it by $$ DF|_{p}(v) := \lim_{t \to 0} \frac{F(p + tv) - F(p)}{t}. $$ Note that $DF|_{p} \colon V \rightarrow \mathbb{R}$.
- We can treat $U$ as a smooth manifold and calculate the differential of $F$ at a point $p \in U$. This will give us a map $dF|_{p} \colon T_p U \rightarrow \mathbb{R}$.
- We can treat both $U$ and $\mathbb{R}$ as smooth manifolds and calculate the "full differential" of $F$ at a point $p$ which I'll denote by $F_{*}$. This will give us a map $F_{*}|_{p} \colon T_p U \rightarrow T_{F(p)} (\mathbb{R})$.
What is the relation between the three distinct maps? Like you noted, we have natural isomorphisms $f \colon T_pU \rightarrow V$ and $g \colon T_{F(p)}(\mathbb{R}) \rightarrow \mathbb{R}$. In terms of those isomorphisms, we have the relations
$$ dF|_{p} = DF|_{p} \circ f, \,\,\, dF|_{p} = g \circ F_{*}|_{p}, \,\,\, g^{-1} \circ DF|_{p} \circ f = F_{*}|_{p}. $$
Now, in your case, we have $F = \det$ and $p = I$. By the formula you are given, it seems that you are asked to calculate $DF|_{p}$. Instead, you have tried to calculate $F_{*}|_{p}$, hence the need for isomorphisms $f$ and $g$ to relate the formula you are asked to show with your calculation. Note that like Qiaochu said in his comment, it is much easier to calculate $DF|_{p}(X)$ without choosing a basis but your more complicated calculation agrees with the expected result.