Second derivative of $\det(\mathbb{1}+tA)$?

Edit: Here's a totally different argument, probably simpler, with more linear algebra and less calculus. Say $\lambda_1,\dots,\lambda_n$ are the eigenvalues of $A$, listed according to (algebraic) multiplicity. Since $\det(I+tA)$ is the product of the eigenvalues of $I+tA$ it follows that $$F(t)=\prod_{j=1}^n(1+t\lambda_j).$$Multiplying that out in the imagination makes it clear that $$F'(0)=\sum\lambda_j=\text{tr}(A),$$and $$F''(0)=2\sum_{j< k}\lambda_j\lambda_k.$$Since the eigenvalues of $A^2$ are $\lambda_j^2$ it follows that $$F''(0)=\left(\sum\lambda_j\right)^2-\sum\lambda_j^2=(\text{tr}(A))^2-\text{tr}(A^2),$$as below.

Update: There's a slightly iffy spot above that nobody seemed to notice. It's easy to see that $\lambda$ is an eigenvalue of $A^2$ if and only if $\lambda=\omega^2$ where $\omega$ is an eigenvalue of $A$, but above we need more than that: We need to know that the algebraic multiplicities are the same. Maybe that's obvious? (Ok, it's clear from the Jordan form, probably clear just from the fact that the algebraic multiplicity is the dimension of the "generalized eigenspace", but it should be easier than that...)

Cheap trick: It's clear if the eigenvalues are distinct. And matricies with distinct eigenvalues are dense, hence the trace of $A^2$ is $\sum\lambda_j^2$ for every $A$.

Or, in better taste: Since $\det(\lambda I-A)=\prod(\lambda-\lambda_j)$, $$\begin{align}\det(\lambda^2I-A^2)&=\det(\lambda I-A)\det(\lambda I+A) \\&=(-1)^n\prod(\lambda-\lambda_j)\prod(-\lambda-\lambda_j) \\&=\prod(\lambda^2-\lambda_j^2).\end{align}$$ Hence $\det(\lambda I-A^2)=\prod(\lambda-\lambda_j^2)$.

Thought of this because I was bothered by the fact that the expression for $F(t)$ derived below doesn't look like a polynomial:

Exercise Use the fact that $\text{tr}(A^k)=\sum_j\lambda_j^k$ to show that $\exp\left(t\,\text{tr}A-\frac{t^2}2\text{tr}(A^2)+\frac{t^3}3\text{tr}(A^3)+\dots\right)=\prod(1+t\lambda_j)$ (for small $t$).

Original: Yes, $F'(0)=\text{tr} A$.

Say $B(t)=I+tA$. Note that $B(t)$ is invertible if $t$ is small enough. So if $t$ and $h$ are both small then $$F(t+h)=\det(B(t)+hA) =\det(B(t))\det(I+hB(t)^{-1}A).$$Taking the derivative with respect to $h$ shows that $$\begin{align}F'(t)&=\det(B(t))\text{tr}(B(t)^{-1}A) \\&=F(t)\text{tr}((I-tA+t^2A^2-t^3A^3\dots)A) \\&=(1+t\,\text{tr}A+\dots)(\text{tr}A-t\,\text{tr}(A^2)+\dots).\end{align}$$Hence $$F''(0)=(\text{tr}A)^2-\text{tr}(A^2).$$

Bonus: There's a differential equation above, saying that $$F'/F=\text{tr}A-t\,\text{tr}(A^2)+\dots.$$With the initial condition $F(0)=1$ this shows that $$F(t)=\exp\left(t\,\text{tr}A-\frac{t^2}2\text{tr}(A^2)+\frac{t^3}3\text{tr}(A^3)+\dots\right),$$which should allow you to find as many derivatives as you want.

If in $A(t)=(a_{ij}(t))_{1\leq i,j \leq n}$ each of the components is a $k$ times differentiable real-valued function, then we have (see e.g. this link): \begin{align*} \frac{d^{k}}{dt^k} \begin{vmatrix} a_{11}(t)&a_{12}(t)&\cdots&a_{1n}(t)\\ a_{21}(t)&a_{22}(t)&\cdots&a_{2n}(t)\\ \vdots&\vdots&&\vdots\\ a_{n1}(t)&a_{n2}(t)&\cdots&a_{nn}(t)\\ \end{vmatrix} =\sum_{{k_1+k_2+\cdots+k_n=k}\atop{k_j\geq 0, 1\leq j\leq n}}\frac{k!}{k_1!k_2!\ldots k_n!} \begin{vmatrix} a_{11}^{(k_1)}(t)&a_{12}^{(k_1)}(t)&\cdots&a_{1n}^{(k_1)}(t)\\ a_{21}^{(k_2)}(t)&a_{22}^{(k_2)}(t)&\cdots&a_{2n}^{(k_2)}(t)\\ \vdots&\vdots&&\vdots\\ a_{n1}^{(k_n)}(t)&a_{n2}^{(k_n)}(t)&\cdots&a_{nn}^{(k_n)}(t)\\ \end{vmatrix} \end{align*}

Here we have the special case $n=2$ and $A(t)=1+tA$ with $A$ an $n\times n$ constant matrix. We obtain \begin{align*} &\color{blue}{\frac{d^{2}}{dt^2} \begin{vmatrix} 1+a_{11}t&a_{12}t&\cdots&a_{1n}t\\ a_{21}t&1+a_{22}t&\cdots&a_{2n}t\\ \vdots&\vdots&&\vdots\\ a_{n1}t&a_{n2}t&\cdots&1+a_{nn}t\\ \end{vmatrix}}\\ &\qquad\quad=\sum_{{k_1+k_2+\cdots+k_n=2}\atop{k_j\geq 0, 1\leq j\leq n}}\frac{2!}{k_1!k_2!\ldots k_n!} \begin{vmatrix} (1+a_{11}t)^{(k_1)}&(a_{12}t)^{(k_1)}&\cdots&(a_{1n}t)^{(k_1)}\\ (a_{21}t)^{(k_2)}&(1+a_{22}t)^{(k_2)}&\cdots&(a_{2n}t)^{(k_2)}\\ \vdots&\vdots&&\vdots\\ (a_{n1}t)^{(k_n)}&(a_{n2}t)^{(k_n)}&\cdots&(1+a_{nn}t)^{(k_n)}\\ \end{vmatrix}\\ &\qquad\quad\color{blue}{=2\sum_{{k_1+k_2+\cdots+k_n=2}\atop{0\leq k_j\leq 1, 1\leq j\leq n}}\frac{1}{k_1!k_2!\ldots k_n!} \begin{vmatrix} (1+a_{11}t)^{(k_1)}&(a_{12}t)^{(k_1)}&\cdots&(a_{1n}t)^{(k_1)}\\ (a_{21}t)^{(k_2)}&(1+ta_{22})^{(k_2)}&\cdots&(a_{2n}t)^{(k_2)}\\ \vdots&\vdots&&\vdots\\ (a_{n1}t)^{(k_n)}&(a_{n2}t)^{(k_n)}&\cdots&(1+a_{nn}t)^{(k_n)}\\ \end{vmatrix}} \end{align*} Observe the simplification in the last step in the index range, where we set $0\leq k_j\leq 1$. Since if there would be an index $k_j=2$ the corresponding second derivative produces zeros in the $j$-th row, so that the determinant evaluates to $0$.

Example: $n=2$

Let's do the calculation for small $n=2$. We get \begin{align*} \frac{d^{2}}{dt^2} \begin{vmatrix} 1+a_{11}t&a_{12}t\\ a_{21}t&1+a_{22}t\\ \end{vmatrix} &=2\sum_{{k_1+k_2=2}\atop{0\leq k_j\leq 1, 1\leq j\leq 2}}\frac{1}{k_1!k_2!} \begin{vmatrix} (1+a_{11}t)^{(k_1)}&(a_{12}t)^{(k_1)}\\ (a_{21}t)^{(k_2)}&(1+a_{22}t)^{(k_2)}\\ \end{vmatrix}\\ &=2 \begin{vmatrix} (1+a_{11}t)^{(1)}&(a_{12}t)^{(1)}\\ (a_{21}t)^{(1)}&(1+a_{22}t)^{(1)}\\ \end{vmatrix}\\ &=2 \begin{vmatrix} a_{11}&a_{12}\\ a_{21}&a_{22}\\ \end{vmatrix}\\ &=2(a_{11}a_{22}-a_{12}a_{21})\tag{1} \end{align*}

On the other hand we have \begin{align*} \frac{d^{2}}{dt^2} \begin{vmatrix} 1+a_{11}t&a_{12}t\\ a_{21}t&1+a_{22}t\\ \end{vmatrix} &=\frac{d^{2}}{dt^2}\left[(1+a_{11}t)(1+a_{22}t)-a_{12}a_{21}t^2\right]\\ &=\frac{d^{2}}{dt^2}\left[1+a_{11}t+a_{22}t+a_{11}a_{22}t^2-a_{12}a_{21}t^2\right]\\ &=2(a_{11}a_{22}-a_{12}a_{21}) \end{align*} in accordance with (1).

Note the answer of @DavidCUllrich coincides with the result above \begin{align*} \color{blue}{(\text{tr}A)^2-\text{tr}(A^2)} &=(a_{11}+a_{22})^2-\text{tr}\begin{pmatrix}a_{11}&a_{12}\\a_{21}&a_{22}\end{pmatrix}^2\\ &=(a_{11}+a_{22})^2-\text{tr}\begin{pmatrix}a_{11}^2+a_{12}a_{21}&\ast \\ \ast&a_{12}a_{21}+a_{22}^2\end{pmatrix}\\ &\,\,\color{blue}{=2(a_{11}a_{22}-a_{12}a_{21})} \end{align*}

_{The following is inspired by David C. Ullrich's answer, namely its bonus section. Recall that $\color{blue}{\det \big( \exp(\mathrm M) \big) = \exp \big( \mbox{tr} (\mathrm M) \big)}$.}

---

$$\begin{aligned} f (t) := \det ( \mathrm I_n + t \mathrm A ) &= \det \big( \exp \big( \ln \big( \mathrm I_n + t \mathrm A \big) \big) \big)\\ &= \exp \big( \underbrace{\mbox{tr} \big( \ln \big( \mathrm I_n + t \mathrm A \big) \big)}_{=: g (t)} \big) = \exp \big( g (t) \big) \end{aligned}$$

From the Maclaurin expansion of $\ln (1+x)$, we obtain

$$\begin{aligned} g (t) &= \mbox{tr} \big( t \mathrm A - \frac{t^2}{2} \mathrm A^2 + \frac{t^3}{3} \mathrm A^3 - \cdots \big)\\ &= t \,\mbox{tr} (\mathrm A) - \frac{t^2}{2} \mbox{tr} (\mathrm A^2) + \frac{t^3}{3} \mbox{tr} (\mathrm A^3) - \cdots\end{aligned}$$

and

$$g ' (t) = \mbox{tr} (\mathrm A) - t \, \mbox{tr} (\mathrm A^2) + t^2 \, \mbox{tr} (\mathrm A^3) - \cdots$$

and

$$g '' (t) = - \mbox{tr} (\mathrm A^2) + 2t \, \mbox{tr} (\mathrm A^3) - \cdots$$

Differentiating $f$ twice,

$$f '' (t) = \big( g''(t) + \left( g'(t) \right)^2 \big) \, f(t)$$

and, thus,

$$f''(0) = \big ( - \mbox{tr} (\mathrm A^2) + \left( \mbox{tr} (\mathrm A) \right)^2 \big) \, \underbrace{f(0)}_{=1} = \color{blue}{\big( \mbox{tr} (\mathrm A) \big)^2 - \mbox{tr} \left( \mathrm A^2 \right)}$$