Second (and higher) derivatives of maps between manifolds

Your doubt is well-placed: Even in the case $f:M\to \mathbb R$ you cannot sensibly define $D^2_u f : T_u M \times T_u M \to \mathbb R$ unless $u$ is a critical point of $f$. Otherwise your definition $$D^2 f(\partial_i, \partial_j)=\frac{\partial^2 f}{\partial x^i \partial x^j} \ \ \text{extended by multilinearity}$$ depends on the coordinate system $x^i$: if we have some other coordinate system $y^\alpha$ then by multilinearity we should have $$ D^2 f\left(\frac{\partial}{\partial x^{i}},\frac{\partial}{\partial x^{j}}\right) =\frac{\partial y^{\alpha}}{\partial x^{i}}\frac{\partial y^{\beta}}{\partial x^{j}}D^{2}f\left(\frac{\partial}{\partial y^{\alpha}},\frac{\partial}{\partial y^{\beta}}\right)=\frac{\partial y^{\alpha}}{\partial x^{i}}\frac{\partial y^{\beta}}{\partial x^{j}}\frac{\partial^{2}f}{\partial y^{\alpha}\partial y^{\beta}}; $$ but the chain rule gives $$\frac{\partial^{2}f}{\partial x^{j}\partial x^{i}}=\frac{\partial}{\partial x^{i}}\left(\frac{\partial y^{\beta}}{\partial x^{j}}\frac{\partial f}{\partial y^{\beta}}\right)=\frac{\partial y^{\alpha}}{\partial x^{i}}\frac{\partial y^{\beta}}{\partial x^{j}}\frac{\partial^{2}f}{\partial y^{\alpha}\partial y^{\beta}}+\frac{\partial^{2}y^{\beta}}{\partial x^{i}\partial x^{j}}\frac{\partial f}{\partial y^{\beta}}.$$

In Riemannian geometry this is fixed by using the covariant Hessian $$D^2f (X,Y) = XYf - (\nabla_X Y) f;$$ but without the additional structure of a conenction there is no way to think of $k^\text{th}$-order derivatives as taking $k$ tangent vectors.

Instead the general formulation uses higher tangent bundles: the first derivative is a map $TM \to TN$, so the second derivative is a map $TTM \to TTN$. Thus the picture you should have in your head is that the first input (a vector $v \in T_u M$) is the base direction you're moving in, and then the second input (a vector in $T_v TM$) is the direction you're varying that $v$ in.