Does obtaining Lie algebras via differentiation work for a general Lie group?

The matrix groups can be defined as regular level sets of functions $f: GL(n) \to M$ where $M = \mathbb R$ (e.g. for $SL(n)$) or $M = n\times n$ matrices (e.g. for $O(n)$). In general, if you have a map $f: M \to N$ and $y \in N$ is a regular value then $f^{-1}(y)$ is a submanifold and $T_xf^{-1}(y) = \ker df_x : T_x M \to T_y N$. So in the case of $O(n)$ for example you're differentiating the function $X \mapsto XX^t$ at the identity to see that its Lie algebra (identifiable with $T_e O(n)$ is skew-symmetric matrices.

A Lie group is also a differentiable manifold; in particular we can define its tangent space at the identity. Intuitively, the tangent space is the set of directions $v$ such that if you start at the identity in $G$ and move infinitesimally in direction $v$, you stay in $G$.

Let's say that $G$ is defined by the vanishing of some differentiable function $f$: so $G = \{ X: f(X)=0\}$. The tangent space consists of matrices $M$ such that $f(I + \epsilon M) =0$ to first order in $\epsilon$: using the Taylor expansion and neglecting higher terms, what we want is that $df_I(M)=0$ where $df_I$ is the total derivative at the identity. Elements of the kernel of the total derivative are exactly those vectors which can be realised as $\gamma'(0)$ for some differentiable $\gamma: (-1,1)\to G$: the usual definition of tangent space is the equivalence classes of such functions $\gamma$ under the relation $\gamma_1\sim \gamma_2$ iff $\gamma_1'(0)=\gamma_2'(0)$.

The Lie algebra associated to $G$ has as its underlying vector space the tangent space at the identity, so it really is obtained by differentiating and setting the total derivative equal to zero.

Looking at your $O(n)$ example, the "$f$" is $f(X)=XX^t-I$. To find the total derivative at the identity, we ought to have $f(I+\epsilon M) = I + \epsilon df_I(M) +$ higher order terms. To order one in epsilon we have $f(I+\epsilon M)= \epsilon (M + M^t)$, so the total derivative at the identity is $M+M^t$. The vanishing of this is exactly the condition you gave for being in the Lie algebra of $O(n)$.