Definition of Lie Algebra
Two good books are:
- Introduction to Lie Algebras by Humphreys.
- Introduction to Lie Algebras by Erdmann and Wildon.
The most natural examples of Lie algebras are the matrices and the Lie bracket is the usual commutator $[X,Y] = XY-YX$.
Moreover, the real motivation comes from the study of Lie groups. A Lie algebra of a Lie group is precisely the set of all left invariant vector fields and they have a natural Lie bracket.
There is also some motivation from Physics: see Poisson equation for example.
Lie algebras arise naturally in multiple contexts. Suppose you have an associative algebra $A$ which is not necessarily commutative. Then a natural thing to do is study how far it deviates from commutativity. That is, it is natural to consider the bracket $[X,Y]=XY-YX$ which measures the failure of commutativity. But now that we've defined this object, we might wonder what properties it satisfies. You can verify through direct computation that this bracket satisfies the Jacobi identity $[X,[Y,Z]]+[Z,[X,Y]]+[Y,[Z,X]]=0$ and also that $[X,X]=0$. So any associative algebra can be made into a Lie algebra.
Another reason the Jacobi identity is natural is that it is an example of an operation satisfying the product rule familiar from differentiation (the Leibniz rule.) You know that the derivative satisfies $D[fg]=(Df)g+f(Dg)$ and any operator $D$ on an algebra satisfying this rule is said to be a derivation. Now one way to write the Jacobi identity is: $[X,[Y,Z]=[[X,Y],Z]+[Y,[X,Z]]$. That is, the function $D_X$ defined by $D_X(Y)=[X,Y]$ satisfies the Leibniz rule when applied to $[Y,Z]$: $D_X[Y,Z]=[D_XY,Z]+[X,D_YZ]$.
$\mathfrak{gl}(n,\mathbb{R})$ are the most natural examples of Lie algebras. However I never feel they explains why we should study Lie algebras with a $multiplication$ rule as strange as the Lie bracket (it is not commutative nor associative). I only understand this after reading Barry Simon's Representations of finite and compact groups.
This book has a different flavour from most books on Lie algebras in that it is written from an analyst's perspective. As a result, everything is done on $\mathbb{C}$ or $\mathbb{R}$ or $\mathbb{H}$ (so no fields with positive characteristics), and most proof are computational (not conceptual). Since a lot of people doing Lie theory nowadays are trying to apply the theory to number theory or algebraic geometry, Barry left out some important themes, but his book gives a very nice motivation for the study of Lie theory. After all, it is for the study of differential equations that Lie himself invented this theory.
Now come back to your question. Let $M$ be a smooth manifold and $X$ a vector field on $M$. The existence of differential equations says that
For each $p\in M$ we have $\delta_p>0$, and a smooth function \begin{equation} \phi_p:(-\delta_p,\delta_p)\to M\end{equation} such that $\phi_p(0)=p$ and $\frac{d}{dt}\phi(t)=X(\phi_p(t))$.
That is, given a smooth vector field on a smooth manifold, then at each point we have a local flow that has this vector field as its velocity.
Let's just assume $M$ is really nice and we can take $\delta_p=\infty$ for all $p$ (each flow can be extended forever). Then for fixed $t$ this flow defines the exponential map $\exp(tX):M\to M$ that maps \begin{equation} p\mapsto \phi_p(t). \end{equation} For each $p\in M$, this function tells you where you are after travelling along $X$ after $t$ seconds.
This map on $M$ induces a map $\exp(tX)_*$ on the tangent bundle. For each tangent vector $v_p$ at $p$ you get an tangent vector at $\exp(tX)(p)$ by pushing forward $v_p$ along $X$. Then a general fact from differential geometry says
\begin{equation} \frac{d}{dt}|_{t=0}\{\exp(tX)_{*}Y\}=X\circ Y-Y\circ X \end{equation} where $Y$ is another vector field and $\circ$ denotes teh composition of two vector fields (vector fields acting on smooth functions by differentiation). (For this fact and the push-forward, see the great answer here).
So you see, you have the Lie bracket here. Since the left-hand side is the derivative of $Y$ along $X$, you see the Lie bracket does arise naturally (at least as natural as derivatives).
Now let $M$ have some more structures, say, it is a Lie group, which is just a group and meanwhile a manifold. Then the exponential map has much better structures. For instance, if you look at a very special type of vector fields, the translation-invariant vector fields (just constant vector fields if you are looking at Euclidean spaces, and this vector space $N$ is the Lie algebra we are studying), then the exponential map is a diffeomorphism around the unity of the group and the origin of $N$.
Well, to conclude, my guess is: in the first place we want to study Lie groups (like $\mathbb{R}^2$ or $\mathbb{S}^1$), then we found a very good linear object, invariant vector fields/ Lie algebras, that pretty much characterizes the structure of Lie groups via the exponential map. And the Lie bracket happens to be the most natural operation on this linear structure (it is just taking derivative of a vector field along another).