Historical reference request on Nilpotent groups
In 1870, the American mathematician, Benjamin Pierce first introduced the term nilpotent in the context of his work on the classification of Algebras. In Algebra, an element $x$ of a ring $R$ is said to be nilpotent if there exists some positive integer $n$ such that $x^{n}=0$.
In group theory, a nilpotent group is a group having a special property that makes it 'almost abelian' through repeated application of the commutator operation defined by $[x,y]=xyx^{-1}y^{-1}$
For justification of the term nilpotent, start with a nilpotent group $G$ and an element $g$ of $G$ and define a function $ f : G \longrightarrow G $ by $f(x) = [x, g] = xgx^{-1}g^{-1}$. This function is sometimes referred to as being the adjoint action. Then this function is nilpotent in the sense that there exists a natural number $n$ such that $f^{n}$, the $n$-th iteration of $f$ sends every element $x$ of $G$ to the identity element.