Matrix ring $\cong \Bbb F_2[x]/x^2 = $ dual numbers over $\Bbb Z/2$

It's the case $\rm\ R = \mathbb F_2$ of $\rm R[t]/t^2\:.\ $ This is known as the algebra of dual numbers over the ring $\rm R$. Such rings and higher order analogs $\:\rm R[t]/t^n \;$ prove quite useful when studying (higher) derivations algebraically since such rings provide very convenient algebraic models of tangent / jet spaces. For example, they permit easy transfer of properties of homomorphisms to derivations -- see for example section 8.15 in Jacobson, Basic Algebra II. See this post for further discussion and links.

I'm not sure what would be natural in this context, but I would find it natural to denote $0 = \begin{pmatrix}0&0\\0&0\end{pmatrix}$, $1 = \begin{pmatrix}1&0\\ 0&1\end{pmatrix}$, $\epsilon = \begin{pmatrix}0&1\\0&0\end{pmatrix}$ , $1+\epsilon = \begin{pmatrix}1&1\\0&1\end{pmatrix}$ Where 0 is a additive unit, $1$ is a multiplicative unit and $\epsilon^2 = 0$.

So this ring arises as $\mathbb Z[\epsilon]/(2,\epsilon^2)$.

$R$ is a certain subring of the ring $B$ of upper triangular $2 \times 2$ matrices over $\mathbb{F}_2$. $B$ arises as the endomorphism ring of a complete flag in $(\mathbb{F}_2)^2$, so in that sense it's a natural geometric object to look at. The construction generalizes to a vector space over any field.

I'm not sure exactly what you mean by "natural situation," though, or why you should expect that every finite ring arises in such a situation. For any finite field $\mathbb{F}_q$, every subring of $\mathcal{M}_n(\mathbb{F}_q)$ is a finite ring, for example. As George Lowther says, for finite commutative rings we can also consider quotients $\mathbb{F}_q[x]/p(x)$ for arbitrary polynomials $p(x)$. These break up into products of rings of the form $\mathbb{F}_q[x]/r(x)^n$ for irreducible $r$, which is a form of the Chinese remainder theorem and is related to the structure theorem for finitely generated modules over a principal ideal domain (in this case, $\mathbb{F}_q[x]$). Is this "natural"?