Name for the fact that a mattress can't be evenly rotated by repeatedly applying the same transformation?

The rotations of the mattress can be represented by an algebraic structure called a group. This group in particular is called the Klein four-group and is isomorphic to the direct product of two copies of $\mathbb Z / 2\mathbb Z$. (One of those copies is the exchange (T,B), and the other is (N,S)). The group has the property that every operation (except for the trivial "do nothing" operation) has order $2$, i.e., any operation performed twice will bring the mattress to its original state.

If all the elements of a group can be attained by repeatedly applying one operation $g$, the group is said to be cyclic and can be generated by $g$. In the case of the mattress, since no operation has order 4, the group is not cyclic. It is generated by the two flips (T,B) and (N,S), but not by a single flip.

The result you are wondering about with the more general case of the direct product of $\mathbb Z / m\mathbb Z$ and $\mathbb Z / n\mathbb Z$ (i.e., "cycling through substates $m$ and $n$") is described in the Wikipedia article on the direct product of groups. In short, $\mathbb Z / m\mathbb Z \times \mathbb Z / n\mathbb Z$ is cyclic and generated by $(1,1)$ if and only if $\gcd(m,n) = 1$.

---

(thank you to @pjs36 for pointing out the missing critical terms!)