Smallest possible subgroup of $\,D_6\,$ containing two elements of $\,D_6\,$

You should be more precise in specifying your problem. I'll assume that $D_6$ refers to the symmetry group of a regular hexagon. (There are different conventions as to how to index the dihedral groups.) I assume furthermore that $r$ refers to a rotation of $60^{\circ}$, say counterclockwise for definiteness, and that $s$ refers to a reflection symmetry, say for definiteness a reflection across the horizontal axis that runs through two vertices of the hexagon.

The key relation is that $rs = sr^{-1}$. Another way of writing is this that $srs^{-1} = r^{-1}$; in fact, this can be generalized to the statement that conjugating any counterclockwise rotation by a reflection yields the clockwise rotation by the same angle as the original counterclockwise rotation.

For $H$ to be a subgroup, it must be closed under products and inverses. Now $r^2 s$ and $s r^2$ are both reflections (why? there's an easier answer in terms of linear algebra). So squaring either of these elements will produce the identity and they are their own inverses, so we haven't produced any new elements yet. But we can multiply these two elements together: $(r^2 s)(s r^2) = r^4$ and $(s r^2)(r^2 s) = s r^4 s = r^2$. Furthermore, it is easy to see that $(sr^2)r^4 = s$ is also in $H$. Thus, $$H \supset \{ e, s, r^2, r^2 s, r^4, r^4 s = s r^2 \}.$$ But the set on the right hand side is already a subgroup; you could either argue that elements of the form $r^i s^j$ where $i$ is an even integer are closed under taking products and inverses. Or you could recognize these six elements as the symmetries of the inscribed equilateral triangle whose vertices are every other vertex of the hexagon.


Use what you know about the elements of the Dihedral group of order 12(the symmetries of a regular hexagon) which is denoted here as $D_6$ and work to obtain a subgroup generated by the elements you're given $r^2s, sr^2$. (The linked pages denotes r by R, and s by r.)

But for your elements, $r, s$, we take $r$ to be a rotation of $60^\circ$ (counterclockwise, perhaps, just be consistent) where $r$ has order $6$, and $s$ refers to a reflection, e.g., the reflection across the horizontal axis that runs through two opposing vertices of the hexagon, with $s$ of order $2$. See this link for more information on $D_6.$

enter image description here

More to the point, you need to obtain "closure" of the group generated by $r^2s$ and $sr^2$ under the group operation of $G = D_6$. We want to be sure to include all possible (unique) products in $H$, and ensure each element's inverse is included in $H$.

As you probably noted, the squares of each of your elements $r^2s$ and $sr^2$ is the identity $e$, hence of the given elements is of order 2. This means that each of these elements must is its own inverse. But we need to obtain closure in $H$: what elements do you get when you compute $(r^2s)(sr^2)=r^2s^2r^2$?: you get $r^4\in H$ since $s^2 = e$. And what do you get when you compute $(sr^2)(r^2s) = sr^4s$? You get: $r^2 \in H$. (Why?). Note that $r^4r^2 = e$, since $r$ is of order 6. But we also need to look at $(sr^2)(r^4) = sr^6 = s \in H.$ Finally, we compute $(r^4)s = sr^2$, which we already know is in $H$.

Thus, we have $\bf{H = \{e, s, r^2, r^4, r^2s, sr^2\}},$ which is closed under taking inverses and closed under taking products, and $e\in H$. So $H$ is indeed a subgroup, and in particular, $H \leq G = D_6$. It cannot be any smaller without failing then to be a subgroup. Indeed, it's the smallest subgroup of $G$ containing the given elements $r^2s, sr^2$!


Let me add a couple of comments. I assume the notation @MichaelJoyce has spelled out.

Both $r^2 s$ and $s r^2$ are involutions (elements of order 2), so by Lagrange's Theorem the order of $H$ must be divisible by 2. This is a helpful piece of information, signalling that your first guess had to be corrected.

Also, it is not difficult to prove that any group generated by two involutions $a$ and $b$ is dihedral, of order $2 n$, where $n$ is the order of $a b$ (or infinite, if $a b$ has infinite order).

In your case $r^2 s \cdot s r^2 = r^4$ has order 3, so you get a dihedral group of order 6 as shown in the other answers.