Why isn't $D_\infty$ the set of symmetries of a circle?

What you think of when taking $n$ to $\infty$ is this:

That is, you've got polygons inscribed in a common circle. Note however that even in that model, the limit to infinity would not get the full circle, because only rational multiples of $2\pi$ will appear.

But the limit that leads to $D_\infty$ is more like this:

And if you continue that to infinity, it's easy to see that the the sides will approach a straight line with equidistant points, and a rotation will end up as a translation on that line.

Although polygons "approach" the circle, the circle has quite a few more points than just the union of the vertices of all the $n$-gons inscribing it, and consequently it "should" have more symmetries. The symmetry group (rotations and reflections) of the circle is known as $\text{O}(2, \mathbb{R})$ or sometimes just $\text{O}(2)$, which can be seen as the set of all orthogonal ($A^t = A^{-1}$) 2-by-2 matrices with real entries, under multiplication.

Instead, you could think of $D_\infty$ as being the "limit" of the presentations for the finite dihedral groups, $$D_n = \langle r, s \mid r^n = s^2 = 1,\, srs = r^{-1} \rangle.$$ As $n$ goes to infinity, the statement that $r^n = 1$ simply drops out, because what "$r^n = 1$" is really saying is "$\text{ord}(r) = n$", and we write $\text{ord}(r) = \infty$ to mean $r$ has infinite order. So $$D_\infty = \langle r, s \mid s^2 = 1,\, srs = r^{-1}\rangle$$ which some authors take as the definition of $D_\infty$. Others define the infinite dihedral group as the isometries of $\mathbb{Z}$ as a subset of $\mathbb{R}$, in which case $x \mapsto x + 1$ can be thought of as $r$ (a "rotation") and $x \mapsto -x$ can be thought of as $s$ (a "reflection").

$D_\infty$ is the group of symmetries of an $\infty$-agon, of which a simple model is the set $\mathbb Z$ inside of $\mathbb R$.