Periodic lightray paths trapped between two nested mirror circles

I don't know how to answer your question, but I'll make a reformulation of the question.

Consider the unit tangent bundle to the outer circle, and consider the subset consisting of vectors pointing into the circle. This is an annulus, parameterized by $(\theta,\varphi)\in A=[0,2\pi]\times [0,\pi]/\{ (0,\varphi)\sim (2\pi,\varphi)\}$, where $\theta$ parameterizes the point on the outer circle, and $\varphi$ gives the angle that the unit vector makes with the tangent vector. One may consider the first-return map under the geodesic flow $F: A\to A$ (flow in the direction of the vector until you hit the outer circle again, then reflect). This is a piecewise smooth map. For each $\theta$, there are angles $0< f_1(\theta) < f_2(\theta) < \pi$ such that $F(\theta,\varphi)= (\theta+2\varphi, \varphi)$ for $0\leq \varphi \leq f_1(\theta)$ and $f_2(\theta)\leq \varphi \leq \pi$ (in particular, $F$ is the identity on the boundary of $A$). For fixed $\theta$ and $f_1(\theta) < \varphi < f_2(\theta)$, $F(\theta,\varphi)$ is a more complicated trigonometric function depending on how the geodesic reflects off the inner circle and bounces back to the outer circle, but it has the property that $\varphi$ is increasing, and $\theta$ is decreasing. There is a natural measure on geodesic flow in the plane, the Liouville measure, which restricts to a measure on $A$. Clearly $F$ preserves this measure. I haven't computed the measure, but it is invariant under rotation, so is independent of $\theta$, and is invariant under reflection $(\theta,\varphi)\to (\theta,\pi-\varphi)$. One could reparameterize the $\varphi$ coordinate in terms of the Liouville measure to get an area-preserving homeomorphism of the annulus. So I would suggest you could do a literature search for results on periodic points for area-preserving homeomorphisms of an annulus.


Following up on @Ian's comments: the magic words are "monotone twist maps of the annulus" and "Aubry-Mather theory". googling either of the above (or looking in Katok-Hasselblad, who have a whole chapter on the subject) is your best bet.