How to find the distance along a sphere from an angle?
I interpreted the problem as asking for a function that, for general $R$ and $h$, gives the distance between the two points for any reflection angle (i.e., not only for the case in which the beams are tangent), and allows to calculate the maximal reflection angle and the maximal distance. This function can then be used to calculate the maximal angle and the maximal distance in the specific scenario provided in the OP, with $R=6371$ Km and $h=500$ Km.
Let us consider the Earth circumference as represented by the circle $x^2+y^2=R^2$, with the center in the origin and where $R=6371$. We can place our object in $A(0,R+h)$ on the $y$-axis, representing a point that is $h$ Km up the Earth surface.
Now let us draw two lines passing through $A$, symmetric with respect to the $y$-axis and intersecting the circumference. For each line, let us consider the intersection point that is nearer to the $y$-axis. Let us call the two new points $B$ (in the first quadrant) and $C$ (in the second quadrant). These represent the two points on Earth surface.
Due to the symmetry of the construction, we can continue by analyzing only one of these two points, e.g. $C$. The equation of the line containing $AC$ can be written as $y=sx+R+h$, where $s$ is its positive slope. To determine where this line crosses the circumference, we can set
$$sx+R+h=\sqrt{R^2-x^2}$$
whose solutions are
$$x=\frac{-sR-sh \pm \sqrt{\Delta}}{s^2 + 1}$$
where $\Delta=s^2 R^2 - 2h R - h^2$.
As stated above, we are interested in the less negative solution for $x$, as it is that nearer to the $y$-axis. So we get that the $x$-coordinate of $C$ is
$$X_C=\frac{-sR-sh + \sqrt{\Delta}}{s^2 + 1}$$
and the $y$-coordinate is
$$Y_C=\frac{s(-sR-sh + \sqrt{\Delta})}{s^2 + 1}+R+h$$
As a result, the equation $y=tx$ of the line $OC$ has slope
$$t=\frac{Y_C}{X_C}x=s-\frac{(s^2+1)(R+h)}{sh+sR-\sqrt{\Delta}}$$
Now setting $\angle{BAC}=\alpha$ and $\angle{BOC}=\beta$, we have $s=\cot(\alpha/2)$ and $t=-\cot(\beta/2)$.
Thus, we get
$$ \cot(\beta/2) =\frac{(\cot^2(\alpha/2)+1)(R+h)}{\cot(\alpha/2)(R+h)-\sqrt{\Delta}} - \cot(\alpha/2)$$
and then
$$ \beta =2\cot^{-1}\left[\frac{(\cot^2(\alpha/2)+1)(R+h)}{\cot(\alpha/2)(R+h)-\sqrt{\Delta}} - \cot(\alpha/2)\right] $$
So the length of the arc $D$ corresponding to $ \beta$, which is the distance along the spherical surface asked in the OP, is
$$ D =2R\cot^{-1}\left[\frac{(\cot^2(\alpha/2)+1)(R+h)}{\cot(\alpha/2)(R+h)-\sqrt{\Delta}} - \cot(\alpha/2)\right] $$
where $\Delta=\cot^2(\alpha/2)R^2 - 2h R - h^2$.
The last equation can be simplified as
$$ D =2R\cot^{-1}\left[\frac{(R+h)+\sqrt{\Delta}}{\cot(\alpha/2)(R+h)-\sqrt{\Delta}} \right] $$
For example, for $\alpha=\pi/2$ and $h= (\sqrt{2}-1)R$, as expected we have $\Delta=0$ (this is the situation where $\alpha$ is a right angle and the light beams are tangent to the surface). In this case, $\beta$ is also a right angle and $D=\pi/2\,R$. Accordingly the formula above gives this result, as shown by WA here.
For any value of $R$ and $h$, the maximal angle $\alpha_{max}$ and the maximal distance $D_{max}$ (i.e., those obtained with the beams tangent to the surface) can be determined by considering the case in which $\Delta=0$. This case occurs when $\cot^2(\alpha/2)R^2 - 2h R - h^2=0$. Solving for $\alpha$ in the range $0 \leq \alpha \leq \pi$ we get
$$\alpha_{max} = 2 \cot^{-1}\left(\frac{\sqrt{h(h+2R)}}{R}\right)$$
Interestingly, when $\Delta=0$, the formula for the distance is considerably simplified, and by few calculations reduces to
$$ D_{max} =2R\cot^{-1}\left[\tan(\alpha/2)\right] $$
As shown here, in the specific scenario described in the OP, substituting $R=6371$ and $h=500$, we get
$$\alpha= 2 \cot^{-1}\left(\frac{10 \sqrt{66210}}{6371}\right) \approx 2.3739 \,\,\text{radians}$$
which corresponds to about $136$ degrees. Here is the plot of the distance $D$ (in Km) as a function of $\alpha$ (in radians) for $R=6371$ and $h=500$, as obtained by WA. The plot confirms the maximal real value of $\alpha$, concordant with the predicted value of $2.3739$. The blue and red lines indicate the real and imaginary part, respectively.
Lastly, from the simplified formula for the maximal distance, taking $R=6371$ and $\alpha=2.3739$, we get
$$D_{max}\approx 4891 \, \text{Km}$$
We could make a triangle with the center of the sphere, one of the observer and the mirror. From this triangle, we know
the angle at the observer is a right angle since the radius is perpendicular to the tangent.
the side from the center to the observer mesure $6371$ km (radius).
the side from the center to the mirror mesure $6871$ km (radius $+$ height of the mirror).
We are able to find the angle at the center of the sphere. $$\cos \theta=\frac{6371}{6871}$$ $$\theta=0.383848\ \text{rad}$$ The same triangle could be built with the second observer. So the angular distance between the two observers is $$2\theta=0.767696\ \text{rad}$$ We multiply this value with the radius to get the distance between the two observers. $$\text{distance}=2\theta\times6371=4891\ \text{km}$$
More generaly, if an observer is on a sphere of radius $r$ uses a mirror placed at a height $h$ above the surface. The further distance that he could reach is given by $$\text{distance}=2\times r\times\arccos\left(\frac{r}{r+h}\right)$$
I understood the problem as follows (see the picture).
We have $OA=OA'=R$ and $OM=R+h$, where $R=6371$ km is the radius of Earth and $H=500$ km its the height of the reflector over the Earth’s surface. The point $P$ on the tangent is chosen to provide $AP||OM$. Given the reflection angle $\angle PMA=\alpha$, find the distance $d$ between the points $A$ and $A’$ along the sphere. But $d=2R\angle AOM=2R\beta$. We have $PM=AM\cos\alpha=OA\sin\beta=R\sin\beta$. By theorem of cosinuses
$$AM^2=OA^2+OM^2-2OA\cdot OM\cos\beta=R^2+(R+H)^2-2R(R+H) \cos\beta.$$
Thus
$$(R^2+(R+H)^2-2R(R+H) \cos\beta) \cos^2\alpha= R^2\sin^2\beta.$$
Putting $h=H/R$, we have
$$(1+(1+h)^2-2(1+h) \cos\beta)\cos^2\alpha=\sin^2\beta=1-\cos^2\beta.$$
This is a quadratic equation for $\cos\beta$, whose solution gives
$$\cos\beta=1+h-\sqrt{(h^2+2h+1)\cos^4\alpha-(h^2+2h+2)\cos^2\alpha+1}.$$