Wald's General Relativity, section 6.3 Page 144

Let's start with the Schwarzschild geodesic $$ \frac{1}{2}\dot{r}^2 + V(r) = \frac{1}{2}E^2, $$ with $V(r)$ the effective potential $$ V(r) = \frac{L^2}{2r^3}(r-2M). $$ Consider light rays that are deflected in the Schwarzschild geometry, i.e. photons that start at a large distance ($r$ large, $\dot{r}<0$), propagate until they reach a distance of closest approach at coordinate $r=R_0$ (where $\dot{r}=0$), and propagate outward again. Therefore, $R_0$ is a solution of the equation $$ \frac{1}{2}E^2 = V(r) = \frac{L^2}{2r^3}(r-2M), $$ or $$ r^3 -b^2(r-2M) =0, $$ with $b=L/E$. If $b<b_c = 3^{3/2}M$ then this equation has no positive roots, which means that the photon will continue to spiral towards the centre. If $b>b_c$, the equation has two positive roots and one unphysical negative root. We're only interested in the largest of the positive roots, since we started at a very large radius ($r \gg R_0$) and the photons will never reach any distance smaller than $R_0$.

The solution $$ R_0 = \frac{2b}{\sqrt{3}}\cos\left[\frac{1}{3}\cos^{-1}(-b_c/b)\right] $$ can be verified by using $4\cos^3x =\cos 3x + 3\cos x$. One can also verify that it is the largest root, since $b>b_c$ and $$ \cos^{-1}(-b_c/b) < \cos^{-1}(-1) = \pi\\ \cos\left[\frac{1}{3}\cos^{-1}(-b_c/b)\right] > \cos(\pi/3) = 1/2\\ R_0 > \frac{b}{\sqrt{3}} > \frac{b_c}{\sqrt{3}} = 3M, $$ and $3M$ is the coordinate where $V(r)$ has a maximum. The other root (let's call it $R_1$) will lie between $2M< R_1 < 3M$. This root applies to photons that start at very small radii $(2M < r < R_1)$ and try to escape, but cannot move beyond the turning point $R_1$, after which they spiral towards the centre.