Varying the action with respect to the metric in one dimension
Here are the key points, which I'll express for an arbitrary number of spacetime dimensions because that makes the pattern more clear:
Inside $\sqrt{g}$, the thing denoted $g$ is the magnitude of the determinant of the metric tensor $g_{ab}$.
$g^{ab}$ are the components of the inverse of the metric tensor, defined by the condition $\sum_b g^{ab}g_{bc}=\delta^a_c$.
Specialized to one-dimensional spacetime, that general pattern reduces to this: $g^{tt}$ is the inverse of the quantity $g$ inside the square root. To make this explicit, we can write the metric tensor as $g$ (because it only has one component), and then $g^{tt}\equiv g^{-1} = 1/g$.
Now that we've deciphered the notation, we can do the calculation. To reduce clutter, I'll write the action as $$ I \propto \int dt\ g^{1/2} (g^{-1} K - m^2) $$ with $K\equiv \sum_n (dX_n/dt)^2$. This gives \begin{align} \frac{\delta I}{\delta g} &\propto (g^{-1} K - m^2)\frac{\delta }{\delta g}g^{1/2} + g^{1/2}\frac{\delta}{\delta g} (g^{-1}K-m^2) \\ &= (g^{-1} K - m^2)\frac{g^{-1/2}}{2} - g^{1/2}g^{-2}K \\ &= -\frac{g^{-1/2}}{2}(g^{-1}K+m^2), \end{align} so setting $\delta I/\delta g=0$ gives the desired equation of motion.
In the 1D world-line, the square root $$\sqrt{g_{tt}}~=~e~>~0$$ of the metric $g_{tt}$ is an einbein. And the inverse metric is $$g^{tt}~=~\frac{1}{e^2}.$$
Therefore Witten's Lagrangian (1) is $$L~=~\frac{\dot{x}^2}{2e}-\frac{e m^2}{2}.\tag{1'}$$ Variation wrt. the einbein $e$ leads to $$\frac{\dot{x}^2}{e^2}+ m^2~\approx~0,\tag{2'}$$ which is Witten's eq. (2).
For more information, see e.g. this related Phys.SE post.