Constraints of relativistic point particle in Hamiltonian mechanics
Hints to the question (v1):
We cannot resist the temptation to generalize the background spacetime metric from the Minkowski metric $\eta_{\mu\nu}$ to a general curved spacetime metric $g_{\mu\nu}(x)$. We use the sign convention $(-,+,+,+)$.
Let us parametrize the point particle by an arbitrary world-line parameter $\tau$ (which does not have to be the proper time).
The Lagrange multiplier $\lambda=\lambda(\tau)$ (which OP mentions) depends on $\tau$, but it does not depend on the canonical variables $x^{\mu}$ and $p_{\mu}$. Similarly, $x^{\mu}$ and $p_{\mu}$ depend only on $\tau$.
The Lagrange multiplier $\lambda=\frac{e}{2}$ can be identified with an einbein$^1$ field $e$. See below where we outline a simple way to understand the appearance of the on-shell constraint $$\tag{1}p^2+m^2~\approx~0, \qquad p^2~:=~g^{\mu\nu}(x)~ p_{\mu}p_{\nu}~<~0.$$
Start with the following square root Lagrangian for a massive relativistic point particle $$\tag{2}L_0~:=~ -m\sqrt{-\dot{x}^2}, \qquad \dot{x}^2~:=~g_{\mu\nu}(x)~ \dot{x}^{\mu}\dot{x}^{\nu}~<~0, $$ where dot means differentiation wrt. the world-line parameter $\tau$. Here the action is $S_0=\int \! d\tau~ L_0 $. The stationary paths includes the geodesics. More precisely, the Euler-Lagrange equations are the geodesics equations.
Introduce an einbein field $e=e(\tau)$, and Lagrangian $$\tag{3}L~:=~\frac{\dot{x}^2}{2e}-\frac{e m^2}{2}.$$ Contrary to the square root Lagrangian (2), this Lagrangian (3) also makes sense for massless point particles, cf. this Phys.SE post.
Show that the Lagrangian momenta are $$\tag{4}p_{\mu}~=~ \frac{1}{e}g_{\mu\nu}(x)~\dot{x}^{\nu}.$$
Show that the Euler-Lagrange equations of the Lagrangian (3) are $$\tag{5} \dot{p}_{\lambda}~\approx~\frac{1}{2e}\partial_{\lambda}g_{\mu\nu}(x)~ \dot{x}^{\mu}\dot{x}^{\nu}, \qquad \dot{x}^2+(em)^2~\approx~0.$$
Show that the Lagrangian (3) reduces to the square root Lagrangian (2) when integrating out the einbein field $$\tag{6} e~>~0.$$ The inequality (6) is imposed to remove an unphysical negative branch, cf. my Phys.SE answer here.$^2$
Perform a (singular) Legendre transformation$^3$ of the Lagrangian (3), and show that the corresponding Hamiltonian becomes $$\tag{7}H~=~ \frac{e}{2}(p^2+m^2).$$ This Hamiltonian (7) is precisely of the form Lagrange multiplier times constraint (1).
Show that Hamilton's equations are precisely eqs. (4) and (5).
The arbitrariness in the choice of the world-line parameter $\tau$ leads to reparametrization symmetry$^4$ $$\tau^{\prime}~=~f(\tau), \qquad d\tau^{\prime} ~=~ d\tau\frac{df}{d\tau},\qquad \dot{x}^{\mu}~=~\dot{x}^{\prime\mu}\frac{df}{d\tau},\qquad e~=~e^{\prime}\frac{df}{d\tau},\qquad $$ $$\tag{8} p_{\mu}~=~p_{\mu}^{\prime},\qquad L~=~L^{\prime}\frac{df}{d\tau},\qquad H~=~H^{\prime}\frac{df}{d\tau}\qquad S~=~S^{\prime},$$ where $f=f(\tau)$ is a bijective function.
Thus one may choose various gauges, e.g. $e={\rm const.}$
References:
- J. Polchinski, String Theory, Vol. 1, Section 1.2.
--
Footnotes:
$^1$ An einbein is a 1D version of a vielbein.
$^2$ As a consistency check of the sign (6), if we in the static gauge $$\tag{9}ix^0_M~=~x^0_E~=~\tau_E~=~i\tau_M$$ Wick rotate from Minkowski to Euclidean space, then in eq. (3), the Euclidean Lagrangian $L_E=-L_M>0$ becomes positive as it should.
$^3$ Strictly speaking, in the singular Legendre transformation, one should also introduce a momentum $$\tag{10}p_e~:=~\frac{\partial L}{\partial \dot{e}}~=~0$$ for the einbein $e$, which leads to a primary constraint, that immediately kills the momentum $p_e$ again. Note that $\frac{\partial H}{\partial e}\approx 0$ becomes one of Hamilton's equations.
$^4$ Reparametrization is a passive transformation. For a related active transformation, see this Phys.SE post.