Lagrange multiplier and constraint force
Let there be given a (configuration) manifold $M$. Often in physics one assumes that a constraint function $\chi$ obeys the following regularity conditions:
$\chi: \Omega\subseteq M \to \mathbb{R}$ is defined in an open neighborhood $\Omega$ of the constrained submanifold $C\subset M$;
$\chi$ is (sufficiently$^1$ many times) differentiable in $\Omega$;
The gradient $\vec{\nabla} \chi$ is non-vanishing on the constrained submanifold $C\subset M$.
Here it is implicitly understood that $\chi$ vanishes on the constrained submanifold $C\subset M$, i.e.
$$C\cap \Omega ~=~\chi^{-1}(\{0\})~:=~\{x\in\Omega \mid \chi(x)=0\}.$$
[Also we imagine that the full constrained submanifold $C\subset M$ is covered by a family $(\Omega_{\alpha})_{\alpha\in I}$ of open neighborhoods, each with a corresponding constrained function $\chi_{\alpha}: \Omega_{\alpha}\subseteq M \to \mathbb{R}$, and such that the constraint functions $\chi_{\alpha}$ and $\chi_{\beta}$ are compatible in neighborhood overlaps $\Omega_{\alpha}\cap \Omega_{\beta}$.] Since there (locally) is only one constraint, the constrained submanifold will be a hypersurface, i.e. of codimension 1. [More generally, there could be more than one constraint: Then the above regularity conditions should be modified accordingly. See e.g. Ref. 1 for details.]
The above regularity conditions are strictly speaking not always necessary, but greatly simplify the general theory of constrained systems. E.g. in cases where one would like to use the inverse function theorem, the implicit function theorem, or reparametrize $\chi\to\chi^{\prime}$ the constraints. [The rank condition (3.) can be tied to the non-vanishing of the Jacobian $J$ in the inverse function theorem.]
Quantum mechanically, reparametrizations of constraints may induce a Faddeev-Popov-like determinantal factor in the path integral.
Example 1a: OP's 1st example (v1) $$\tag{1a} \chi(x,y)~=~x^2+y^2-\ell^2$$ would fail condition 3 if $\ell=0$. If $\ell=0$, then $C=\{(0,0)\}\subset M=\mathbb{R}^2$ is just the origin, which has codimension 2. On the other hand, the $\chi$-constraint satisfies the regularity conditions 1-3 if $\ell>0$.
Example 1b: OP's 1st example (v3) $$\tag{1b} \chi(x,y)~=~\sqrt{x^2+y^2}-\ell$$ is not differentiable at the origin $(x,y)=(0,0)$, and hence would fail condition 2 if $\ell=0$. On the other hand, the $\chi$-constraint satisfies the regularity conditions 1-3 if $\ell>0$.
Example 2a: Assume $\ell>0$. OP's 2nd example (v1) $$\tag{2a} \chi(x,y)~=~\sqrt{x^2+y^2-\ell^2}$$ would fail condition 1 and 2. The square root is not well-defined on one side of the constrained submanifold $C$.
Example 2b: Assume $\ell>0$. OP's 2nd example (v3) $$\tag{2b} \chi(x,y)~=~(x^2+y^2-\ell^2)^2$$ would fail condition 3 since the gradient $\vec{\nabla} \chi$ vanishes on the constrained submanifold $C$.
References:
- M. Henneaux and C. Teitelboim, Quantization of Gauge Systems, 1994; Subsection 1.1.2.
--
$^1$ Exactly how many times differentiable depends on application.