Existence of curve of constant curvature connecting two points

First, the quoted statement only asserts the exists of a curve with geodesic curvature $k_g\equiv c$ for some $c\in\mathbb{R}$. This formulation does not exclude the possibility that for some different $c'$ there is no such curve.

However, your question is still interesting.

Lets first look at $\mathbb{R}^2$, then curves of constant geodesic curvature are arcs of circles, but two points $p$ and $q$ only lie on common circles of diameter $\ge \vert p - q\vert$. This gives an upper bound on the geodesic curvature.

Also on a compact manifold there seems to be an obstruction: Suppose $p,q\in M$ and $\gamma$ is a geodesic joining $p$ and $q$. If there is a curve $\alpha$ of constant geodesic curvature $\equiv c$ which does not intersect $\gamma$, then $\gamma$ and $\alpha$ bound a domain $\Omega\subset M$ and by the Gauss-Bonnet theorem (see p. 277 in DoCarmo's Curves and Surfaces) we have $$ c \cdot \mathrm{Length}(\alpha)+ \int_\Omega K + \theta_1 + \theta_2 = 2 \pi \chi(\Omega), $$ where $K$ the Gauss-curvature and $\theta_i$ are the external angles at $p$ and $q$. Since all quantities can be bounded in terms of the geometry of $M$, we obtain a bound $$ -\lambda\le c \le -\lambda, $$ for $\lambda$ only depending on $(M,g)$. In particular, not every $c$ can be achieved as geodesic curvature. A little caveat is that, we had to assume that $\alpha$ does not intersect a geodesic between $p$ and $q$, but I doubt (without good argument at the moment), that this will change the the picture very much.

An interesting follow up question would be to try and say something about the set $$ C_{p,q}=\{c:\exists \alpha \text{ smooth curve joining } p,q \text{ with geodesic curvature }\equiv c \}\subset\mathbb{R} $$ If $M$ is complete, then $0\in C_{p,q}$ and I would assume that it also contains a zero neighbourhood.

Also see here, but it looks like the author only consider closed curves.

As an elaboration on Jan Bohr's answer, there is in fact a local version: With some caveats, a geodesic can be perturbed to a family of curves with constant, nonzero geodesic curvature and the same endpoints.

First, consider the torsionless, constant curvature Frenet-Serret equation, of which the geodesic equation is a special case: $$\begin{align} \text{Differentail Equation}:&\ \ \ \nabla_{\dot{\gamma}}\dot{\gamma}=N,\ \ \ \nabla_{\dot{\gamma}}N=-\frac{\langle N,N\rangle}{\langle\dot{\gamma},\dot{\gamma}\rangle}\dot{\gamma}, \\ \text{Initial Conditions}:&\ \ \ \gamma(0)=p,\ \ \ \dot{\gamma}(0)=u,\ \ \ N(0)=n \\ \text{Constraint}:&\ \ \ \langle u,n\rangle=0 \end{align}$$

If we begin with a minimizing unit speed geodesic $\gamma_0$ of length $L$ which terminates at $q$ with initial velocity $u_0$, we know this is a solution to this ODE with $n=0$. We wish to construct a nonzero variation of $n$ which preserves the endpoint.

To this end, define $\Gamma(n,u)=\gamma_{n,u}(L)$, where $\gamma_{u,n}$ is the solution of the initial value problem for fixed $n,u$. We know that $\Gamma(0,u_0)=q$, and it follows from the ODE theorem on smooth dependence of initial conditions that $\Gamma$ is smooth at $(0,u_0)$. Furthermore, if we assume $p$ is not conjugate to $q$ along $\gamma_0$, we know that the differential $\frac{d\Gamma}{du}(0,u_0)$ is full rank. Therefore, by implicit function theorem, there is a function $F(n)$ defined on a neighborhood of $0$ satisfying $F(0)=u_0$ and $\Gamma(n,F(n))=q$. This corresponds precisely with a family of curves of constant scalar curvature $\kappa=\|F(n)\|^2\|n\|$.

The conjugacy assumption is necessary here: Without it the geodesic connecting antipodal points on the $2$-sphere would be a counterexample, as no such variation exists there. This variation isn't the only possibility, though; in dimension $\ge 3$ one can also add torsion.