Are small $\varepsilon$-balls convex in geodesic metric spaces?

The standard sub-Riemannian metric on the 3-dimensional real Heisenberg group $\mathbb{H}^3$ is a counterexample. Topologically, it is homemorphic to $\mathbb{R}^3$, so in particular it is locally contractible. It is geodesic, but no ball is convex.

Thanks to the homogeneity and dilation structure, it suffices to know that, say, the unit ball centered at the origin is non-convex. It looks sort of like an apple, as in this cutaway view:

But every horizontal line through the $z$-axis is a geodesic.

(Oh, I guess you did ask for a compact example. So take a compact subset with nonempty interior: for any point in the interior, all sufficiently small balls are non-convex.)

To understand this in more detail, let's identify $\mathbb{H}^3$ with $\mathbb{R}^3$ with standard coordinates $(x,y,z)$. An admissible path (or horizontal path) in $\mathbb{H}^3$ is a path $\gamma = (x, y, z)$ which satisfies $$z(t) - z(0) = \frac{1}{2} \int_0^t (x(s) y'(s) - y(s) x'(s))\,ds.\tag{*}$$ The length of such a path is defined to be simply the Euclidean length of the 2-dimensional projected path $(x,y)$. The distance between two points is the length of the shortest admissible path (which can be shown to exist).

So any $C^1$ path $(x,y)$ in $\mathbb{R}^2$, with $z(0)$ specified, has a unique lift to an admissible path in $\mathbb{H}^3$, by defining $z(t)$ by (*), whose length is the same as that of $(x,y)$ itself. By Green's theorem, this is saying that $z(T) - z(0)$ measures the signed area enclosed by the curve $(x,y)$ on $[0,T]$, where if it does not form a closed curve, we can add straight line segments from the origin $(0,0)$ to the curve's endpoints. (A straight line through the origin contributes zero to the integral in (*).)

For an example, let's consider the closed ball $B$ centered at $(0,0,0)$ with radius $\pi/2$. A semicircle in the plane with endpoints $(0,0)$ and $(1,0)$ encloses area $\pi/8$ (after tacking on the line segment from $(1,0)$ back to $(0,0)$), so this lifts to an admissible path connecting $(0,0,0)$ to $(1,0,\pi/8)$. The length of this curve is just $\pi/2$, so the point $p_1 = (1,0,\pi/8)$ is in $B$. Likewise, so is $p_2 = (-1,0,\pi/8)$.

The straight line segment joining $(1,0)$ to $(-1,0)$ lifts to an admissible path connecting $p_1$ to $p_2$ (the integral in (*) vanishes, and zero area is enclosed), and being a line it is the shortest possible path. It passes through the point $q = (0,0,\pi/8)$.

Any admissible path connecting $(0,0,0)$ to $(0,0,\pi/8)$ must project to a closed curve starting and ending at the origin, and enclosing area $\pi/8$. By the isoperimetric inequality, the shortest such curve is a circle, of radius $r = \sqrt{1/8}$. The length of this curve is $2 \pi r = \pi/\sqrt{2} > \pi/2$. So $q$ is not in $B$.

Intuitively, you can enclose area more cheaply if you're willing to end at a different $x$ or $y$ coordinate from your starting point, since you get the straight line back home for free.

Negative part

The answer is no without further assumptions, here is a counterexample (a bit nasty, it is not locally simply connected) :

• In the plane consider first the two positive semi axis. ($\mathbb{R}_+\times\{0\}$ and $\{0\}\times\mathbb{R}_+$).
• for each $n$ add the segment from $(0,2^{-n})$ to $(2^{-n},0)$.
• put on it the length metric induced by the usual metric on $\mathbb{R}^2$.

Then every ball centered at the origin of radius $1.01\times 2^{-n}$ is not convex : the points $(0,2^{-n})$ to $(2^{-n},0)$ are inside while the geodesic between these points is the segment between theme, which leaves the ball. (The $1.01$ is there to handle closed and open balls at the same time.)

Positive part

As you are probably aware, the answer is yes on a smooth riemannian manifold. I suspect the same thing is true on a smooth Finsler manifold. A $CAT(k)$ metric space would have this property since the distance to a point is then convex (on balls of radius $\tfrac{\pi}{2\sqrt{k}}$ if $k>0$).

This leaves a lot of room for improvement, for instance :

• Can we build simply connected examples ? (I would think yes)
• Can we build example with curvature bounded from below in the sense of Alexandrov ? (I would say no but I'm not sure why)

There is actually a much more striking result than non-convexity of balls in the Heisenberg group discussed in the answer of Nate Eldredge.

While the definition of the sub-Riemannian metric on $\mathbb{H}^3$ has been explained by Nate Eldredge, I will skip that part and just formulate the theorem.

Theorem. Let $A=\{(x,y,z_1),(x,y,z_2)\}\subset\mathbb{H}^3$, where $z_1\neq z_2$ be a set consisting of two points in the Heisenberg group. Then the smallest geodesically convex set containing $A$ is $\mathbb{H}^3$.

That means there are very few convex sets and in particular the smallest geodesically convex set containing a ball must be $\mathbb{H}^3$.

This is a remarakable result because, the Heisenberg group is one of the nicest non-Euclidean metric spaces that allows one to carry out most of the analysis in Euclidean spaces. It also serves as an ulitmate example for testing new conjectures in analysis on metric spaces.

The above result is Theorem 1.1 in:

Monti, R. Rickly, M.: Geodetically convex sets in the Heisenberg group. J. Convex Anal. 12 (2005), 187-196.

Remark: The space $\mathbb{H}^3$ considered above is often denoted by $\mathbb{H}^1$. Although, as a linear space it is $\mathbb{R}^3$, it is the first Heisenberg group. Moreover the sub-Riemannian metric discussed above is often called the Carnot-Caratheodory metric.