Why is proving $C^{\infty}$ regularity of sub Riemannian geodesics so hard?

The reason the problem is hard is that we do not have a good handle on what abnormal (=singular) geodesics can look like. See the chapter of my book that describes abnormal geodesics. Progress is being made in the Carnot case, but it is slow.

I would look at a fairly recent article (EXTREMAL CURVES IN NILPOTENT LIE GROUPS ENRICO LE DONNE, GIAN PAOLO LEONARDI, ROBERTO MONTI, AND DAVIDE VITTONE; I think in GAFA) on the Carnot case for state-of-the art stuff, and the first paper by myself and that of Liu-Sussmann just to get a feel of abnormal geodesics.

It is POSSIBLE the problem can be solved without understanding abnormals - by some direct analysis. No one has done anything like this: the abnormals always seem to be there, hiding.

According to the MathSciNet, the first general contribution to the solution of the problem is given in the following paper:

E. Hakavuori, E. Le Donne, Non-minimality of corners in subriemannian geometry. Invent. Math. 206 (2016), no. 3, 693–704. (MathSciNet review).

The paper mentioned by Richard Montgomery is:

E. Le Donne, G. P. Leonardi, R. Monti, D. Vittone, Extremal curves in nilpotent Lie groups. Geom. Funct. Anal. 23 (2013), no. 4, 1371–1401. (MathSciNet review).