Distance from Lat/Lng point to Minor Arc segment

For 100 - 1000m spherical problems, it is easy to just convert to cartesian space, using a equirectangular projection.
Then it continues with school mathematics:
Use the function "distance from line segment" which is easy to find ready implemented. This fucntion uses (and sometimes returns) a relative forward/backward position for the projected point X on the line A,B. The value is

• in the interval [0,1] if the projected point is inside the line segment.
• it is negative if X is outside before A,
• it is >1 if outside after B.
  If the relative position is between 0,1 the normal distance is taken, if outside the shorter distance of the both start and line-end points, A,B.

An example of such / or very similar an cartesian implementaion is Shortest distance between a point and a line segment

Adding a Java version to wdickerson answer:

public static double pointToLineDistance(double lon1, double lat1, double lon2, double lat2, double lon3, double lat3) {
    lat1 = Math.toRadians(lat1);
    lat2 = Math.toRadians(lat2);
    lat3 = Math.toRadians(lat3);
    lon1 = Math.toRadians(lon1);
    lon2 = Math.toRadians(lon2);
    lon3 = Math.toRadians(lon3);

    // Earth's radius in meters
    double R = 6371000;

    // Prerequisites for the formulas
    double bear12 = bear(lat1, lon1, lat2, lon2);
    double bear13 = bear(lat1, lon1, lat3, lon3);
    double dis13 = dis(lat1, lon1, lat3, lon3);

    // Is relative bearing obtuse?
    if (Math.abs(bear13 - bear12) > (Math.PI / 2))
        return dis13;

    // Find the cross-track distance.
    double dxt = Math.asin(Math.sin(dis13 / R) * Math.sin(bear13 - bear12)) * R;

    // Is p4 beyond the arc?
    double dis12 = dis(lat1, lon1, lat2, lon2);
    double dis14 = Math.acos(Math.cos(dis13 / R) / Math.cos(dxt / R)) * R;
    if (dis14 > dis12)
        return dis(lat2, lon2, lat3, lon3);
    return Math.abs(dxt);
}

private static double dis(double latA, double lonA, double latB, double lonB) {
    double R = 6371000;
    return Math.acos(Math.sin(latA) * Math.sin(latB) + Math.cos(latA) * Math.cos(latB) * Math.cos(lonB - lonA)) * R;
}

private static double bear(double latA, double lonA, double latB, double lonB) {
    // BEAR Finds the bearing from one lat / lon point to another.
    return Math.atan2(Math.sin(lonB - lonA) * Math.cos(latB), Math.cos(latA) * Math.sin(latB) - Math.sin(latA) * Math.cos(latB) * Math.cos(lonB - lonA));
}

First, some nomenclature:
Our arc is drawn from p1 to p2.
Our third point is p3.
The imaginary point that intersects the great circle is p4.
p1 is defined by lat1,lon1; p2 by lat2,lon2; etc.
dis12 is the distance from p1 to p2; etc.
bear12 is the bearing from p1 to p2; etc.
dxt is cross-track distance.
dxa is cross-arc distance, our goal!

Notice that the cross-track formula relies on the relative bearing, bear13-bear12

We have 3 cases to deal with.

Case 1: The relative bearing is obtuse. So, dxa=dis13.

Case 2.1: The relative bearing is acute, AND p4 falls on our arc. So, dxa=dxt.

Case 2.2: The relative bearing is acute,AND p4 falls beyond our arc. So, dxa=dis23

The algorithm:

Step 1: If relative bearing is obtuse, dxa=dis13
Done!
Step 2: If relative bearing is acute:
2.1: Find dxt.
2.3: Find dis12.
2.4: Find dis14.
2.4: If dis14>dis12, dxa=dis23.
Done!
2.5: If we reach here, dxa=abs(dxt)

MATLAB code:

function [ dxa ] = crossarc( lat1,lon1,lat2,lon2,lat3,lon3 )
%// CROSSARC Calculates the shortest distance in meters 
%// between an arc (defined by p1 and p2) and a third point, p3.
%// Input lat1,lon1,lat2,lon2,lat3,lon3 in degrees.
    lat1=deg2rad(lat1); lat2=deg2rad(lat2); lat3=deg2rad(lat3);
    lon1=deg2rad(lon1); lon2=deg2rad(lon2); lon3=deg2rad(lon3);

    R=6371000; %// Earth's radius in meters
    %// Prerequisites for the formulas
    bear12 = bear(lat1,lon1,lat2,lon2);
    bear13 = bear(lat1,lon1,lat3,lon3);
    dis13 = dis(lat1,lon1,lat3,lon3);

    diff = abs(bear13-bear12);
    if diff > pi
        diff = 2 * pi - diff;
    end
    %// Is relative bearing obtuse?
    if diff>(pi/2)
        dxa=dis13;
    else
        %// Find the cross-track distance.
        dxt = asin( sin(dis13/R)* sin(bear13 - bear12) ) * R;

        %// Is p4 beyond the arc?
        dis12 = dis(lat1,lon1,lat2,lon2);
        dis14 = acos( cos(dis13/R) / cos(dxt/R) ) * R;
        if dis14>dis12
            dxa=dis(lat2,lon2,lat3,lon3);
        else
            dxa=abs(dxt);
        end   
    end
end

function [ d ] = dis( latA, lonA, latB, lonB )
%DIS Finds the distance between two lat/lon points.
R=6371000;
d = acos( sin(latA)*sin(latB) + cos(latA)*cos(latB)*cos(lonB-lonA) ) * R;
end

function [ b ] = bear( latA,lonA,latB,lonB )
%BEAR Finds the bearing from one lat/lon point to another.
b=atan2( sin(lonB-lonA)*cos(latB) , ...
    cos(latA)*sin(latB) - sin(latA)*cos(latB)*cos(lonB-lonA) );
end

Sample outputs: Demonstrate all cases. See maps below.

>> crossarc(-10.1,-55.5,-15.2,-45.1,-10.5,-62.5)
ans =
   7.6709e+05
>> crossarc(40.5,60.5,50.5,80.5,51,69)
ans =
   4.7961e+05
>> crossarc(21.72,35.61,23.65,40.7,25,42)
ans =
   1.9971e+05

Those same outputs on the map!:

Demonstrates case 1:

Demonstrates case 2.1:

Demonstrates case 2.2:

Credit to: http://www.movable-type.co.uk/scripts/latlong.html
for the formulas
and: http://www.darrinward.com/lat-long/?id=1788764
for generating the map images.

And adding a python translation of Sga's implementation:

    def bear(latA, lonA, latB, lonB):
        # BEAR Finds the bearing from one lat / lon point to another.
        return math.atan2(
            math.sin(lonB - lonA) * math.cos(latB),
            math.cos(latA) * math.sin(latB) - math.sin(latA) * math.cos(latB) * math.cos(lonB - lonA)
        )


    def pointToLineDistance(lon1, lat1, lon2, lat2, lon3, lat3):
        lat1 = math.radians(lat1)
        lat2 = math.radians(lat2)
        lat3 = math.radians(lat3)
        lon1 = math.radians(lon1)
        lon2 = math.radians(lon2)
        lon3 = math.radians(lon3)
        R = 6378137

        bear12 = bear(lat1, lon1, lat2, lon2)
        bear13 = bear(lat1, lon1, lat3, lon3)
        dis13 = distance( (lat1, lon1), (lat3, lon3)).meters

        # Is relative bearing obtuse?
        if math.fabs(bear13 - bear12) > (math.pi / 2):
            return dis13

        # Find the cross-track distance.
        dxt = math.asin(math.sin(dis13 / R) * math.sin(bear13 - bear12)) * R

        # Is p4 beyond the arc?
        dis12 = distance((lat1, lon1), (lat2, lon2)).meters
        dis14 = math.acos(math.cos(dis13 / R) / math.cos(dxt / R)) * R
        if dis14 > dis12:
            return distance((lat2, lon2), (lat3, lon3)).meters
        return math.fabs(dxt)