How can I format a day from a date as 2nd intead of "2" or 3rd instead of "3"?

You can utilize a method like this in your controller to append the suffix to the day portion of the date, but this approach won't work directly in VF using an <apex:param /> input into a formatter. You'll have to construct the date string in the controller.

Caveat: this code doesn't work as written for integer values greater than 100 - but there are few months with more than 100 days, so I figured it was safe enough as written.

public static String getDayOfMonthSuffix(Integer n) {
    if (n == null) {
        return '';
    }

    if (n >= 11 && n <= 13) {
        return 'th';
    }

    Integer modResult = Math.mod(n, 10);        
    if (modResult == 1) { 
        return 'st'; 
    } else if (modResult == 2) { 
        return 'nd'; 
    } else if (modResult == 3) { 
        return 'rd'; 
    } else { 
        return 'th';
    }
}

Update

Here's some example controller code and VF markup

VF

<apex:outputText value="{!showFormattedDate}" />

Apex

public with sharing YourController {

    public string showFormattedDate() {

        // get a date
        Date today = Date.today();

        // use some parameter substitution to build the string
        string formattedDate = String.format('{0} {1}, {2}', new List<String>{ today.month(), getDayOfMonthSuffix(today.day()), today.year() });

        return formattedDate;
    }

    public String getDayOfMonthSuffix(Integer n) {
        if (n == null) {
            return '';
        }

        if (n >= 11 && n <= 13) {
            return 'th';
        }

        Integer modResult = Math.mod(n, 10);        
        if (modResult == 1) { 
            return 'st'; 
        } else if (modResult == 2) { 
            return 'nd'; 
        } else if (modResult == 3) { 
            return 'rd'; 
        } else { 
            return 'th';
        }
    }
}

You can also add it as a formula on the object if you don't want to make/use a custom controller:

IF(ISBLANK(Date__c),NULL, 
IF(AND(DAY(Date__c) > 10, DAY(Date__c) < 14),"th", 
IF(MOD(DAY(Date__c),10) = 1,"st", 
IF(MOD(DAY(Date__c),10) = 2,"nd", 
IF(MOD(DAY(Date__c),10) = 3,"rd","th")))))