Convert to camelCase

Retina, 56 bytes

Byte count assumes ISO 8859-1 encoding.

T`'\`
S_`\W|_|(?<=[a-z])(?=[A-Z][a-z])
T`L`l
T`l`L`¶.
¶

Try it online!

Explanation

This implements the specification quite literally:

T`'\`

Remove apostrophes and backticks.

S_`\W|_|(?<=[a-z])(?=[A-Z][a-z])

Split the string around non-word characters (in regex this also excludes digits and underscores), or underscores or positions that have a lower case letter on the left and and upper case, lower case on the right. This would create some empty segments when there are two non-letter, non-digit characters in a row, or more important at the beginning of the string. We get rid of those with the _ option. Here, "splitting" means put each remaining part on its own line.

T`L`l

Convert everything to lower case.

T`l`L`¶.

Convert each character that occurs after the linefeed to upper case. This will conveniently skip the first word because there's no linefeed in front of it.

Get rid of the linefeeds to join everything back together.


Java, 198 190 bytes

+3 bytes because I forgot that \W+ == [^a-zA-Z0-9_]+ and I need to match [^a-zA-Z0-9]+

-11 bytes thanks to user20093 - ?: instead of if/else

Because, Java.

s->{String[]a=s.replaceAll("`|'","").split("[\\W_]+|(?<=[a-z])(?=[A-Z][a-z])");s="";for(String w:a){String t=w.toLowerCase();s+=a[0]==w?t:t.toUpperCase().charAt(0)+t.substring(1);}return s;}

This is a lambda. Call like so:

UnaryOperator<String> op = s->{String[]a=s.replaceAll("`|'","").split("[\\W_]+|(?<=[a-z])(?=[A-Z][a-z])");s="";for(String w:a){String t=w.toLowerCase();s+=a[0]==w?t:t.toUpperCase().charAt(0)+t.substring(1);}return s;};
System.out.println(op.apply("Programming Puzzles & Code Golf"));

Readable version:

public static String toCamelCase(String s) {
    String[] tokens = s
            .replaceAll("`|'", "") // 1. Remove all apostrophes
            .split("[\\W_]+|(?<=[a-z])(?=[A-Z][a-z])"); // 2. Split on [\W_]+ or between [a-z] and [A-Z][a-z]
    s = ""; // Reusing s for building output is cheap
    for (String token : tokens) {
        String lowercaseToken = token.toLowerCase(); // 3. Lowercase every word
        s += tokens[0].equals(token)?lowercaseToken:lowercaseToken.toUpperCase().charAt(0) + lowercaseToken.substring(1); // 4. Uppercase first char of all but first word
        // ^ 5. Join all words back together
    }
    return s;
}

JavaScript (ES6), 156 154 152 148 145 141 140 bytes

Thanks @Neil (6 bytes), @ETHproductions (3 bytes), and @edc65 (7 bytes)

a=>a[r='replace'](/`|'/g,a='')[r](/[a-z](?=[A-Z][a-z])/g,'$& ')[r](/[^\W_]+/g,b=>a+=(a?b[0].toUpperCase():'')+b.slice(!!a).toLowerCase())&&a

Removes apostrophes, then does a replace to split on special characters/before surrounded capitals, then combines with proper casing. Unfortunately, toLowerCase() and toUpperCase() are annoyingly long and hard to avoid here...