Rejecting mutual TLS gRPC connection based on RSA public key size

You can customize certificate checking by providing your own javax.net.ssl.TrustManagerFactory to Netty's SslContextBuilder. You'd probably want to implement X509ExtendedTrustManager, do your check, and then delegate to a "real" implementation for the rest of the cert chain checking.

You can do something like this to get the default-configuration TrustManagerFactory:

TrustManagerFactory tmf = TrustManagerFactory.getInstance(
    TrustManagerFactory.getDefaultAlgorithm());
tmf.init((KeyStore) null);
// loop through tmf.getTrustManagers() checking for one implementing X509TrustManager