What is the difference of TrustManager PKIX and SunX509?

From a basic usage point of view, difference is how resulting TrustManagers are initialised, as per Java Cryptography Architecture Oracle Providers Documentation for JDK 8

SunX509: A factory for X509ExtendedTrustManager instances that validate certificate chains according to the rules defined by the IETF PKIX working group in RFC 3280 or its successor. This TrustManagerFactory supports initialization using a Keystore object, but does not currently support initialization using the class javax.net.ssl.ManagerFactoryParameters.

PKIX: A factory for X509ExtendedTrustManager instances that validate certificate chains according to the rules defined by the IETF PKIX working group in RFC 3280 or its successor. This TrustManagerFactory currently supports initialization using a KeyStore object or javax.net.ssl.CertPathTrustManagerParameters.

One thing to note is, Java Cryptography Architecture Standard Algorithm Name Documentation for JDK 8, only lists PKIX as a TrustManagerFactory algorithm. SunX509 is left to provider's documentation because it is a vendor provided implementation, whereas PKIX is provided by all vendors. For example if you're running on IBM JRE, there is no SunX509, but IbmX509. Consecutively, if we hardcode "SunX509" in our application, we will receive a NoSuchAlgorithmException. Hence, for portability, it is best to use platform default algorithm as below, as both will work for keystone files (currently both Sun and IBM JREs default to PKIX).

TrustManagerFactory trustManagerFactory=
    TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());

While both factories can be initialised with a KeyStore parameter, using PKIX allows for alternatives, which can be configured using initialisation parameters. An interesting example is using LDAPCertStoreParameters for using an LDAP certificate store instead of a keystore file (an example here).

There is an issue in Oracle's bug tracking system that adds a little more clarity to this question

https://bugs.openjdk.java.net/browse/JDK-8169745

From the issue:

The SunX509 trust manager is implemented in SimpleValidator.java for compatibility use only, and no new features will be added. The PKIX trust manager is the default and recommended trust manager.

In the SunX509 validator/trust manager implementation, we used to check known critical extensions only. The supported extensions are white listed in sun/security/validator/EndEntityChecker.java. If an extension is critical and not present in the white list, the cert cannot pass the SunX509 validation. The PKIX validator/trust manager supports more rich extensions and features.

In the Oracle Providers documentation, it currently says:

"SunX509: A factory for X509ExtendedTrustManager instances that validate certificate chains according to the rules defined by the IETF PKIX working group in RFC 3280 or its successor."

This is misleading since it does not support all of the required extensions (and probably other requirements) of RFC 3280, and it is not strictly compliant with RFC 3280 and may not support all required extensions. We can also discourage its use. And we should update the RFC 3280 references to 5280 throughout this document.