What are the differences of checking a self-signed certificate vs ignore it?

If it's an official service you are integrating with the provider should really have a valid, publicly signed certificate installed for the sake of security.

Assuming that you need to continue on with your provider using a self signed certificate, the big difference between ignoring the certificate and adding it as trusted is the following scenario. This ues DNS poisoning as an example of a man in the middle attack.

Take the following example:

• api.example.com has a self signed cert with thumbprint XXX listening on the IP 5.5.5.5.
• Adding it to your trust store makes your system expect the thumbprint to be XXX when connecting
• If someone was able to poison your DNS - make api.example.com resolve to 6.6.6.6 - the thumbprint would be YYY.
• By adding the cert to your store your product would refuse to connect to the malicious site.
• By disabling the check entirely, your product would happily connect to the malicious api.example.com at 6.6.6.6.

By importing a known good self-signed certificate where the private key is unique and not compromised, the connection is just as safe as a full global CA PKI signed certificate. Those are after all also simply stored and trusted at some point.

The reason to get a PKI CA signed cert is one of practicality more than security; if you have to trust a self-signed certificate on many clients, that can be a lot of work. And doing key rotation almost exponentially increases that work.

If you control the server and client, like during development, and you don’t have the skill or resources to setup a private CA, then adding a specific self-signed certificate to a trust store is not so bad. Accepting any certificate is bad, never do that.

If you ignore the certificate check, anyone who can gain the man-in-the-middle position (with the ability to modify traffic) between you and the other system can read/modify the plain-text traffic. An attacker will simply break the TLS/SSL tunnel by starting his own TLS server using his self-signed certificate, route your traffic to it, decrypt it, and proxy it to the real server. There are many tools which make this attack pretty easy, for instance, mitmproxy for HTTPS or stunnel in general for TLS/SSL.

An attack can get into the man-in-the-middle position in many different ways. Thus, it is tough to completely rule out, that there is no way for an attacker to gain this position even if you're a network security expert.

If there is no way to replace the self-signed certificate with a publicly signed certificate, you can trust self-signed the certificate manually by adding it to a Java keystore using keytool and trust this store. This is sometimes called Certificate or Public Key Pinning