SSL handshake exception: "Algorithm constraints check failed: MD5withRSA"

keyser gave direction for answer in comment.

Problem was in key's length. In short: "Starting from 7u40, the use of x.509 certificates with RSA keys less than 1024 bits in length is restricted."

So the right way to solve this problem it is using certificates with key's length at least 2048 bits.

The problem is caused by Oracle disabling hash algorithms which are no longer considered to be secure. Take a look at

JRE_HOME/lib/security/java.security

It contains the following properties:

jdk.certpath.disabledAlgorithms
jdk.tls.disabledAlgorithms

You can adjust them appropriately. For example, remove MD5 from the former and MD5withRSA from the latter.

Hint for docker images:

there is additional config file /etc/crypto-policies/back-ends/java.config in some docker images like keycloak in my case which overrides values in java.security