I'm currently migrating some server software from Java 8 to Java 11. Everything works fine except for the fact that TLS connections are refused when my server runs on Java 11 (and only 11). I.e., everything works when running on up to and including Java 10. As soon as I switch to Java 11 (with absolutely no code changes inbetween, of course) I get a TLS decrypt error. My application works fine otherwise when I disable SSL/TLS or when I use IE/Edge to connect to my server (via TLS). When using Chrome 70 or Firefox 62 I get ERR_SSL_PROTOCOL_ERROR or SEC_ERROR_BAD_SIGNATURE, respectively.
I tried to analyze this using WireShark (see my dump). My conclusion is that Java 11 and Chrome/Firefox negotiate a bad cipher suite (rsa_pss_rsae_sha256), which somehow doesn't work with Java 11. Java 8 and IE/Edge seem to negotiate a different cipher suite (rsa_pkcs1_sha256), which works.
Does someone know how to fix this problem or at least work around it for the moment (other than disabling SSL/TLS or using IE/Edge, of course)? Thanks in advance.
This is likely to be a compatibility failure with the new PSS algorithms in TLS 1.3. You can read a good explanation for the reasoning behind the introduction of PSS here.
Until it's fixed you can prevent your server from negotiating TLS 1.3 by editing lib/security/java.security and adding TLSv1.3 to the jdk.tls.disabledAlgorithms property.