Java 11 was released with TLSv1.3 support, used by default.
It works OK in context of HTTPS and SSL sockets, but it seems that when using SSLEngine there are additional hurdles due to changes in TLSv1.3 behavior.
So there is a robust implementation of communication via NIO using SSLEngine that no longer works when TLSv1.3 is enabled. There are no obvious errors, in form of exceptions or SSL errors, two nodes will just send wrap/unwrap messages back and forth and eventually timeout.
I am interested in an exact list of behavior changes between SSLEngine using TLSv1.2 and SSLEngine using TLSv1.3, and if possible a migration checklist between these. Unfortunately, SSLEngine javadocs of Java 11 does not have this information - no new methods in Java 11 and no reference to TLSv1.3.
In the end we needed to read the remaining data from buffer after handshake is finished, unwrap it and update handshake status. Looks like an edge case which we did not handle previously.
Relevant commit: IGNITE-11298 Fixes to support TLSv1.3 in Communication