I'm working on a crawler which needs to make HTTPS requests to various servers. Because of that, I'd like my crawler to support all possible protocols/ciphers.
I'm currently using Java 8 but willing to upgrade (don't think it matters).
I created the SSLContext as follows:
SSLContext sslContext = SSLContext.getInstance("TLSv1.2");
I guess that every server that supports TLSv1.3 supports 1.2 as well.
Is this going to be backward compatible? i.e. support TLS 1.1/1.0 and SSL?
(Is SSL even being used nowadays btw?)
What else do I need to do in order to promise a successful SSL/TLS handshake?
Thanks!
Because of that, I'd like my crawler to support all possible protocols/ciphers.
I'm currently using Java 8 but willing to upgrade (don't think it matters).I guess that every server that supports TLSv1.3 supports 1.2 as well.
Is this going to be backward compatible? i.e. support TLS 1.1/1.0 and SSL?
(Is SSL even being used nowadays btw?)
Yes, at least for the near future almost everyone who supports 1.3 will also support 1.2, with the exception of a few sites set up specifically to test 1.3 compatibility. (I'd bet many will even continue to support 1.1, e.g. PCI still allows it albeit grudgingly, but you don't need that; any Java that supports 1.1 also supports 1.2.) A TLSv1.2 context supports lower versions (if allowed, see below), and the default TLS context also supports 1.2 and lower in j8 (the default in j7 did not support 1.1 and 1.2 clientside).
No one should be using SSLv3 since 2016 at latest and Java releases since j8u31 in 2014-12 have been configured to prohibit it, although the code is still present if you change the configuration to permit it. See the entry for jdk.tls.disabledAlgorithms in JRE/lib/security/java.security for j8 or JAVA/conf/security/java.security for j9 up. Note this kind of 'disable' cannot be reversed by the code calling SSLSocket.setEnabledProtocols and related methods, only by changing the config or the security property at startup; in contrast protocols 'disabled' by choice of context can be enabled whenever desired, and conversely.
No one should have been using SSLv2 since the turn of the century (either 2000 or 2001, depending on how pedantic you are; at this remove the difference doesn't matter). Java doesn't (and never did) implement it anyhow.
Use of TLSv1.0 at this time is arguable. Theoretically it is vulnerable to BEAST, but that proved not to be much of a danger after all and is now widely further mitigated by 1/n splitting the application data. Since Java includes it by default, I wouldn't bother removing it, but you might be somewhat wary of any servers you find that actually select it.
You can't support all ciphersuites because Java doesn't implement them all; fortunately you don't need to. No one on the Internet uses PSK (except the variant for resumption in 1.3) SRP or Kerberos; Java implements only the last. Probably none or very few require Camellia SEED ARIA IDEA or AES-CCM, none of which Java implements (at least in the standard cryptoproviders), although you might encounter some that support at least some of these if offered. Nobody should support RC4, 'export' suites, 'eNULL', and single-DES; again Java releases have prohibited RC4 in java.security since 8u60 in 2015-08 and default-disabled the others for a long time, along with 'anonymous' suites (aka 'aNULL'). That last is the only one I would consider changing; I can imagine rare but reasonable cases to run a public server without certificate authentication, but otherwise secure.
But there are security parameters other than protocol and cipher that can prevent connection. In particular Java is now configured to reject handshakes using a key or parameters too short to be secure, or certificates signed using MD5 which is broken for that purpose, even though these are used in a ciphersuite that is acceptable to both endpoints. These settings are similarly in disabledAlgorithms with those for certs also in jdk.certpath.disabledAlgorithms and you may need to relax them. There is some risk that your connections using weakened parameters can be compromised, but since you presumably don't have any sensitive information to send to unknown servers and won't trust any of the information you receive from them anyway, such compromise may not be a problem.
Next and probably biggest is certificate validation. You will run into a lot of sites using certificates that are actually invalid, and many more that can't be validated against the standard Java truststore JRE/lib/security/cacerts or a similar one like Windows' or Mozilla's. Since you presumably don't care if you are actually getting data from a real or legitimate server, you can ignore the certificate errors by using a nondefault TrustManager that just returns success without actually validating the server's cert chain at all. If you search other Qs on SO you can find probably a hundred where someone proposes 'solving' a TLS or https problem in Java by 'just cut&paste this TrustManager to fix all problems without any thought, understanding or effort'. In many cases this is proposed, by people who don't actually understand TLS, for problems that aren't cert problems in the first place, so the 'solution' doesn't help and just makes the system totally insecure if and when the actual problems are fixed. I think yours is the first Q I've seen where this solution is actually appropriate.
Finally, there is a nominally optional feature in TLS, Server Name Indication (SNI), that many servers now require, especially CDNs, WAFs, IDS/IPSes and similar things that handle traffic for multiple domains. Depending on which API(s) and options you use to make your connections, older versions of Java did not always send SNI. However, recent j8 and up should do so unless you deliberately disable it -- and you might find some broken servers where you do need to disable it like this recent Q.