Zarathustra :
I am using the new java.net.http.HttpClient with the sendAsync method. The HttpClient is inside a Singelton and is created once like so: HttpClient.newBuilder().build() so really nothing special.
Those requests can be POST or GET but I don't know which causes the trouble.
There are just a few requests a day but from time to time one thread uses 100% of a cpu core. And not imminently but after some time when the request has finished.
So I did a thread dump when there were even 2 of those endless loops occurring, the following 2 threads stood out:
"HttpClient-4-Worker-5" #144 daemon prio=5 os_prio=0 cpu=511298.10ms elapsed=520.71s tid=0x00007f684403e800 nid=0x2d6b runnable [0x00007f68ac162000]
java.lang.Thread.State: RUNNABLE
at jdk.internal.net.http.common.SSLFlowDelegate$Writer.processData([email protected]/SSLFlowDelegate.java:771)
at jdk.internal.net.http.common.SSLFlowDelegate$Writer$WriterDownstreamPusher.run([email protected]/SSLFlowDelegate.java:645)
at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run([email protected]/SequentialScheduler.java:147)
at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run([email protected]/SequentialScheduler.java:198)
at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule([email protected]/SequentialScheduler.java:271)
at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule([email protected]/SequentialScheduler.java:224)
at jdk.internal.net.http.common.SSLFlowDelegate$Writer.triggerWrite([email protected]/SSLFlowDelegate.java:722)
at jdk.internal.net.http.common.SSLFlowDelegate.doHandshake([email protected]/SSLFlowDelegate.java:1024)
at jdk.internal.net.http.common.SSLFlowDelegate.doClosure([email protected]/SSLFlowDelegate.java:1094)
at jdk.internal.net.http.common.SSLFlowDelegate$Reader.unwrapBuffer([email protected]/SSLFlowDelegate.java:500)
at jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData([email protected]/SSLFlowDelegate.java:389)
- locked <0x00000000fba68950> (a java.lang.Object)
at jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run([email protected]/SSLFlowDelegate.java:263)
at jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run([email protected]/SequentialScheduler.java:175)
- locked <0x00000000fbbca3e8> (a java.lang.Object)
at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run([email protected]/SequentialScheduler.java:147)
at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run([email protected]/SequentialScheduler.java:198)
at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
at java.lang.Thread.run([email protected]/Thread.java:834)
Locked ownable synchronizers:
- <0x00000000fc1ff920> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"HttpClient-4-Worker-2" #82 daemon prio=5 os_prio=0 cpu=4266156.67ms elapsed=4311.42s tid=0x00007f6844007000 nid=0x29ee runnable [0x00007f686fffd000]
java.lang.Thread.State: RUNNABLE
at jdk.internal.net.http.common.SSLFlowDelegate$Writer.processData([email protected]/SSLFlowDelegate.java:771)
at jdk.internal.net.http.common.SSLFlowDelegate$Writer$WriterDownstreamPusher.run([email protected]/SSLFlowDelegate.java:645)
at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run([email protected]/SequentialScheduler.java:147)
at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run([email protected]/SequentialScheduler.java:198)
at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule([email protected]/SequentialScheduler.java:271)
at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule([email protected]/SequentialScheduler.java:224)
at jdk.internal.net.http.common.SSLFlowDelegate$Writer.triggerWrite([email protected]/SSLFlowDelegate.java:722)
at jdk.internal.net.http.common.SSLFlowDelegate.doHandshake([email protected]/SSLFlowDelegate.java:1024)
at jdk.internal.net.http.common.SSLFlowDelegate.doClosure([email protected]/SSLFlowDelegate.java:1094)
at jdk.internal.net.http.common.SSLFlowDelegate$Reader.unwrapBuffer([email protected]/SSLFlowDelegate.java:500)
at jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData([email protected]/SSLFlowDelegate.java:389)
- locked <0x00000000f97668d0> (a java.lang.Object)
at jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run([email protected]/SSLFlowDelegate.java:263)
at jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run([email protected]/SequentialScheduler.java:175)
- locked <0x00000000f97668f0> (a java.lang.Object)
at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run([email protected]/SequentialScheduler.java:147)
at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run([email protected]/SequentialScheduler.java:198)
at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
at java.lang.Thread.run([email protected]/Thread.java:834)
Locked ownable synchronizers:
- <0x00000000f9894cc0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Same observation but on another container where only one thread was affected.
"HttpClient-3-Worker-2" #120 daemon prio=5 os_prio=0 cpu=1100568.51ms elapsed=1113.79s tid=0x00007eff3003b800 nid=0x479 runnable [0x00007eff83bf8000]
java.lang.Thread.State: RUNNABLE
at sun.security.ssl.SSLEngineImpl.wrap([email protected]/SSLEngineImpl.java:136)
- eliminated <0x00000000f9796e08> (a sun.security.ssl.SSLEngineImpl)
at sun.security.ssl.SSLEngineImpl.wrap([email protected]/SSLEngineImpl.java:116)
- locked <0x00000000f9796e08> (a sun.security.ssl.SSLEngineImpl)
at javax.net.ssl.SSLEngine.wrap([email protected]/SSLEngine.java:519)
at jdk.internal.net.http.common.SSLFlowDelegate$Writer.wrapBuffers([email protected]/SSLFlowDelegate.java:821)
at jdk.internal.net.http.common.SSLFlowDelegate$Writer.processData([email protected]/SSLFlowDelegate.java:736)
at jdk.internal.net.http.common.SSLFlowDelegate$Writer$WriterDownstreamPusher.run([email protected]/SSLFlowDelegate.java:645)
at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run([email protected]/SequentialScheduler.java:147)
at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run([email protected]/SequentialScheduler.java:198)
at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule([email protected]/SequentialScheduler.java:271)
at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule([email protected]/SequentialScheduler.java:224)
at jdk.internal.net.http.common.SSLFlowDelegate$Writer.triggerWrite([email protected]/SSLFlowDelegate.java:722)
at jdk.internal.net.http.common.SSLFlowDelegate.doHandshake([email protected]/SSLFlowDelegate.java:1024)
at jdk.internal.net.http.common.SSLFlowDelegate.doClosure([email protected]/SSLFlowDelegate.java:1094)
at jdk.internal.net.http.common.SSLFlowDelegate$Reader.unwrapBuffer([email protected]/SSLFlowDelegate.java:500)
at jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData([email protected]/SSLFlowDelegate.java:389)
- locked <0x00000000f9797010> (a java.lang.Object)
at jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run([email protected]/SSLFlowDelegate.java:263)
at jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run([email protected]/SequentialScheduler.java:175)
- locked <0x00000000f9797030> (a java.lang.Object)
at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run([email protected]/SequentialScheduler.java:147)
at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run([email protected]/SequentialScheduler.java:198)
at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
at java.lang.Thread.run([email protected]/Thread.java:834)
Some example code I am using
httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofString())
.thenApply(logResponse());
Java Version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment (build 11.0.2+9-Debian-3)
OpenJDK 64-Bit Server VM (build 11.0.2+9-Debian-3, mixed mode, sharing)
The problem also occurs when preferring HTTP 2
Updates
Am I using the HttpClient in a wrong way? Could this be a Server Problem? Is it maybe this bug https://bugs.openjdk.java.net/browse/JDK-8207009?
- Client connects to a cloudflare service
- When the issue occurs there are is no open connection to cloudflare via netstat visible
- I can pin the problem to http2 + tlsv1.3 (using the nginx docker image:
nginx:1.15-alpinewith tls1.3 enabled of course)
Zarathustra :
As @jspcal said before disable TLS 1.3.
tl;dr: disable tlsv1.3 via extending/overwriting
<java_home>/conf/security/java.security the jdk.tls.disabledAlgorithms property
Since my application is running in a docker container I changed the base image to disable tls1.3
FROM openjdk:11-jre
...
RUN sed -i "/jdk.tls.disabledAlgorithms=/ s/=.*/=TLSv1.3, SSLv3, RC4, MD5withRSA, DH keySize < 1024, EC keySize < 224, DES40_CBC, RC4_40, 3DES_EDE_CBC/" $(readlink -f /usr/bin/java | sed "s:bin/java::")/conf/security/java.security
As far as I know there is no way of setting this (security) property via a system property! See also sun.security.util.DisabledAlgorithmConstraints#PROPERTY_TLS_DISABLED_ALGS which actually ready the property.
Update: Bug is still present in 11.0.2