This morning UIOC, the phenomenon is that the interface that calls a service always times out. It is very strange that this environment has no problem without a version last night. I went to the company for various investigations, and finally through the stack information printed before the restart of the problem service, I saw that almost all threads were waiting for the connection pool to release the connection and obtain an idle connection. The error is as follows:
"xxxxxxxxxxxxx" prio = 10 tid = 0x00007f6b7c002800 nid = 0x40ff waiting on condition [0x00007f6b37020000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f97918b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkUntil(LockSupport.java:239)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUntil(AbstractQueuedSynchronizer.java:2072)
at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:129)
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:281)
at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:62)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:176)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:172)
at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:100)
at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:212)
The problem is already obvious, because the connection pool is used, but the connection is not enough, causing a lot of waiting, and the subsequent requests cannot get the connection, so it eventually times out, and this waiting has a snowball effect.
The main reason for the problem is that we set the httpclient connection pool parameters unreasonably. We are using httpclient 4.4.1. In addition to the connection timeout and response timeout parameters, the two parameters related to the connection pool are as follows:
1. MaxtTotal: the size of the entire connection pool
2. DefaultMaxPerRoute: A subdivision of MaxTotal according to the connected host
For example: MaxtTotal=400 DefaultMaxPerRoute=200
When only connecting to http://x.com, the concurrency to this host is at most 200; instead of 400;
When connecting to http://x.com and http://y.com, the concurrency to each host is at most 200; that is, the total is 400 (but not more than 400), so the effective setting is DefaultMaxPerRoute.
I have been a little confused about thread pools and connection pools before . To summarize:
The concept of thread pool and connection pool (http connection pool or db connection pool) is actually the same. Repeated creation will consume system resources, so it is necessary to use the pool to save it for repeated use. For the scenario of using the tomcat container, each running thread in the tomcat worker thread pool is processing an upstream request, and when it needs to access a third-party interface or DB, it takes the connection from the corresponding pool for synchronous request operation (generally, no other Start a thread to asynchronously request a third-party interface or DB). During the synchronous request operation (after the request is issued), the tomcat worker thread will switch to the waiting state (WAITING), and after the request response returns, the operating system will finally wake up the thread and switch to the running state (RUNNABLE) according to the scheduling algorithm. The operation (accessing the third-party interface or DB) responds very quickly, and the two context switches of this thread are also very fast. With jstack, the intermediate state is almost invisible.
Related Links:
Parameter settings, code writing, and risks that you must know when using httpclient
http://jinnianshilongnian.iteye.com/blog/2089792