I have five tomcat instances behind nginx.
Sometimes the nginx upstream_response_time is very big, more than 1 second, while the tomcat local access log shows the process time is only 50ms(I use %D to log process time).
What is the possible reason and how to fix it? It does not seems the network is slow since other applications run fast.
Update:
Seems the nginx upstream_response_time = %D + 1 sec.
The servers normally queue the requests until a thread is available to handle it. If there are many requests in the queue but only a few threads, a single thread might handle the request quite fast, but if you add the time, the request was queued, the consumer sees a much longer time.
See: How to increase number of threads in tomcat thread pool?
Measuring the number of queued requests for tomcat
See if you can increase the number of threads or decrease the accept_count, but keep in mind, that the number of other resources like database connections might also need to be increased. Also keep in mind, that more threads might mean more competing for resources.
It might be worth to try change the parameters for this. Normally the access-log should also show the time the message is queue and handled, but I am not sure.