准备
本次测试具备的环境很简单，项目都是 java 项目，jdk 为1.7（1.8 也成），tomcat 为7版本，war 包放进 tomcat 路径下的 webapps 就成。
然后改数据库配置文件
sql 文件放到数据库内执行，生成相应的数据库以及表
首页：http://www.ganziwen.cn:8082/dangdang/main/main.jsp
注册页面：http://www.ganziwen.cn:8082/dangdang/user/registForm.jsp
线程栈死锁
问题描述
数据库保证已经完全连上，但是首页的图片无法加载出来
排查方案
出现这种问题尝试结合之前讲的架构从几个方面去分析，主要是排查法：
一、请求没法送到服务器
- 负载机 cpu、内存、网络、磁盘
二、请求发送到服务器但是无法被处理
- 应用服务器/数据库服务器 cpu、内存、网络、磁盘
- 容器连接池
- 数据库连接池
- 代码逻辑
- sql 语句慢
- JVM 的堆内存/gc 频繁（jstat -gcutil pid）
- 线程栈问题（jstack pid）
- 磁盘 io(nmon/sar -d/iostat -x)
排查过程
这里我们的负载机排查很好排查：直接访问一个服务器上的其他链接：
从这张图片，可以侧面论证，我们的负载机是没有问题的，最起码可以访问服务器，同时也说明 web 容器的连接池并没有满，因为可以访问我们的 82 端口，那么尝试从其他方面去考虑，看看 jvm 和线程栈
首先我们查一下 java 进程，得到 pid 为 8071：
# ps -ef|grep java|grep -v grep root 8071 1 0 13:10 pts/0 00:00:10 /opt/jdk1.8/bin/java -Djava.util.logging.config.file=/opt/tomcat7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Dignore.endorsed.dirs= -classpath /opt/tomcat7/bin/bootstrap.jar:/opt/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat7 -Dcatalina.home=/opt/tomcat7 -Djava.io.tmpdir=/opt/tomcat7/temp org.apache.catalina.startup.Bootstrap start
然后我们看一下 jvm 的 gc 情况：
# jstat -gcutil 8071 3000 S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 100.00 64.96 82.81 96.73 92.25 25 0.189 1 0.051 0.239 0.00 100.00 64.96 82.81 96.73 92.25 25 0.189 1 0.051 0.239 0.00 100.00 65.23 82.81 96.73 92.25 25 0.189 1 0.051 0.239 0.00 100.00 65.23 82.81 96.73 92.25 25 0.189 1 0.051 0.239
可以发现。，gc 的频率并不是很高，暂时排除是 gc 频繁导致的
接下来看看线程栈问题：
jstack 8071 > 1.log
打开 1.log：
我们可以发现，在http-nio-8082-exec-XXX这种的线程中状态几乎都为 waitting 或者 blocked，而且有一个 deadblock
Found one Java-level deadlock: ============================= "http-bio-8082-exec-10": waiting to lock monitor 0x00007fa8a4005c98 (object 0x00000000e161a920, a java.lang.Object), which is held by "http-bio-8082-exec-8" "http-bio-8082-exec-8": waiting to lock monitor 0x00007fa888005c98 (object 0x00000000e16159d8, a java.lang.Object), which is held by "http-bio-8082-exec-10"
我们看到这里，大概可以判断到应该是线程栈死锁导致的，而且可以看到在锁住的线程调用的方法的路径以及方法名是：org.tarena.common.DbUtil.getConnection(DbUtil.java:43)
"http-nio-8082-exec-109" #139 daemon prio=5 os_prio=0 tid=0x00007f915c840800 nid=0x2087 waiting for monitor entry [0x00007f91336b2000] java.lang.Thread.State: BLOCKED (on object monitor) at org.tarena.common.DbUtil.getConnection(DbUtil.java:43) - waiting to lock <0x00000000ec36cbb8> (a java.lang.Object) at org.tarena.dao.impl.DangDAOImpl.getConnection(DangDAOImpl.java:18) at org.tarena.dao.impl.BookDAOImpl.findHotBoardBooks(BookDAOImpl.java:159) at org.tarena.action.main.HotBoardBook.execute(HotBoardBook.java:20) at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:441) at com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:280) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:243) at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:165) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:252) at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:68) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:122) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:195) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:195) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:179) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:75) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:94) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:235) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:89) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:130) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:267) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:126) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:138) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:87) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:165) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:179) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:176) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:237) at org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:52) at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:488) at org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:77) at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:91) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1137) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734) - locked <0x00000000e0f1c108> (a org.apache.tomcat.util.net.NioChannel) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748)