ali的ons mq运行一段时间后消费下降并导致堆积问题查验

1：问题现象：

运行的instance一段时间(20h)就下降，重启之后消费正常然后又不行了；原以为是ons版本1.2.7改成laest1.7.7.final;没效果；经验之觉：肯定是代码没优化好：

处理流程一：单纯以为应该是gc没做好；有big Object ;./jmap发现了MsgContent；查project使用 ConcurrentHashMap<String ,MsgContent>一直add没有remove；so 添加remove并且就加上value = null;利于gc发现没太大效果；

public static ConcurrentHashMap<String ,MsgContent> map = new ConcurrentHashMap<String ,MsgContent>();
//遍历map中的value，然后查看value中的time值是不是超过了两分钟，是的话就删除掉对应的key  
public static void removeInvalidKey(ConcurrentHashMap<String,MsgContent> map){
        for (MsgContent value : map.values()) {
            if (System.currentTimeMillis()-value.getTime() > 2 * 60 * 1000) {
                MsgMatch.map.remove(value.getUid());
                value = null;//强制把对象设置null,check object被gc回收（System.gc()）
            }
        }
    }

 num     #instances         #bytes  class name
----------------------------------------------
   1:        651850      208798320  [C
   2:        651267       15630408  java.lang.String
   3:         71571       10226008  <constMethodKlass>
   4:         71571        9172944  <methodKlass>
   5:          6020        6965584  <constantPoolKlass>
   6:         20793        5553840  [I
   7:        153195        4902240  java.util.HashMap$Entry
   8:         24879        4784448  [B
   9:        189633        4551192  java.util.concurrent.ConcurrentLinkedDeque$Node
  10:          6020        4496624  <instanceKlassKlass>
  11:          5076        4044384  <constantPoolCacheKlass>
  12:         78356        2507392  java.util.concurrent.ConcurrentHashMap$HashEntry
  13:         64274        2506768  com.xxx.xxx.access.mysql.entity.MsgContent

处理流程二：经过流程一；instance能正常跑(30h),还没找到病原体；没办法去找thread Stack快照：发现线程runable一个地方(这时jvm已经小露病源了)如图:

"ConsumeMessageThread_7" prio=10 tid=0x00007f6498008000 nid=0x43 runnable [0x00007f6558c82000]
   java.lang.Thread.State: RUNNABLE
	at java.util.concurrent.ConcurrentLinkedDeque.contains(ConcurrentLinkedDeque.java:1085)
	at com.xxx.xxxx.access.alimq.EvMsgRtListener.consume(EvMsgRtListener.java:169)
	at com.aliyun.openservices.ons.api.impl.rocketmq.ConsumerImpl$MessageListenerImpl.consumeMessage(ConsumerImpl.java:97)
	at com.aliyun.openservices.shade.com.alibaba.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- <0x000000070841ea38> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"ConsumeMessageThread_5" prio=10 tid=0x00007f6498004000 nid=0x42 runnable [0x00007f6558d83000]
   java.lang.Thread.State: RUNNABLE
	at java.util.concurrent.ConcurrentLinkedDeque.contains(ConcurrentLinkedDeque.java:1085)
	at com.xxx.xxxx.access.alimq.EvMsgRtListener.consume(EvMsgRtListener.java:169)
	at com.aliyun.openservices.ons.api.impl.rocketmq.ConsumerImpl$MessageListenerImpl.consumeMessage(ConsumerImpl.java:97)
	at com.aliyun.openservices.shade.com.alibaba.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:417)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- <0x000000070841f708> (a java.util.concurrent.ThreadPoolExecutor$Worker)

代码此处：

此处的queue是一个定时任务；涉及到遍历及remove key操作，因为ConcurrentLinkedDeque此处操作会严重拖耗性能，每一次重构需要重新排序；详细参考JAVA集合框架中的常用集合及其特点、适用场景、实现原理简介

此时问题发现注释解决：总结一下：之前多次遇到过同样场景：运行一段时间cpu飙升；消费能力下降；：也是涉及到远程调用http SocketTimeout(5000) ---》5000修改为1s;缩短时间，避免长时间进行响应阻塞，thread运行

CloseableHttpClient httpclient = HttpClients.createDefault();
		HttpPost http = new HttpPost(url);

		/**
		 * setConnectTimeout：设置连接超时时间，单位毫秒。
		 * setConnectionRequestTimeout：设置从connect Manager获取Connection 超时时间，单位毫秒
		 * setSocketTimeout：请求获取数据的超时时间，单位毫秒。 如果访问一个接口，多少时间内无法返回数据，就直接放弃此次调用
		 */
		RequestConfig requestConfig = RequestConfig.custom().setConnectTimeout(5000).setConnectionRequestTimeout(1000)
				.setSocketTimeout(5000).build();
		http.setConfig(requestConfig);
		HttpEntity inEntity = EntityBuilder.create().setText(json).setContentType(ContentType.APPLICATION_JSON).build();
		http.setEntity(inEntity);
		CloseableHttpResponse response = httpclient.execute(http);

ps：提到这缓存；设计缓存要清楚各个组件性能及优缺点：

简单一点用hashMap;上文就提到清理无效的数据时；如何彻底gc防止数据过多导致溢出；一个好的替代方案是weakHashMap;是使用弱引用维护一张哈希表；but 作为专业缓存，功能上略有不足；详见：WeakHashMap和HashMap的区别；更详细的：话说ReferenceQueue

ali的ons mq运行一段时间后消费下降并导致堆积问题查验

猜你喜欢