Records system performance tuning process

Review Questions

---

The online environment, due to the need for business scenarios require the program can still function properly in normal 4G machine. The original configuration environment for 8-core 16G, micro-services deployment, a total of six modules. And now it requires to run properly on a 4-core 4G devices.

K

---

• 1. Merge module during various conflicts, all kinds of Bean not load properly
• 2. Original event processing performance of about 3000 ~ 1w per second, now suddenly dropped to several hundred;
• 3. There is the phenomenon of loss events, but also the loss of more serious
• 4. We found that the system cache has continued to rise, free memory than can be found with almost gone after -m (about remaining 100M, in fact, is almost run out, will not be reduced)

Troubleshooting

1. Code Violation

---

• Package name conflict. Duplicate package names on the design of the different modules
• Class name conflicts. @Configuration @Bean @Controller @Service @RepositoryBean instance names such notes is not specified.

2. Event processing performance slow

---

The conventional process is as follows:

项目采用SpringBoot构建，引入 spring-boot-stater-redis
1. 通过HTTP接收到异步事件，存储到Redis;
2. 存储的同时，将事件通过Redis的发布订阅发送到不同的处理单元进行处理；
3. 每个事件处理单元通过Redis订阅，然后处理事件；
4. 起一个定时器，每秒钟从Redis中查询一个时间窗口的事件，构建索引，然后bulkIndex到ES
复制代码

2.1 problems found

1. Redis的订阅发布，内部会维护一个container线程，此线程会一直存在；
2. 每次订阅，都会产生一个新的前缀为RedisListeningContainer-的线程处理;
3. 通过jvisualvm.exe 查看线程数，该类线程数一直在飙升
复制代码

2.2 locate the problem

2.2.1 Redis Subscribe to post questions

Implement the program are as follows:

@Bean
RedisMessageListenerContainer manageContainer(
        RedisConnectionFactory factory, MessageListener listener) {
  RedisMessageListenerContainer manageContainer = 
                    new RedisMessageListenerContainer ();
  manageContainer.setConnectionFactory(factory);
  // manageContainer.setTaskExecutor();
}
复制代码

Code is commented that, the actual code is not the line, which is not providedtaskExecutor

• Query the spring-redis.xsd regarding the listener-containerdescription of default task-executorand subscription-task-executorusing SimpleAsyncTaskExecutor.
• Location in the source code

RedisMessageListenerContainer.class

...
protected TaskExecutor createDefaultTaskExecutor() {
    String threadNamePrefix = (beanName != null ? beanName + "-" :
    DEFAULT_THREAD_NAME_PREFIX) ;
    return new SimpleAsyncTaskExecutor(threadNamePrefix);
}
...
复制代码

SimpleAsyncTaskExecutor.class

...
protected void doExecute(Runnable task) {
    Thread thread = 
        (this.threadFactory != null 
            ? this.threadFactory,newThread(task) 
            : createThread(task));
    thread.start();
}
...
复制代码

• SimpleAsyncTaskExecutorThe execute () method, is very shameless new Thread()call thread.start()to perform the task

2.2.2 Event processing is time-consuming operation, resulting in more and more threads, even OOM

2.3 Problem Solving

Find the cause of the problem, there are three main Solutions:

• Configuration manageContainer.setTaskExecutor();and choose to create their own thread pool;

• Remove part of the publication subscription in favor of Springproviding the observer mode , the scene of most of the event processing, this is accomplished by publication.SpringUtils.getApplicationContext() .publihEvent(newEventOperation(eventList));

• Using the Rectormodel to achieve efficient asynchronous event processing;

创建了2个线程组（参考netty的底层实现）:
1. 一个用于处理事件接收 “event-recv-executor-”
    coreSize = N * 2，CPU密集型
2. 一个用于事件的异步处理 “event-task-executor-” 
    coreSize = N / 0.1，IO密集型
复制代码

事件处理逻辑
@Override
public void onApplicationEvent (EventOperation event) {
    eventTaskExecutor.execute(() -> {
        doDealEventOperation(event);
    });
}
复制代码

3. Event loss

---

The conventional process is as follows:

项目采用SpringBoot构建，引入 spring-boot-stater-redis
1. 后台维护了一个定时器，每秒钟从Redis中查询一个时间窗口的事件
复制代码

3.1 problems found

在后台定位日志输出，正常情况下，应该是每秒钟执行一次定时，
但实际是，系统并不保证一定能每隔1S执行一次，
由于系统中线程比较多，CPU的切换频繁，
导致定时有可能1S执行几次或者每隔几秒执行一次
复制代码

3.2 locate the problem

3.2.1 regular tasks unreliable

由于定时并无法保证执行，而定时任务获取事件时，是按照时间窗口截取，
通过redisTemplate.opsForZSet().rangeByScore(key, minScore, maxScore)实现，
势必会造成有数据无法被加载到程序中，而一直保存在Redis中，无法获取，也无法删除
复制代码

3.3 Problem Solving

Find the cause of the problem, there are two main Solutions:

• Increase fault tolerance , will widen the time window, the original is the time window 1S apart, separated by 1MIN modify the [temporary solution, in extreme cases, may still cause the problem];

• Using MQ consumption , this method requires the deployment of additional MQ server, in the high cluster configuration can be used, improper configuration at low machine;

• Using blocking queue , use Lock.newCondition()or most common network monitoring mode while()can;

The question is used in the third form. From a separate thread, blocking monitor.

1. 事件接收后，直接塞到一个BlockingQueue中；
2. 当BlockingQueue有数据时，While循环不阻塞，逐条读取队列中的信息；
3. 每隔1000条数据，或者每隔1S，将数据写入ES，并分发其他处理流程
复制代码

4. The system cache is constantly rising

---

In 4G machine and found after a bidding process for some time, the system cache growth is very fast, almost to the last fully occupied:

大概每秒钟10M的涨幅
复制代码

4.1 problems found

1. 因为对于ES的了解，插入数据时，先写缓存，后fsync到磁盘上，因此怀疑ES可能存在问题；
2. 项目中日志使用log4j2不当：
    * 日志输出过多，
    * 日志没有加判断：if (log.isInfoEnabled()) 
    * 日志文件append过大，没有按照大小切分等（本项目此问题之前已解决）
复制代码

4.2 locate the problem

4.2.1 ES insertion mechanisms

经过隔段分析，将有可能出现问题的地方，分别屏蔽后，进行测试。
最终定位到，在ES批量写入数据时，才会出现cache大量增长的现象
复制代码

4.3 Problem Solving

Check with the command memory free -m,

• buffer: As a buffer cachememory, a block device to read and write buffer
• cachedRepresentation page cache的内存and文件系统的cache
• If the cached value is large, the number of live documentation cache lot

ES operational data underlying mechanisms :

When data is written, ES memory to rise slowly, because too many small files (ES itself will create a large number of small files in the index time), linux dentryand inode cachewill increase. Can refer to: ES memory to locate the problem continues to rise

In fact, this problem is not completely resolved, but in exchange for caching performance to some extent.

• 1. Modify the system parameters to improve memory priority slab release:

echo 10000 > /proc/sys/vm/vfs_cache_pressure；
复制代码

• 2. ES modify configuration parameters

## 这些参数是之前优化的
threadpool.bulk.type: fixed
threadpool.bulk.min: 10
threadpool.bulk.max: 10
threadpool.bulk.queue_size: 2000

threadpool.index.type: fixed
threadpool.index.size: 100
threadpool.index.queue_size: 1000

index.max_result_window: 1000000
index.query.bool.max_clause_count: 1024000


# 以下的参数为本次优化中添加的：

# 设置ES最大缓存数据条数和缓存失效时间
index.cache.field.max_size: 20000
index.cache.field.expire: 1m

# 当内存不足时，对查询结果数据缓存进行回收
index.cache.field.type: soft

# 当内存达到一定比例时，触发GC。默认为JVM的70%[内存使用最大值]
#indices.breaker.total.limit: 70%

# 用于fielddata缓存的内存数量，
# 主要用于当使用排序操作时，ES会将一些热点数据加载到内存中来提供客户端访问
indices.fielddata.cache.expire: 20m
indices.fielddata.cache.size: 10%

# 一个节点索引缓冲区的大小[max 默认无限制]
#indices.memory.index_buffer_size: 10%
#indices.memory.min_index_buffer_size: 48M
#indices.memory.max_index_buffer_size: 100M

# 执行数据过滤时的数据缓存，默认为10%
#indices.cache.filter.size: 10%
#indices.cache.filter.expire: 20m

# 当tranlog的大小达到此值时，会进行一次flush操作，默认是512M
index.translog.flush_threshold_size: 100m

# 在指定时间间隔内如果没有进行进行flush操作，会进行一次强制的flush操作，默认是30分钟
index.translog.flush_threshold_period: 1m

# 多长时间进行一次的磁盘操作，默认是5S
index.gateway.local.sync: 1s
复制代码

Retrospect

---

• For this tuning process, the main direction or modify the code, i.e. code improper use, or results in ill-considered
• Secondly, the underlying ES implementation mechanism is not very familiar with, to target specific problem;
• The optimization process involves the positioning of the GC, the configuration of the underlying system parameters like Linux
• Since the log transmission using HTTP, so that each transmission is a new thread. IO overhead is relatively large, follow-up will be considered replaced by long link .

Attachment:

If errors are described in the text content, or there is no clear description of the problem you can send me mail, [email protected]if there are other questions, you can contact me and we discuss together.

• I would like to make progress together, grow together.