Recently, the load of a business library has been much higher than usual. The most intuitive impression is that the original load is up to 100%. Now it has not doubled several times or increased exponentially, but suddenly doubled by 100 times, resulting in back-end business data The write surge has caused severe performance blockage.
Introduce read and write separation, optimization has achieved initial results
This kind of question aroused my interest and curiosity. After communicating with the business side, I learned that this business records the receipt data.
Simply put, it's like you have sent a Weibo to see how many people have read it, how many people have left a message, etc. Therefore, there are no transactions in this type of scenario, there will be data-intensive writing, and there will be clear statistical requirements.
The current statistical frequency is to do statistics every 7 minutes, there will be several types of statistical scenarios, currently basically all query statements at the full table scan level. The current database architecture is very simple, it is a master-slave, plus MHA high availability.
The improvement direction of the problem is to reduce the pressure of the main library, which is the pressure of reading and writing. The pressure of writing comes from the concurrent writing pressure of the business, and the pressure of reading comes from the pressure of the full table scan, which puts a lot of pressure on the CPU and IO.
There are still priorities for solving these two problems. First, statistical SQL has caused system resources to become a bottleneck. As a result, the original simple Insert has also become a slow log SQL. In contrast, writing requirements are hard requirements.
The statistical requirements are auxiliary requirements, so in this scenario, the quick response method for communicating with the business side is to transfer the statistical requirements of the main database to the secondary database.
The load of the read request was transferred, and the writing pressure was greatly relieved. Later, after optimization at the application level of the business side, the overall load situation was relatively optimistic.
The monitoring load of the main library is as follows:
It can be seen that there is a significant reduction trend, the CPU load has dropped from more than 90% to less than 10%. The IO pressure has also dropped from nearly 100% to about 25%.
The monitoring load of the slave library is as follows:
It can be seen that the pressure has increased significantly. The CPU level is not obvious enough, and the main pressure lies at the IO level, that is, the cost of scanning the full table data is extremely high.
This is the first step of optimization. On this basis, index optimization is started, but through comparison, the effect is found to be very limited.
Because of the statistical requirements from the database side, the added index can only be downgraded from a full table scan to a full index scan, and the overall load improvement of the system is very limited, so we need to make some improvements and optimizations to the existing architecture.
plan 1
Taking into account the cost of resources and usage scenarios, we temporarily adjust the architecture to the following way: that is, add two data nodes, and then plan to enable middleware to design distributed architecture.
For the slave library, temporarily in order to save costs, the original server has been expanded with resources, that is, a single-machine multi-instance model, so that the pressure of writing can be fully supported.
But this method has a potential hidden danger, that is, from the middleware level of the library to play the role of data statistics. Once performance problems occur, the pressure on the middleware is extremely high, which may cause the original statistical tasks to be blocked.
At the same time, the resource bottleneck from the library side is IO pressure in addition to disk space. At present, space expansion cannot solve this flaw.
After further communication with business classmates, they found that their creation of this type of table is a dynamic configuration method, which is difficult to implement in current middleware solutions. And for business, statistical requirements have become more opaque.
Scenario 2
An effective way to improve is to do data routing at the application level. For example, there are 10 services: service 1, service 2 at the first node, service 3, service 5 at the second node, and so on.
The data source is mapped according to this routing configuration method, which is relatively controllable and easier to expand, so the architecture method is changed to this:
In the whole improvement, the most critical link is the improvement of statistical SQL performance. If the improvement of SQL statistical performance can produce initial results, subsequent structural improvements will be easier.
Introduce columnar storage to optimize statistical performance
Subsequent explosive growth of business began again, making the optimization of statistical requirements the key to this optimization.
The original main library was under great read and write pressure. Through the separation of read and write, the pressure on read nodes began to increase sharply. With the expansion of business, the demand for statistical queries has increased.
For example, there were originally 10 queries, but now it may become 30. As a result, the statistical pressure will increase, resulting in a decrease in system response, and thus the latency of the slave database will also start to increase.
The maximum delay is 3 hours. According to this situation, the statistical significance is actually not great.
I made several improvements to this:
The first is to have a detailed communication with the business side, and have a clearer understanding of the business scenario. In fact, this business scenario is quite suitable for solutions such as Redis, but the relational type is chosen between cost and cost performance. MySQL, conclusion: keep the status quo for now.
Regarding reading pressure, not only can't support exponential pressure at present, even the status quo is worrying. Each statistical requirement of the business involves 5 SQL, and every scenario needs to be optimized.
The final preliminary effect achieved is that there are 5 fields and 3 indexes. What is less controllable is that once the data volume of a certain table is too large and the delay is caused, the delay of the entire system will become larger, which will cause statistical requirements. All collapsed.
Therefore, adding an index to solve the hard statistical needs is considered as strong and insufficient. Conclusion: The effect of index optimization is limited, and other feasible solutions need to be sought.
For the writing pressure, the subsequent sharding strategy can be used to solve the sharding strategy. The sharding strategy here is different from our traditional logic. This is based on application-level sharding, and the application side does this data routing. In this way, sharding can easily expand for explosive business growth.
With this level of guarantee, the statistical requirements of the business are migrated to the slave library, and the writing pressure can be smoothly connected. At present, there is a lot of room for writing pressure, which can fully support the exponential pressure. Conclusion: Business data routing should be improved after statistical pressure has eased.
In order to quickly improve the status quo, I wrote a script to automatically collect and manage it, which will periodically kill overtime query sessions.
But the delay still exists, and the query is still slow. It is difficult to imagine how much this delay will be under the condition of exponential pressure.
After doing a lot of comparison tests, according to the data volume of a single table of 35 million, 8 tables with the same data volume, 5 statistical SQL, it takes about 17-18 minutes to complete the statistics, and each table needs about 2 points on average. Many minutes.
Because it is not without transaction correlation, the delay of this scenario is certain according to the business scenario and technical implementation. Our improvement method is to improve the efficiency of statistical query while ensuring that the pressure of the system is within a controllable range.
An effective way is to use a data warehouse solution. MySQL does not natively support database warehouses, but there are third-party solutions:
One type is ColumnStore, which is modified on the basis of InfiniDB.
One is Infobright, in addition to other large-scale solutions, such as Greenplum's MPP solution.
ColumnStore 的方案有点类似于这种 MPP 方案，需要的是分布式节点，所以在资源和架构上 Infobright 更加轻量一些。
我们的表结构很简单，字段类型也是基本类型，而且在团队内部也有大量的实践经验。
改进之后的整体架构如下，原生的主从架构不受影响：
需要在此基础上扩展一个数据仓库节点，数据量可以根据需要继续扩容。
表结构如下：
CREATE TABLE `receipt_12149_428` (
`id` int(11) NOT NULL COMMENT '自增主键',
`userid` int(11) NOT NULL DEFAULT '0' COMMENT '用户ID',
`action` int(11) NOT NULL DEFAULT '0' COMMENT '动作',
`readtimes` int(11) NOT NULL DEFAULT '0' COMMENT '阅读次数',
`create_time` datetime NOT NULL COMMENT '创建时间'
) ;
导出的语句类似于：
select *from ${tab_name} where create_time between xxx and xxxx into outfile '/data/dump_data/${tab_name}.csv' FIELDS TERMINATED BY ' ' ENCLOSED BY '\"';
Infobright 社区版是不支持 DDL 和 DML 的，后期 Infobright 官方宣布：不再发布 ICE 社区版，将专注于 IEE 的开发，所以后续的支持力度其实就很有限了。对于我们目前的需求来说是游刃有余。
来简单感受下 Infobright 的实力：
>select count( id) from testxxx where id>2000;
+------------+
| count( id) |
+------------+
| 727686205 |
+------------+
1 row in set (6.20 sec)
>select count( id) from testxxxx where id<2000;
+------------+
| count( id) |
+------------+
| 13826684 |
+------------+
1 row in set (8.21 sec)
>select count( distinct id) from testxxxx where id<2000;
+---------------------+
| count( distinct id) |
+---------------------+
| 1999 |
+---------------------+
1 row in set (10.20 sec)
所以对于几千万的表来说，这都不是事儿。我把 3500 万的数据导入到 Infobright 里面，5 条查询语句总共的执行时间维持在 14 秒，相比原来的 2 分多钟已经改进很大了。
我跑了下批量的查询，原本要 18 分钟，现在只需要不到 3 分钟。
引入动态调度，解决统计延迟问题
通过引入 Infobright 方案对已有的统计需求可以做到完美支持，但是随之而来的一个难点就是对于数据的流转如何平滑支持。
我们可以设定流转频率，比如 10 分钟等或者半个小时，但是目前来看，这个是需要额外的脚本或工具来做的。
在具体落地的过程中，发现有一大堆的事情需要提前搞定。
其一：比如第一个头疼的问题就是全量的同步，第一次同步肯定是全量的，这么多的数据怎么同步到 Infobright 里面。
第二个问题，也是更为关键的，那就是同步策略是怎么设定的，是否可以支持的更加灵活。
第三个问题是基于现有的增量同步方案，需要在时间字段上添加索引。对于线上的操作而言又是一个巨大的挑战。
其二：从目前的业务需求来说，最多能够允许一个小时的统计延迟，如果后期要做大量的运营活动，需要更精确的数据支持，要得到半个小时的统计数据，按照现有的方案是否能够支持。
这两个主要的问题，任何一个解决不了，数据流转能够落地都是难题，这个问题留给我的时间只有一天。
所以我准备把前期的准备和测试做得扎实一些，后期接入的时候就会顺畅得多。
部分脚本实现如下：
脚本的输入参数有两个，一个是起始时间，一个是截止时间。第一次全量同步的时候，可以把起始时间给的早一些，这样截止时间是固定的，逻辑上就是全量的。
另外全量同步的时候一定要确保主从延迟已经最低或者暂时停掉查询业务，使得数据全量抽取更加顺利。
所以需要对上述脚本再做一层保证，通过计算当前时间和上一次执行的时间来得到任务可执行的时间。这样脚本就不需要参数了，这是一个动态调度的迭代过程。
考虑到每天落盘的数据量大概在 10G 左右，日志量在 30G 左右，所以考虑先使用客户端导入 Infobright 的方式来操作。
从实践来看，涉及的表有 600 多个，我先导出了一个列表，按照数据量来排序，这样小表就可以快速导入，大表放在最后，整个数据量有 150G 左右，通过网络传输导入 Infobright，从导出到导入完成，这个过程大概需要 1 个小时。
而导入数据到 Infobright 之后的性能提升也是极为明显的。原来的一组查询持续时间在半个小时，现在在 70 秒钟即可完成。对于业务的体验来说大大提高。
完成了第一次同步之后，后续的同步都可以根据实际的情况来灵活控制。所以数据增量同步暂时是“手动挡”控制。
从整个数据架构分离之后的效果来看，从库的压力大大降低，而效率也大大提高。
引入业务路由，平滑支持业务扩容
前面算是对现状做到了最大程度的优化，但是还有一个问题，目前的架构暂时能够支撑密集型数据写入，但是不能够支持指数级别的压力请求，而且存储容量很难以扩展。
From my understanding, data routing at the business level is the best way, and it is more friendly in terms of expansion.
So the further improvement plan is as follows:
To achieve load balancing through data routing, the effect is obvious from the current point of view, and it is also a controllable way for the business when the subsequent expansion is to be continued.
The following is the pressure of the IO from the library in some recent optimization time periods:
After solving the problem several times, supplementing and following up the plan, we have completed the initial failure to the successful landing, and the optimization and sharing of the architecture of MySQL performance expansion has basically ended. If there is a better way to achieve it, welcome to share in the message area!