This article introduces the resource management and control technology of TiDB database and verifies the effect through business testing. Resource management and control technology aims to solve resource isolation and load problems when multiple businesses share a cluster. Through the concept of resource groups, the computing and I/O resources of different businesses can be restricted, resource isolation and priority scheduling can be achieved, and system utilization and stability can be improved. sex.
business background
As the use of TiDB by businesses continues to expand and deepen, when multiple businesses share a cluster, I believe many users have also encountered the problem of mutual influence between different loads. In previous versions, TiDB was also trying different methods to alleviate or solve such problems. A typical example is to introduce the TiFlash column storage component to distinguish online processing transactions on TiKV and analytical tasks on TiFlash at the storage engine level, and physically isolate different loads at the storage layer. This kind of architecture optimization has many benefits, but if the business requires access to TiKV to get results, there is no way to deal with it.
We have more than a dozen online production clusters. Considering issues such as cost and operation and maintenance, multiple services share one cluster. On the premise that we deploy the TP business and AP business separately as much as possible, we usually still encounter the following problems:
● When a business is at its peak, it will occupy too many resources used by other businesses, thus affecting the normal operation of other businesses.
○ We hope to protect the resource holdings of different businesses and ensure that businesses can allocate basic operating resources without being run on.
● When the important business in the cluster is at a low point, there are more remaining resources. If we can bring in the off-peak business, we can make full use of the resources and reduce costs and increase efficiency. But this requires that the business running at off-peak hours needs to be controlled and not occupy too many resources at other times.
● When the cluster encounters temporary performance problems caused by SQL, it can only stop SQL.
○ We prefer not to kill its execution, but to temporarily limit its resource consumption and allow it to run slowly without affecting other cluster services.
In the context of such business pain points, TiDB v7.1.0 proposed resource management and control technology. We followed up on this technology immediately and tried to explore isolation solutions for the use of multi-tenant resources in converged systems.
TiDB resource management and control technology
Resource control technology (Resource Control) can ensure service quality when the load changes drastically. It also provides multi-tenant isolation capabilities of the database, which can effectively reduce database operating costs.
2.1 Principle description
The TiDB resource management and control feature provides two layers of resource management capabilities, including flow control capabilities at the TiDB layer and priority scheduling capabilities at the TiKV layer. After binding a user to a resource group, the TiDB layer will perform flow control on the user's read and write requests based on the quota set by the resource group to which the user is bound, and the TiKV layer will schedule the requests based on the priority of the quota mapping. Through the two layers of control, flow control and scheduling, application resource isolation can be achieved to meet quality of service (QoS) requirements.
● TiDB flow control: TiDB flow control uses the token bucket algorithm ( https://en.wikipedia.org/wiki/Token_bucket ) for flow control. If there are not enough tokens in the bucket and the resource group does not specify the BURSTABLE attribute, requests belonging to the resource group will wait for the token bucket to be backfilled with tokens and try again. The retry may time out and fail.
● TiKV scheduling: You can set the absolute priority (PRIORITY) for the resource group. Different resources are scheduled according to the PRIORITY setting. Tasks with high PRIORITY will be scheduled first. If PRIORITY is not set, TiKV will map the RU_PER_SEC value of the resource group to the priority of the read and write requests of the respective resource group, and use priority queue scheduling at the storage layer to process the requests based on their respective priorities.
TiDB resource management and control technology uses resource groups to divide the cluster into multiple logical units. Each resource group can limit the computing and I/O resources it requires. When the cluster has idle resources, specific settings can be used to allow some resource groups to exceed their limits and make full use of cluster resources. It basically solves the problem of resource competition caused by the merger of multiple businesses and ensures the stability of the business. The following is a conceptual diagram of this technology:
Resource Control is completed based on TiDB's flow control and TiKV's scheduling function. At the same time, the BURSTABLE function allows it to exceed the constrained quota of the resource group, so that it can ensure the normal operation of the service.
2.2 Management methods
Resource management and control introduces the concept of resource group (Resource Group). By setting the corresponding relationship between "user" and "resource group", the session is bound to the user group, and the "usage (RU)" is used to define the resource limit. The structure is as follows ( https://tidb.net/blog/67d82266 ):
For details on resource groups, resource quotas, scheduling priorities and other features, please refer to the official website ( https://docs.pingcap.com/zh/tidb/stable/tidb-resource-control ).
It is specially stated here that the resource group setting is very flexible, which is very convenient for administrators to adjust according to business usage scenarios. I think this also greatly improves the usability of TiDB. Different levels are set respectively:
● User level. Bind users to specific resource groups. After binding, the newly created session of the corresponding user will be automatically bound to the corresponding resource group.
● Session level. Set the resource group of the current session through SET RESOURCE GROUP.
● Statement level. Set the resource group of the current statement through the optimizer hint RESOURCE_GROUP().
2.3 Technical application points
In summary, this technology mainly solves the following common business problems:
● When there are multiple business loads in the system, resources are isolated to ensure that the response time of transaction services is not affected by data analysis or batch services.
● When the system load is low, busy applications are allowed to exceed the set read and write quotas, maximizing resource utilization, improving hardware utilization, and reducing operating costs.
● Limit the resource consumption of burst SQL to avoid causing cluster performance problems.
● Provide accurate feedback on usage statistics and complete reasonable cost allocation for different businesses.
○ Obtain the actual usage through the monitoring panel and assist users to reasonably improve the configuration. At the same time, in line with enterprise management goals, TiDB can help enterprises accurately calculate the usage of database resources in each department and achieve reasonable cost allocation.
● Provide flexible resource binding methods.
○ Supports specifying resource binding methods at the user level, session level, and statement level to meet the resource control needs of different scenarios.
After sorting out its basic principles and design goals, it seems that it can well solve the business pain points of our actual production environment, so we started further actual testing and verification.
Business verification
TiDB can estimate the overall RU capacity of the cluster based on hardware deployment or actual load. When testing, we directly referred to the estimate based on hardware deployment.
3.1 Business resource simulation
In order to simulate the different most common businesses in our production environment, here we create three tenants, representing three different business loads respectively. Each type of business has different management and control objectives. The following table shows our different businesses running in the same TiDB cluster, and the different operating requirements of each business:
Based on resource management and control technology, we can create resource groups for these three types of users:
● Allocate a higher usage to tenant app_oltp, and relatively lower usage to tenant app_olap and tenant app_other
○ When system resources are tight, the highest priority is to ensure the service quality of tenant app_oltp.
● The resource groups of tenants app_oltp and app_olap are set to burstable.
○ If the tenant app_oltp experiences an unexpected load, the quality may still be guaranteed;
○ When the entire cluster load is idle, tenant app_olap can use idle resources to accelerate its work.
● Create resource groups
CREATE RESOURCE GROUP IF NOT EXISTS rg_oltp RU_PER_SEC = 1000 BURSTABLE PRIORITY = HIGH; CREATE RESOURCE GROUP IF NOT EXISTS rg_olap RU_PER_SEC = 400 BURSTABLE; CREATE RESOURCE GROUP IF NOT EXISTS rg_other RU_PER_SEC = 100;● Our online business already exists. In other words, the business account must already exist when this function is launched. Therefore, the resource group is directly bound to the business during simulation.
ALTER USER app_oltp RESOURCE GROUP rg_oltp; ALTER USER app_olap RESOURCE GROUP rg_olap; ALTER USER app_other RESOURCE GROUP rg_other;3.2 Business operation simulation
We started the short connection business in the test environment to access data in real time, and continuously performed reading and writing operations to simulate different loads of several tenants and observe the business side throughput (QPS) and the resource consumption of the database TiDB (RU usage trend). The entire scene roughly simulates the following scenarios:
- For a business that has a usage upper limit and is running, adjust the cluster resource allocation online:
a. Temporarily expand the available resources of tenant app_other and simulate temporarily adding resources to the online business.
b. Temporarily reduce the tenant app_other quota to simulate temporarily reducing resources for online services.
c. Allow tenant app_other to exceed the resource limit and simulate the temporary cancellation of online business resource usage restrictions.
d. Tenant app_other is not allowed to exceed the resource limit and return to the initial limit state, simulating temporary restrictions on the use of online business resources.
- Simulate the integration and coexistence of different services in the same cluster, and observe the operation of all tenants experiencing a complete cycle of peaks and troughs of the most important services
a. First, three types of loads run at the same time, indicating normal business coexistence.
b. The business traffic peak is coming, and the tenant app_oltp reaches the peak load.
c. The peak value of tenant app_oltp has passed and returned to normal state.
d. The load of tenant app_oltp reaches the lowest value, while other things remain unchanged.
e. Tenant app_oltp The trough has passed and returned to normal state
Increase/decrease business available resources online
For a business that has a usage limit and is running, temporarily adjust the available resources of the tenant app_other to simulate temporarily adding or reducing resources to the online business.
● Initial: Initial business resource quota of tenant app_other
ALTER RESOURCE GROUP rg_other RU_PER_SEC = 100;● Expand: Temporarily expand the available resources of tenant app_other business
ALTER RESOURCE GROUP rg_other RU_PER_SEC = 400;● Reduce: Temporarily reduce the available resources of tenant app_other business
ALTER RESOURCE GROUP rg_other RU_PER_SEC = 50;
As shown in the figure above, you can see that the initial resource quota for the business of tenant app_other is 100, and the business is running stably during this period.
Suppose there is a reason that requires us to temporarily increase its available resources. At this time, increase the available resources RU_PER_SEC = 400. The RUs that can be used by the business will respond immediately and allocate the required resources, and the curve will continue to rise. On the contrary, when we reduce the available resources to RU_PER_SEC = 50, the curve will drop to our expected setting value.
● Summary:
○ It shows that TiDB's resource management and control technology can adjust the business resource usage status online, configure business resources in real time, and greatly improve the business response speed.
○ If this type of business is a sudden abnormal SQL, we can temporarily limit its resource consumption to avoid causing cluster performance problems.
Cancel business quota restrictions online
Allow tenant app_other to exceed resource limits and simulate the scenario of temporarily canceling online business resource usage restrictions.
- Initial: Initial business resource quota of tenant app_other
ALTER RESOURCE GROUP rg_other RU_PER_SEC = 50;- Cancel restrictions: Allow tenant app_other business to exceed the limit of available resources
ALTER RESOURCE GROUP rg_other RU_PER_SEC = 50 BURSTABLE;
As shown in the figure above, you can see that the initial resource quota for the business of tenant app_other is 50, and the business is running stably during this period. At this time, we temporarily cancel its available resource limit. After the cluster receives the configuration, its RU curve continues to rise until the required maximum value.
● Summary:
- ○ Explain that TiDB’s resource management and control technology can adjust the usage status of business resources online and cancel restrictions on the use of business resources in real time.
Limit the maximum available resources of the online business
Tenant app_other is not allowed to exceed the resource limit, simulating temporary restrictions on the use of online business resources.
- Initial: Allow tenant app_other business to exceed the limit of available resources
ALTER RESOURCE GROUP rg_other RU_PER_SEC = 50 BURSTABLE;- Not allowed to exceed the limit: Tenant app_other is not allowed to exceed the limit
ALTER RESOURCE GROUP rg_other RU_PER_SEC = 50;
As shown in the figure above, you can see that there is no limit on the initial resource quota of the tenant app_other's business, and the maximum resources required can be used, and the business is running stably. At this time, we temporarily increase the limit and do not allow the limit to be exceeded. After the cluster receives the configuration, its RU curve continues to decrease until it returns to the original limit state.
● Summary:
- ○ Explain that TiDB’s resource management and control technology can adjust business resource usage status online, add hard upper limits in real time, and do not allow businesses to exceed the limits.
● Summary:
Let’s sort out the above simulation operations, as shown in the figure below. After testing and verification, it has been proven that TiDB’s resource management and control technology can adjust business resource usage status online, allowing TiDB administrators to expand, reduce, and cancel limits in real time based on business operation dynamics. Adding a hard upper limit does not allow operations such as business exceeding the limit. It is very flexible and convenient, greatly reducing the difficulty of operation and maintenance, and also greatly improving the resource usage efficiency of the cluster.
Cross-business coexistence testing
By adjusting the stress test QPS of the tenant app_oltp business, we generate the peaks and troughs of the tenant app_oltp business. Here we simulate the integration and coexistence of different businesses in the same cluster. All businesses experience a complete cycle of peaks and troughs of the most important business, and observe the operation conditions. The process is as follows:
● First, three types of loads run at the same time, indicating normal business coexistence.
● The business traffic peak is approaching, and tenant app_oltp reaches peak load.
● The peak value of tenant app_oltp has passed and returned to normal state.
● The load of tenant app_oltp reaches the lowest value, while other things remain unchanged.
● Tenant app_oltp The trough has passed and returned to normal state
As you can see in the figure above, at the beginning, several services in the cluster coexisted normally, and three types of loads were running at the same time.
● When the tenant app_oltp reaches its peak load and its business traffic peak comes, the system will allocate more resources to it. At the same time, due to insufficient cluster available resources, the RU allocated to the tenant app_olap will be reduced. When the peak load of the tenant app_oltp passes, the tenant app_olap will be allocated The obtained RU is increased and returns to the original state.
● After a period of time, the tenant app_oltp reaches the business valley of its running business and the RU it requires decreases. At this time, the number of idle RUs in the cluster increases. Since the tenant app_olap is set to BURSTABLE, which allows the use of resources beyond the limit, the available RUs of the tenant app_olap increase. , when the trough value of tenant app_oltp passes, the RU allocated by tenant app_olap decreases and returns to the original state.
● Since the tenant app_other has always had a quota limit and requires fewer RUs, it is stably maintained at a low level and does not affect other business operations.
In the previous process, we looked at the problem from the perspective of cluster resource usage. Now we look at it from the perspective of business QPS. As shown in the figure above, the running QPS of different businesses basically increases with the increase of available resources. The service business is expected to decline due to the decrease in resources. It can be concluded that by using the resource management and scheduling capabilities provided by TiDB, multiple tenants with different demands can coexist in one system, improving resource usage efficiency on the basis of ensuring their respective service goals.
Summarize
We verified the resource adjustment for a single online business and simulated the operation of each business during the complete peak and trough operation cycle of important businesses. The test data and results of each key point were in line with our expectations, proving that the resource Feasibility of control technology. It also states:
● TiDB's resource management and control technology can dynamically track business load conditions and allocate required resources in real time, proving that its operation has good real-time performance.
● When there are multiple service loads in the system, resource isolation can be achieved to ensure that important services are not affected by other access.
● When the system load is low, busy applications are allowed to exceed the set quota, which can maximize resource utilization, improve hardware utilization, and reduce operating costs.
Cross-business system multi-tenant solution
Based on how we use TiDB online, we can develop a preliminary cross-business system multi-tenant solution. The deployment architecture of other business systems requires detailed analysis of specific situations.
TiDB multi-tenant technology is used here to enable multiple business systems to use a unified cluster to ensure that different business loads are isolated from each other, do not interfere with each other, and do not affect each other. Then, for statistical analysis needs, the real-time HTAP capability of TiFlash can also be used. To realize cross-business data correlation query, this part of the capability is also physically isolated through TiFLash and TiKV, and will not affect the TP business running online. The architecture diagram of this solution is roughly as follows:
plan description
● Set up different resource groups and RUs according to different services. When the overall resources of the cluster are busy, different services can implement RU-based current limiting and load isolation;
● Set the BURSTABLE attribute of the resource group for important businesses to realize cross-business off-peak resource borrowing;
● Set the priority of important services to HIGH to ensure that the cluster gives priority to ensuring that important business resources are available;
●Introducing TiFlash to solve real-time data analysis needs;
● If necessary for the business, tidb-sever can also be divided into different business nodes to truly achieve resource isolation of the entire cluster.
Solution summary
● Advantages
○ By controlling applications, sessions, and SQL into corresponding resource groups, high-priority services can be satisfied first, and the remaining computing power can be used to satisfy sub-optimal services to achieve full utilization of resources.
○ The system has strong scalability. When the system load is low, even if the busy application exceeds the set RU, it can still obtain the required system resources, thus improving the scalability of the system.
○ Different services can be mixed and deployed on the same cluster to reduce hardware costs
○ Different businesses use resources at staggered peaks to improve overall resource utilization and reduce operating costs.
○ Save hardware costs
○ Highly scalable
○ Flexible management and control of resources
○ Solve the problem of data islands
● Disadvantages
○ It is difficult to determine the resource division strategy. First estimate the allocation based on the hardware situation, calibrate the load after running for a period of time, and then intervene in adjustments. This requires high skills and experience from operation and maintenance personnel, and is prone to errors.
○ The complexity of the cluster system becomes higher, and the cluster resource pool needs to be divided and managed manually, which increases the complexity of the system and makes maintenance more difficult.
○ High system complexity
○ Resource allocation strategy is difficult to formulate
future outlook
● During the test and verification, the author found that how to divide resources is a difficult problem. Calibrating the estimated capacity of RU through hardware configuration is not accurate. The real capacity often deviates greatly, so we need to give a larger resource quota first and observe for a while. After a while, the real RU consumption can be obtained through load calibration and then set to the correct value. If this can be made more intelligent and automated, it may be more perfect to reduce manual intervention. We look forward to official follow-up optimization.
● Currently, the resources included in RU are CPU, disk IO and network IO. It does not yet support the management and control of memory resources. It is expected that the official management and control of memory usage will be added in the future.
● After adjusting the resource group configuration, it will only take effect for new sessions created by users. Many of our online businesses have long connections, which will cause it to fail to take effect. We hope that the official can also optimize this aspect in the future.