I. Introduction
System Features
The customer service system has a work order query function. The work order table features and user requirements are as follows:
1) Tens of millions of data are stored in the work order table.
2) When querying the work order table data, more than a dozen sub-tables need to be associated, and the data of each sub-table is over 100 million.
3) Some data in the work order table is a few years ago, and customers need these data to continue to be updated.
Problem Description
Every time a customer queries data, it can take tens of seconds or even longer to return results.
Solution selection
Method 1: Use database optimization techniques such as indexes and SQL to solve the problem. However, due to the huge amount of data and many associated sub-tables, the optimization effect is not obvious.
Method 2: Use cold and heat separation, but the customer demand restrictions in Article 3), pass the program directly.
Is there any solution to solve the slow query problem of the above-mentioned characteristic system?
The answer is yes, this is the query separation we are going to introduce in this article .
2. What is query separation?
Query separation means: each time data is written, in addition to writing the data to the main database, another copy of the data is saved to another storage system, and the user directly obtains the data from the other storage system when querying the data.
3. In which scenarios are query separation used?
I believe that in the first chapter, we have a preliminary understanding of the use scenarios of query separation.
Below we summarize the situations where query separation should be used in actual business:
- Big amount of data;
- The efficiency of all write data requests is acceptable;
- The efficiency of query data request is very low;
- All data may be modified at any time;
- The business wants us to optimize the function of querying data.
Four, query separation realization ideas
4.1 How to trigger query separation?
How to trigger query separation can be simply understood as: when should a copy of data be saved to the query data.
Generally speaking, there are two ways to separate and trigger the query: business layer code implementation and binlog implementation . The business layer code implementation can be subdivided into two (synchronous and asynchronous) implementation methods.
(1) Business layer code implementation: After writing regular data, query data is created synchronously.
Applicable scenarios: The business code is relatively simple and does not require high response speed for write operations.
(2) Business layer code implementation: After writing regular data, query data is created asynchronously.
Usage scenario: The business code is relatively simple and requires the response speed of write operations. (In real business scenarios, this method is used more often)
(3) The binlog of the monitoring database log is implemented: if there is a data change, the query data will be updated.
Usage scenario: The business code is more complicated, or the change is too costly.
Comparison of the advantages and disadvantages of the three triggering methods.
Several concepts in the table explain:
What is flexible and controllable business logic? Generally speaking, the person who writes business code can quickly judge from the business logic under what circumstances to update the query data, while the person who monitors the database log cannot exhaust all the database change branches, and then associate all the possibilities to In the corresponding update query data logic, any data changes that ultimately result in the need to re-establish the query data.
What is slowing down the write operation speed? How much write operation speed can be slowed by an action of establishing query data? Answer: a lot.
Take a chestnut: When you simply update an order's logo, it only takes 2ms to query the data, and it may involve reconstruction when querying the data (such as indexing, sharding, master-slave backup when using ES to query data, Each action is subdivided into many sub-actions, which we will talk about later). At this time, the process of establishing query data may take 1s, from 2ms to 1s. Do you think the slowdown is large?
Before the query data is updated, users may query outdated data. Here we combine the second trigger logic, for example, an operation is in the order update state, and the data will be queried asynchronously when the state is updated. After the update, the order will change from the "pending review" state to the "reviewed" state. Assuming that the update time of the query data takes 1 second, if the user is querying the order status during this 1 second, although the master data has changed to the "reviewed" status, the final query result still shows the "pending review" status.
4.2 How to achieve query separation?
This is only for the business layer code implementation: after writing regular data, asynchronously establish the query data query separation method to explain. There are two ways to briefly summarize:
(1) Method 1 : Start a separate thread to establish query data
This implementation is relatively simple, but the following problems may occur in actual applications:
- There are too many write operations and too many threads, which will eventually burst the JVM;
- How to automatically retry the thread that created the query data;
- When multi-threaded concurrency, many concurrency scenarios need to be solved.
From the above list of questions, we know that although this method is simple, there are many problems. Therefore, this method is not recommended.
(B) Embodiment 2 : MQ management borrow
specific operation thinking MQ master write operation for each data processing request, will send a notice to MQ, MQ notified after a wake-up thread updates the data query, the following schematic diagram:
understand After the specific operation ideas of MQ, we should also consider the following 5 major issues:
(1) Question 1: How to select MQ?
There is no absolute reference condition for how to select MQ. You can select according to your actual situation.
For example, if the company has already used MQ and has been well applied in the project, then you can choose the MQ that you are familiar with and has been tested in actual combat. After all, the polished technology can avoid pits and detours.
If the company has not yet used MQ and is not familiar with the advantages and disadvantages of various types of MQ and usage scenarios, then the selection of MQ is necessary.
In general, the selection principles are as follows for reference:
1) According to business needs and characteristics, choose several more suitable MQs.
2) Convene all those who can make technical decisions in the technology center to vote for the selection.
3) Consider from the perspective of ease of use and code workload.
(2) Question 2: What should I do if MQ is down?
If you encounter an MQ downtime, we need to ensure that the main process is proceeding normally, and that the workflow cannot be stalled due to the MQ downtime; at the same time, ensure that the data can be processed normally after the MQ is restored.
specific plan:
- In each write operation, add a mark to the main data: NeedUpdateQueryData=true, so that the message sent to MQ is very simple, just a simple signal to update the data, and does not contain the updated data id.
- After receiving the signal, MQ consumers first batch query the master data to be updated, and then batch update the query data. After the update, the master data identifier NeedUpdateQueryData of the query data is updated to false.
- Of course, there are also cases where multiple consumers carry actions at the same time, which involves the issue of concurrency, which is similar to the concurrency processing logic in the separation of cold and hot.
(3) Question 3: What should I do if the thread updating the query data fails?
If the update thread fails, the ID of NeedUpdateQueryData will not be updated, and subsequent consumers will again take out the data identified by NeedUpdateQueryData for processing. But if it keeps failing, we can add an additional number of handling attempts to the main data, such as +1 every time a handling attempt is made, and clear it after success, so as to monitor data that has too many attempts to move.
(4) Question 4: Idempotent consumption of messages
In programming, an idempotent operation is characterized by the same effect of executing an operation multiple times.
For example, after order A of the master data is updated, we insert A into the query data, but at this time the system has a problem. The system mistakenly believes that the query data has not been updated, and then inserts and updates order A again.
The so-called idempotence means that no matter how many times the logic of updating the query data is executed, the result is the result we want. Therefore, when considering the issue of concurrency on the consumer side, we need to ensure that the update query data is idempotent.
(5) Question 5: The timing of the message. For
example, an order A updates the data once to become A1, and thread A moves the data of A1 to the query data. After a while, the back-end order A updates the data once more and becomes A2, and thread B also starts work, moving the data of A2 to the query data.
The so-called timing is that if thread A starts earlier than thread B, but the movement of moving data is completed later than thread B, it is possible that the query data will eventually become expired A1. As shown in the figure below (the sequence number in front of the action represents the order of the actual action):
Solution:
update the last update time last_update_time for each update, and then after each thread updates the query data, check whether the last_update_time of the current order A is the same as the thread just started The time is the same, and if NeedUpdateQueryData is equal to false, if all are satisfied, we will change NeedUpdateQueryData to true, and then do another move.
4.3 How to store query data?
The choice of data storage technology requires comprehensive consideration in combination with actual business needs and its own organizational structure.
At present, Elasticsearch is widely used for search queries with large amounts of data.
4.4 How to use query data?
If in the previous section, we chose Elasticsearch to implement search queries with a large amount of data, then when querying data, we can directly call ES API functions in the code.
This method may have the problem of "inconsistent query data before the data query is updated". At this time, we can refer to the following two solutions to solve:
1) Idea 1: Before the query data is updated to the latest, users are not allowed to query.
2) Idea 2: To remind users that the currently queried data may be the data 1 second ago. If the data is found to be inaccurate, you can try to refresh.
5. The overall plan and shortcomings of query separation
5.1 The overall solution of query separation
5.2 The shortcomings of the
query separation solution Although the query separation solution can solve some problems, we must also clearly realize its shortcomings.
1) When using Elasticsearch to store and query data, some things to be aware of should be clear.
2) After the main data volume becomes larger and larger, the write operation is still slow, and problems will still occur at that time.
3) When the master data and the query data are inconsistent, assuming that the business logic needs to keep the query data consistent, it is also a problem that we need to consider.