Hot data discovery
1. Discover static hot data: The discovery of static hot data is relatively simple, and it is data that can be predicted and predicted in advance. For example: spike event merchandise, discounted promotional merchandise, train tickets on holidays, popular movie tickets, star releases new albums, and big data analysis of popular trend prediction hot spots.
2. Discover dynamic hot data: Create asynchronous monitoring statistics services and hot data services. The asynchronous monitoring statistics service performs statistics on Key requests in one cycle. After reaching the requested level, the hotspot Key is sent to the hotspot data service for collection, and then the hotspot data service aggregates and analyzes these hotspot Keys, and finally pushes them to the business system.
Hot data processing
1. Cache hot data and put it into the LRU queue to eliminate and replace it.
2. Limit and limit the flow of requests, add verification and anti-scratch mechanisms, and limit the flow of a single Key.
3. Fuse downgrade processing, return a fixed value.
4. Hotspot data isolation to prevent other businesses from being affected.
Hot data update
Hot data writing to the database will cause a large number of threads to compete for locks, which will seriously affect the response time of the database.
1. Put the request into the message queue, and send the update request to the database for execution in sequence.
2. Optimize the database storage engine, and concurrently write to a single row of records. (Taobao’s database team has developed a patch on the InnoDB layer for MySQL, which can achieve concurrent queuing of single-row records on the database layer. Deadlock detection inside InnoDB and the switch between MySQL Server and InnoDB will consume more performance. Taobao’s The MySQL core team has also made many other optimizations, such as the patches of COMMIT_ON_SUCCESS and ROLLBACK_ON_FAIL, with the addition of hints in SQL. In the transaction, there is no need to wait for the application layer to submit COMMIT. After the data is executed, the last SQL is submitted directly according to the TARGET_AFFECT_ROW result. Or rollback, can reduce the waiting time of the network)
3. Limit SQL flow at the database level and refuse to execute thread_running when it reaches the threshold.
4. Modify the configuration of the number of concurrent threads in the database innodb_thread_concurrency to limit the number of concurrent threads, which will cause a large number of connection waits.
5. The dynamic migration of hot data to a separate database is difficult to implement.
6. Update the hotspot data in the cache and synchronize it to the database regularly. There will be data inconsistencies within a certain period of time.