We use MySQL as the main data storage, use MySQL's transaction characteristics to maintain data consistency, and use ElasticSearch for data collection and query. At this time, the synchronization scheme between ES and the database is particularly important.
To ensure the synchronization between ES and the database:
1. First add the product to the database. After the product is added successfully, the product is put into the ES. If the entry to the ES fails, the failed product ID is put into the redis cache queue, and the failed product ID is entered into the log file. (If redis hangs, you can get the abnormal product ID from the log and then enter the ES), the
task task refreshes the redis cache queue every second, if the product ID is obtained from the cache queue, it will obtain the product from the database according to the product ID. The data then goes into ES.
Develop es and database synchronization gadgets:
1. Full import
2. Batch import according to ID
To ensure the synchronization between ES and the database:
1. First add the product to the database. After the product is added successfully, the product is put into the ES. If the entry to the ES fails, the failed product ID is put into the redis cache queue, and the failed product ID is entered into the log file. (If redis hangs, you can get the abnormal product ID from the log and then enter the ES), the
task task refreshes the redis cache queue every second, if the product ID is obtained from the cache queue, it will obtain the product from the database according to the product ID. The data then goes into ES.
Develop es and database synchronization gadgets:
1. Full import
2. Batch import according to ID