I have a problem today
Scenario : Data is synchronized across regions, and after it is written to the target database at the target end, the data is extracted from the target database for signature, and reported to the relay node for synchronization verification.
Phenomenon : However, in some cases, there is a problem with the data retrieved from the target database. The synchronous data of the INSERT type is empty, and the synchronous data of the UPDATE type is taken out with the old value.
Positioning: The landing link of data is a synchronous process and not asynchronous, so there is no possibility of reading first and then writing; the data is fetched only after the data has been written into the library.
Then, there is no problem with the upper layer logic, only the possibility of the database is left; because the dynamic data source is used to realize the separation of read and write, and the target library adopts the master-slave mode, then there is only one truth ----> The data is in After writing to the main library, the business logic is processed too fast, and the data of the slave library has not been synchronized to the main library. The business logic is read once, thus reading the old data!
Impact: Since the latest data on the source side is checked, and the consumption of the target side is guaranteed to be orderly, the data can still be guaranteed to be eventually consistent when updating data concurrently; but because the relay node finds that the signature value is inconsistent, it will go to the source side. Query metadata and retransmit it again, which consumes more traffic
Solution : I haven't thought of a better solution for the time being, so I'll record it first, and look forward to the boss's confusion
- After the data is written to the target library, sleep and read it again. It will inevitably lead to a decrease in the consumption speed, and the data synchronization speed will also slow down.
- Read the data to sign and report the action independently, and buffer it in an asynchronous way. It is still passively aware of the time difference between the slave database and the master database data synchronization.