Whether pt-archiver archive data source and target libraries appear inconsistent

Others 2019-09-30 06:02:46 views: null

background

  • Archive table must exist in the source and target libraries
  • Scene pt-archiver archive table are: do not delete the original table data, non-bulk insert target libraries; do not delete the original table data, bulk insert target libraries; non-bulk delete the original table data, non-bulk insert target libraries; batch delete the original table data , bulk insert target library

Version
pt-Archiver --version
pt-Archiver 3.0.12

select @@version;
+-----------+
| @@version |
+-----------+
| 8.0.12 |
+-----------+

Whether there will be inconsistencies

  • Source library has been delete, insert target database yet
  • Target library already insert, delete source database yet

--bulk-insert
using LOAD DATA INFILE manner, compared to the insertion line by line, to create temporary files for the batch of data, the data is written to a temporary file first, when the number of data acquisition is completed, then the import operation, the acceleration the target library insertion speed

--bulk-delete
batch delete, a number of rows of data using a DELETE statement is completed

Generate 100,000 records

sysbench /usr/local/share/^Csbench/oltp_read_write.lua --mysql_storage_engine=innodb --table-size=100000 --tables=1 --mysql-db=test_archiver --mysql-user=admin --mysql-password=admin --mysql-port=8013 --mysql-host=127.0.0.1 --threads=8 --time=10 --report-interval=1 --events=0 --db-driver=mysql prepare

Whether the source and target libraries will be tested in different instances inconsistent

Source Library
192.168.137.133:test_archiver

Target library
192.168.137.1:test_archiver

Open log gerneral
set global general_log = on;

Per 5000 records a commit, each taking process records 10000

nohup pt-archiver --source h=127.0.0.1,u=admin,p=admin,P=8013,D=test_archiver,t=sbtest1 --dest h=192.168.137.1,u=admin,p=admin,P=3306,D=test_archiver --progress 1000 --where "id<100000" --statistics --limit 10000 --sleep 10 --no-check-charset --txn-size 5000 --bulk-delete --bulk-insert &

Midway kill off pt-archiver archiving process, the source and target libraries does not appear inconsistent
ps -ef | grep pt-archiver | awk '{print $ 2}' | xargs kill -9

Target library

select id from sbtest1 order by  id  desc limit 1;
+-------+
| id    |
+-------+
| 10000 |
+-------+
1 row in set (0.00 sec)

Source Library

select id from sbtest1 order by id   limit 1;
+-------+
| id    |
+-------+
| 10001 |
+-------+
1 row in set (0.00 sec)

Source library to perform statement

2019-08-21T07:02:58.600832Z        56 Connect   [email protected] on test_archiver using TCP/IP
2019-08-21T07:02:58.601186Z        56 Query     set autocommit=0
...
2019-08-21T07:02:58.966036Z        56 Query     SELECT MAX(`id`) FROM `test_archiver`.`sbtest1`
2019-08-21T07:02:58.967807Z        56 Query     SELECT CONCAT(@@hostname, @@port)
2019-08-21T07:02:58.989394Z        56 Query     SELECT /*!40001 SQL_NO_CACHE */ `id`,`k`,`c`,`pad` FROM `test_archiver`.`sbtest1` FORCE INDEX(`PRIMARY`) WHERE (id<100000) AND (`id` < '100000') ORDER BY `id` LIMIT 10000
...
2019-08-21T07:02:59.275620Z        56 Query     commit
...
019-08-21T07:02:59.532682Z        56 Query     commit
2019-08-21T07:02:59.834194Z        56 Query     SELECT 'pt-archiver keepalive'
2019-08-21T07:02:59.834835Z        56 Query     DELETE FROM `test_archiver`.`sbtest1` WHERE (((`id` >= '1'))) AND (((`id` <= '10000'))) AND (id<100000) LIMIT 10000
2019-08-21T07:03:09.958289Z        56 Query     SELECT /*!40001 SQL_NO_CACHE */ `id`,`k`,`c`,`pad` FROM `test_archiver`.`sbtest1` FORCE INDEX(`PRIMARY`) WHERE (id<100000) AND (`id` < '100000') AND ((`id` >= '10000')) ORDER BY `id` LIMIT 10000
...
2019-08-21T07:03:10.215958Z        56 Query     commit
...
2019-08-21T07:03:10.670937Z        56 Query     commit
2019-08-21T07:03:10.904398Z        56 Query     SELECT 'pt-archiver keepalive'
2019-08-21T07:03:10.904715Z        56 Query     DELETE FROM `test_archiver`.`sbtest1` WHERE (((`id` >= '10001'))) AND (((`id` <= '20000'))) AND (id<100000) LIMIT 10000  ====》( 该语句由于没有commit 语句会rollback )

Target libraries execute the statement

2019-08-21T07:03:00.317343Z        33 Connect   [email protected] on test_archiver using TCP/IP
2019-08-21T07:03:00.338390Z        33 Query     set autocommit=0
...
2019-08-21T07:03:00.633938Z        33 Query     SELECT CONCAT(@@hostname, @@port)
2019-08-21T07:03:00.920655Z        33 Query     commit
2019-08-21T07:03:01.177267Z        33 Query     commit
2019-08-21T07:03:01.199046Z        33 Query     LOAD DATA LOCAL INFILE '/tmp/jaGuzZfjSept-archiver' INTO TABLE `test_archiver`.`sbtest1`(`id`,`k`,`c`,`pad`) （插入了 1=<id <=10000的记录）
2019-08-21T07:03:11.850618Z        33 Query     commit
2019-08-21T07:03:12.315829Z        33 Query     commit
2019-08-21T07:03:12.337323Z        33 Query     LOAD DATA LOCAL INFILE '/tmp/GQ2ybc3KCzpt-archiver' INTO TABLE `test_archiver`.`sbtest1`(`id`,`k`,`c`,`pad`)  ====》( 该语句由于没有commit 该语句会rollback ，并在 机器/tmp 目录下留下临时文件)

ll /tmp/GQ2ybc3KCzpt-archiver
-rw------- 1 root root 1920000 Aug 21 15:03 /tmp/GQ2ybc3KCzpt-archiver
  • Seen from the log, commit time delete operation source libraries (07: 03: 10.215958Z) is at commit time target library insert operation before (07: 11.850618Z: 03), which may occur when the archive source libraries delete, target the case has not yet insert the library
  • The source and target libraries on different instances, different examples of inconsistency time clock will affect general_log appear in the commit occurs

Whether the source and target libraries appear inconsistent test in the same instance

Source Library
192.168.137.133:test_archiver

Target library
192.168.137.133:test_archiver2

Delete the test data to regenerate 100,000 records

sysbench /usr/local/share/sysbench/oltp_read_write.lua --mysql_storage_engine=innodb --table-size=100000 --tables=1 --mysql-db=test_archiver --mysql-user=admin --mysql-password=admin --mysql-port=8013 --mysql-host=127.0.0.1 --threads=8 --time=10 --report-interval=1 --events=0 --db-driver=mysql cleanup

sysbench /usr/local/share/sysbench/oltp_read_write.lua --mysql_storage_engine=innodb --table-size=100000 --tables=1 --mysql-db=test_archiver --mysql-user=admin --mysql-password=admin --mysql-port=8013 --mysql-host=127.0.0.1 --threads=8 --time=10 --report-interval=1 --events=0 --db-driver=mysql prepare

Commit records for every 100,000 once each taking 100,000 records processed
pt-archiver --source h = 127.0.0.1, u = admin, p = admin, P = 8013, D = test_archiver, t = sbtest1 --dest h = 127.0.0.1, u = admin, p = admin, P = 8013, D = test_archiver2 --progress 1000 --where "id <100000" --statistics --sleep 10 --limit 100000 --no-check- charset --txn-size 100000 --bulk-delete --bulk-insert

Source and target libraries execute the statement


2019-08-22T01:50:35.672490Z         9 Connect   [email protected] on test_archiver using TCP/IP
2019-08-22T01:50:35.673125Z         9 Query     set autocommit=0
...
2019-08-22T01:50:35.685987Z        10 Connect   [email protected] on test_archiver2 using TCP/IP
2019-08-22T01:50:35.686278Z        10 Query     set autocommit=0
...
2019-08-22T01:50:35.708866Z         9 Query     SELECT /*!40001 SQL_NO_CACHE */ `id`,`k`,`c`,`pad` FROM `test_archiver`.`sbtest1` FORCE INDEX(`PRIMARY`) WHERE (id<100000) AND (`id` < '100000') ORDER BY `id` LIMIT 100000
...
2019-08-22T01:50:40.242371Z        10 Query     LOAD DATA LOCAL INFILE '/tmp/X5W2UemPgDpt-archiver' INTO TABLE `test_archiver2`.`sbtest1`(`id`,`k`,`c`,`pad`)
2019-08-22T01:50:43.692914Z         9 Query     SELECT 'pt-archiver keepalive'
2019-08-22T01:50:43.693411Z         9 Query     DELETE FROM `test_archiver`.`sbtest1` WHERE (((`id` >= '1'))) AND (((`id` <= '99999'))) AND (id<100000) LIMIT 100000
2019-08-22T01:50:58.603351Z         9 Query     SELECT /*!40001 SQL_NO_CACHE */ `id`,`k`,`c`,`pad` FROM `test_archiver`.`sbtest1` FORCE INDEX(`PRIMARY`) WHERE (id<100000) AND (`id` < '100000') AND ((`id` >= '99999')) ORDER BY `id` LIMIT 100000
2019-08-22T01:50:58.606390Z        10 Query     commit
2019-08-22T01:50:58.717251Z         9 Query     commit
2019-08-22T01:50:58.780614Z        10 Quit
2019-08-22T01:50:58.781480Z         9 Quit
  • From general log looks, the bulk of the target library is inserted before the bulk source libraries delete, commit the target library insert operation (01: 50: 58.606390Z) also commit in the source database delete operations (01: 50: 58.717251Z) prior to
  • *** In 0.11s during the commit object libraries execution, pt-archiver abnormal termination (this probability is very small # _ #), the source library commit not execute, delete operation will be rolled back, the emergence of the source repository inconsistent data and object libraries issue ***

Note
MySQL8.0 perform load data infile command set secure_file_priv addition, the need to set local-infile = 1 in the [client] and [mysqld] in, otherwise there will be an error
DBD :: mysql :: st execute failed: The used command is not allowed with this MySQL version

pt-archiver commit

  • Operation of the associated code is visible in the target library commit operation is completed, the source library was ready to commit a
  • When the transaction in the operation of large amounts of data, delete the source database commit time-consuming operation will be relatively long, after pt-archiver abnormal termination (commit source database is not yet complete, delete operation will be rolled back), there will be goals data already exists library, the library has not been deleted source data inconsistencies
 7068       if ( $dst ) {
   7069          trace('commit', sub {
   7070             $dst->{dbh}->commit;
   7071          });
   7072       }
   7073       trace('commit', sub {
   7074          $src->{dbh}->commit;
   7075       });
   7076       $txn_cnt = 0;
   7077    }
   7078 }

in conclusion

  • During pt-archiver archiving non-commit, pt-archiver abnormal termination, the source and target libraries will rollback, does not appear inconsistencies
  • pt-archiver abnormally terminated at the time of commit, inconsistencies may occur: The target library already insert, delete source library also no case
  • After pt-archiver abnormal termination (not finished on time and archived manually kill pt processes, etc.), the need to manually check the source and target database library primary key, otherwise the archive will be the primary key conflict error appears again

Use pt-archiver archiving tool Detailed

Archiver-pt
pt-Archiver data archiving

Guess you like

Origin www.cnblogs.com/YangJiaXin/p/11610217.html
Recommended
LFOSSA Yuanlaisusu Open Course | Mastering the Cloud Native Future: Comprehensive Guide to CNCF Certification and Exam Preparation Tips
Ranking
C++ Basic Syntax
bootstrapTable hides a column based on a condition
Why is reentrant lock recommended instead of Synchronized when dynamic high concurrency?
hexo create a blog
[Fully open source and non-encrypted version] Imitation of the eighth district distribution/online signature/multiple sets of download templates/APP distribution hosting/APP packaging and packaging
Polymerization combination
https://www.flysnow.org/2017/05/06/go-in-action-go-log.html
From the perspective of Flutter and the front-end, talk about how to ensure UI fluency under the single-threaded model
nginx-301, 302 redirect
Geolocation by IP Address in ASP.NET
Daily
More
2024-04-26(22)
2024-04-25(32)
2024-04-24(30)
2024-04-23(30)
2024-04-22(5)
2024-04-21(0)
2024-04-20(6)
2024-04-19(5)
2024-04-18(0)
2024-04-17(31)

Code World

© 2018 codetd.com