Table of contents
redis-migrate-tool installation
Migrate using redis-migrate-tool
rmt_redis.c:6446 ERROR: Can't handle RDB format version error solution
Detailed explanation of redis-migrate-tool command
Detailed explanation of rmt.conf
redis cluster to redis cluster (abc prefix filtering)
View redis-migrate-tool status
Info command response introduction:
Check migration results (redis_check)
Insert some test keys (redis_testinsert)
redis-migrate-tool
Introduction
redis-migrate-tool is a convenient and useful tool for data migration between redis.
git address
github:
https://github.com/vipshop/redis-migrate-tool
gitee:
https://gitee.com/vipshop/redis-migrate-tool.gitfeatures
- fast.
- Multithreading.
- Replication based on redis.
- live migration.
- During the migration process, the source cluster does not affect external services.
- heterogeneous migration.
- Support Twemproxy cluster, redis cluster cluster, rdb file and aof file.
- filter function.
- When the target cluster is Twemproxy, the data will skip Twemproxy and import directly to the backend redis.
- The migration status is displayed.
- Perfect data sampling verification.
redis-migrate-tool installation
git clone https://github.com/vipshop/redis-migrate-tool.git
cd redis-migrate-tool
yum -y install automake libtool autoconf bzip2
autoreconf -fvi
./configure
make
./src/redis-migrate-tool
If the ./src/redis-migrate-tool command can output content normally, the installation is successful
Migrate using redis-migrate-tool
Suppose now there are two clusters as follows:
# 拥有一个key。 700key1: 700value1
127.0.0.1:7001 master
127.0.0.1:7002 master
127.0.0.1:7003 master
127.0.0.1:7004 slave
127.0.0.1:7005 slave
127.0.0.1:7006 slave
# 拥有一个key。 800key1: 800value1
127.0.0.1:8001 master
127.0.0.1:8002 master
127.0.0.1:8003 master
127.0.0.1:8004 slave
127.0.0.1:8005 slave
127.0.0.1:8006 slave
You can use the create-local-redis-cluster-easily project to quickly build a redis cluster on a local virtual machine.
Now it is necessary to migrate the data of the 7001-7006 cluster to the 8001-8006 cluster.
First, modify the rmt.conf file to:
[source]
type: redis cluster # 源redis类型
servers :
- 127.0.0.1:7001 # 源redis地址，可以只填一个，保证cluster nodes能获取到正常信息即可
[target]
type: redis cluster # 目标redis类型
servers:
- 127.0.0.1:8001 # 目标redis地址
[common]
listen: 0.0.0.0:8888 # 服务启动后，监听的地址和端口
Start the migration tool:
./src/redis-migrate-tool -c rmt.conf
Output these to indicate a successful migration:
[root@localhost redis-migrate-tool]# ./src/redis-migrate-tool -c rmt.conf
[2019-10-18 05:05:32.523] rmt_core.c:525 Nodes count of source group : 3
[2019-10-18 05:05:32.523] rmt_core.c:526 Total threads count : 2
[2019-10-18 05:05:32.523] rmt_core.c:527 Read threads count assigned: 1
[2019-10-18 05:05:32.523] rmt_core.c:528 Write threads count assigned: 1
[2019-10-18 05:05:32.524] rmt_core.c:836 instances_by_host:
[2019-10-18 05:05:32.524] rmt_core.c:840 127.0.0.1:7002@17002
[2019-10-18 05:05:32.524] rmt_core.c:840 127.0.0.1:7001@17001
[2019-10-18 05:05:32.524] rmt_core.c:840 127.0.0.1:7003@17003
[2019-10-18 05:05:32.524] rmt_core.c:842
[2019-10-18 05:05:32.524] rmt_core.c:2444 Total threads count in fact: 2
[2019-10-18 05:05:32.524] rmt_core.c:2445 Read threads count in fact: 1
[2019-10-18 05:05:32.524] rmt_core.c:2446 Write threads count in fact: 1
[2019-10-18 05:05:32.524] rmt_core.c:2455 read thread(0):
[2019-10-18 05:05:32.524] rmt_core.c:2461 127.0.0.1:7002@17002
[2019-10-18 05:05:32.524] rmt_core.c:2461 127.0.0.1:7001@17001
[2019-10-18 05:05:32.524] rmt_core.c:2461 127.0.0.1:7003@17003
[2019-10-18 05:05:32.524] rmt_core.c:2488 write thread(0):
[2019-10-18 05:05:32.524] rmt_core.c:2494 127.0.0.1:7002@17002
[2019-10-18 05:05:32.524] rmt_core.c:2494 127.0.0.1:7001@17001
[2019-10-18 05:05:32.524] rmt_core.c:2494 127.0.0.1:7003@17003
[2019-10-18 05:05:32.524] rmt_core.c:2551 migrate job is running...
[2019-10-18 05:05:32.525] rmt_redis.c:1706 Start connecting to MASTER[127.0.0.1:7002@17002].
[2019-10-18 05:05:32.525] rmt_redis.c:1740 Master[127.0.0.1:7002@17002] replied to PING, replication can continue...
[2019-10-18 05:05:32.525] rmt_redis.c:1051 Partial resynchronization for MASTER[127.0.0.1:7002@17002] not possible (no cached master).
[2019-10-18 05:05:32.526] rmt_redis.c:1110 Full resync from MASTER[127.0.0.1:7002@17002]: f00d5c7260ae3652fe82b8c76b57b25e2ada8ed5:11300
[2019-10-18 05:05:32.593] rmt_redis.c:1517 MASTER <-> SLAVE sync: receiving 200 bytes from master[127.0.0.1:7002@17002]
[2019-10-18 05:05:32.593] rmt_redis.c:1623 MASTER <-> SLAVE sync: RDB data for node[127.0.0.1:7002@17002] is received, used: 0 s
[2019-10-18 05:05:32.593] rmt_redis.c:1643 rdb file node127.0.0.1:[email protected] write complete
[2019-10-18 05:05:32.594] rmt_redis.c:6601 Rdb file for node[127.0.0.1:7002@17002] parsed finished, use: 0 s.
[2019-10-18 05:05:32.594] rmt_redis.c:1706 Start connecting to MASTER[127.0.0.1:7001@17001].
[2019-10-18 05:05:32.595] rmt_redis.c:1740 Master[127.0.0.1:7001@17001] replied to PING, replication can continue...
[2019-10-18 05:05:32.595] rmt_redis.c:1051 Partial resynchronization for MASTER[127.0.0.1:7001@17001] not possible (no cached master).
[2019-10-18 05:05:32.596] rmt_redis.c:1110 Full resync from MASTER[127.0.0.1:7001@17001]: 56b1e3002b0382acfb526ad6bab81576526b2d62:11523
[2019-10-18 05:05:32.693] rmt_redis.c:1517 MASTER <-> SLAVE sync: receiving 238 bytes from master[127.0.0.1:7001@17001]
[2019-10-18 05:05:32.693] rmt_redis.c:1623 MASTER <-> SLAVE sync: RDB data for node[127.0.0.1:7001@17001] is received, used: 0 s
[2019-10-18 05:05:32.693] rmt_redis.c:1643 rdb file node127.0.0.1:[email protected] write complete
[2019-10-18 05:05:32.694] rmt_redis.c:6601 Rdb file for node[127.0.0.1:7001@17001] parsed finished, use: 0 s.
[2019-10-18 05:05:32.695] rmt_redis.c:1706 Start connecting to MASTER[127.0.0.1:7003@17003].
[2019-10-18 05:05:32.696] rmt_redis.c:1740 Master[127.0.0.1:7003@17003] replied to PING, replication can continue...
[2019-10-18 05:05:32.696] rmt_redis.c:1051 Partial resynchronization for MASTER[127.0.0.1:7003@17003] not possible (no cached master).
[2019-10-18 05:05:32.698] rmt_redis.c:1110 Full resync from MASTER[127.0.0.1:7003@17003]: 5ab5f6c86143bcca51d11d55158398e52e6ff8aa:11318
[2019-10-18 05:05:32.794] rmt_redis.c:1517 MASTER <-> SLAVE sync: receiving 219 bytes from master[127.0.0.1:7003@17003]
[2019-10-18 05:05:32.794] rmt_redis.c:1623 MASTER <-> SLAVE sync: RDB data for node[127.0.0.1:7003@17003] is received, used: 0 s
[2019-10-18 05:05:32.794] rmt_redis.c:1643 rdb file node127.0.0.1:[email protected] write complete
[2019-10-18 05:05:32.795] rmt_redis.c:6601 Rdb file for node[127.0.0.1:7003@17003] parsed finished, use: 0 s.
[2019-10-18 05:05:32.795] rmt_redis.c:6709 All nodes' rdb file parsed finished for this write thread(0).
At this time, the operations on the 7001 cluster will be synchronized to the 8001 cluster in real time to realize data synchronization between the clusters
rmt_redis.c:6446 ERROR: Can't handle RDB format version error solution
Refer to this article:
https://blog.csdn.net/qq_39706515/article/details/130825616?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22130825616%22%2C%22source%22%3A%22qq_39706515%22%7DDetailed explanation of redis-migrate-tool command
[root@localhost redis-migrate-tool]# ./src/redis-migrate-tool -h
This is redis-migrate-tool-0.1.0
Usage: redis-migrate-tool [-?hVdIn] [-v verbosity level] [-o output file]
[-c conf file] [-C command]
[-f source address] [-t target address]
[-p pid file] [-m mbuf size] [-r target role]
[-T thread number] [-b buffer size]
Options:
-h, --help : this help 帮助
-V, --version : show version and exit 打印版本
-d, --daemonize : run as a daemon 以守护进程的方式运行
-I, --information : print some useful information 打印一些使用信息
-n, --noreply : don't receive the target redis reply 不接收目标集群回复
-v, --verbosity=N : set logging level (default: 5, min: 0, max: 11) 设置日志级别（默认：5，最小：0，最大11）
-o, --output=S : set logging file (default: stderr) 设置日志文件（默认：stderr）
-c, --conf-file=S : set configuration file (default: rmt.conf) 设置配置文件（默认：rmt.conf）
-p, --pid-file=S : set pid file (default: off) 设置pid文件（默认：off）
-m, --mbuf-size=N : set mbuf size (default: 512) 设置mbuf的大小（默认：512）
-C, --command=S : set command to execute (default: redis_migrate) 设置要执行的命令（默认：redis_migrate）
-r, --source-role=S : set the source role (default: single, you can input: single, twemproxy or redis_cluster) 设置源redis的类型（默认：single，你可以输入：single, twemproxy 或 redis_cluster）
-R, --target-role=S : set the target role (default: single, you can input: single, twemproxy or redis_cluster) 设置目标redis的类型（默认：single，你可以输入：single, twemproxy 或 redis_cluster）
-T, --thread=N : set how many threads to run the job(default: 1) 设置运行该程序的线程数量（默认：1）
-b, --buffer=S : set buffer size to run the job (default: 1048576 byte, unit:G/M/K) 设置运行该程序的缓冲区大小（默认：1048576，单位：G/M/K）
-f, --from=S : set source redis address (default: 127.0.0.1:6379) 设置源redis地址（默认：127.0.0.1:6379）
-t, --to=S : set target redis group address (default: 127.0.0.1:6380) 设置目标redis地址（默认：127.0.0.1:6380）
-s, --step=N : set step (default: 1) 设置step级别（默认：1）
Commands:
redis_migrate : Migrate data from source group to target group.
将数据从源redis迁移到目标redis
redis_check : Compare data between source group and target group. Default compare 1000 keys. You can set a key count behind.
比较源redis和目标redis的数据。默认比较1000个key。你可以在后面设置key的数量
redis_testinsert : Just for test! Insert some string, list, set, zset and hash keys into the source redis group. Default 1000 keys. You can set key type and key count behind.
只是测试用。插入一些string, list, set, zset 和 hash 类型的key到源redis中。默认1000个key，你也可以在后面指定key类型和key的数量。
The specific meaning of option configuration is detailed in the following rmt.conf configuration file
Detailed explanation of rmt.conf
The configuration file is divided into three parts: source, target and common.
source and target
- type: Indicates the type of this group of redis. Can be these values: single, twemproxy, redis cluster, rdb file, aof file.
- server: A list of redis addresses for this group. If the type is twemproxy (a proxy for Redis), it is similar to the twemproxy configuration file. If the type is an rdb file, then this is the full path of the file.
- redis_auth: redis password
- timeout: timeout for reading and writing redis, in milliseconds. Currently only works on the source redis cluster. The default is 120000.
- hash: The name of the hash function. Only works for twemproxy type. Can be these values: one_at_a_time, md5, crc16, crc32 (crc32 implementation compatible with libmemcached), crc32a (correct crc32 implementation as per the spec), fnv1_64, fnv1a_64, fnv1_32, fnv1a_32, hsieh, murmur 、jenkins
- hash_tag: Use two characters as part of the key to make the key have a specific hash value. For example "{}" or "$$". Hash tag can map different keys to the same node as long as they have the same tag. This configuration only applies to twemproxy.
- distribution: The distribution mode of the key. Only works on twemproxy. Can be the following values: ketama, modula, random.
common
- listen: Listening address and port (name:port or ip:port). The default is 127.0.0.1:8888
- max_clients: The maximum number of clients on the listening port. Default is 100
- threads: The maximum number of threads that redis-migrate-tool can use. The default is the number of cpu cores.
- step: The level at which to parse the request. The larger the allocation, the faster the migration and consumes more memory. Default is 1
- mbuf_size: The size of Mbuf. The default is 512
- noreply: boolean. Decides if you want to detect replies from the target redis. The default is false.
- source_safe: boolean. Whether to protect the memory safety of the source redis. If set to true, the tool guarantees that only one redis is generating rdb files at the same time on a machine. In addition, when you set 'source_safe:true', fewer threads may be used than you set. The default is true
- dir: working directory, used to store files (such as rdb files). The default is the current directory.
- filter: filter key, if it does not match the expression, it will not be migrated. Expression Glob-style (wildcard). Default is empty.
filter supports wildcard expressions:
- h?llo matches hello, hallo and hxllo
- h*llo matches hllo and heeeello
- h[ae]llo matches hello and hallo, not hillo
- h[^e]llo matches hallo, hbllo, ... does not match hello
- h[ab]llo matches hallo and hbllo
Special characters are escaped with "\".
rmt.conf sample
Redis requires a password:
- redis_auth: redis password
For details, refer to <<redis cluster to single>>
Single instance to twemproxy
[source]
type: single
servers:
- 127.0.0.1:6379
- 127.0.0.1:6380
- 127.0.0.1:6381
- 127.0.0.1:6382
[target]
type: twemproxy
hash: fnv1a_64
hash_tag: "{}"
distribution: ketama
servers:
- 127.0.0.1:6380:1 server1
- 127.0.0.1:6381:1 server2
- 127.0.0.1:6382:1 server3
- 127.0.0.1:6383:1 server4
[common]
listen: 0.0.0.0:8888
threads: 2
step: 1
mbuf_size: 1024
source_safe: true
twemproxy to redis cluster
[source]
type: twemproxy
hash: fnv1a_64
hash_tag: "{}"
distribution: ketama
servers:
- 127.0.0.1:6379
- 127.0.0.1:6380
- 127.0.0.1:6381
- 127.0.0.1:6382
[target]
type: redis cluster
servers:
- 127.0.0.1:7379
[common]
listen: 0.0.0.0:8888
step: 1
mbuf_size: 512
redis cluster to single
[source]
type: redis cluster
servers:
- 172.25.2.118:6380
- 172.25.2.118:6381
- 172.25.2.118:6382
- 172.25.2.118:6383
- 172.25.2.118:6384
- 172.25.2.118:6385
redis_auth: 111
[target]
type: single
servers:
- 172.25.2.118:6379
redis_auth: 111
[common]
listen: 0.0.0.0:8888
threads: 2
mbuf_size: 512redis cluster to redis cluster (abc prefix filtering)
[source]
type: redis cluster
servers:
- 127.0.0.1:8379
[target]
type: redis cluster
servers:
- 127.0.0.1:7379
[common]
listen: 0.0.0.0:8888
filter: abc*
rdb to redis cluster
[source]
type: rdb file
servers:
- /data/redis/dump1.rdb
- /data/redis/dump2.rdb
[target]
type: redis cluster
servers:
- 127.0.0.1:7379
[common]
listen: 0.0.0.0:8888
step: 2
mbuf_size: 512
source_safe: false
aof to redis cluster
[source]
type: aof file
servers:
- /data/redis/appendonly1.aof
- /data/redis/appendonly2.aof
[target]
type: redis cluster
servers:
- 127.0.0.1:7379
[common]
listen: 0.0.0.0:8888
step: 2
View redis-migrate-tool status
You can use redis-cli to connect to redis-migrate-tool. The address and port are set in the common configuration.
info command
For example, you can use the info command:
$redis-cli -h 127.0.0.1 -p 8888
127.0.0.1:8888> info
# Server
version:0.1.0
os:Linux 2.6.32-573.12.1.el6.x86_64 x86_64
multiplexing_api:epoll
gcc_version:4.4.7
process_id:9199
tcp_port:8888
uptime_in_seconds:1662
uptime_in_days:0
config_file:/ect/rmt.conf
# Clients
connected_clients:1
max_clients_limit:100
total_connections_received:3
# Memory
mem_allocator:jemalloc-4.0.4
# Group
source_nodes_count:32
target_nodes_count:48
# Stats
all_rdb_received:1
all_rdb_parsed:1
all_aof_loaded:0
rdb_received_count:32
rdb_parsed_count:32
aof_loaded_count:0
total_msgs_recv:7753587
total_msgs_sent:7753587
total_net_input_bytes:234636318
total_net_output_bytes:255384129
total_net_input_bytes_human:223.77M
total_net_output_bytes_human:243.55M
total_mbufs_inqueue:0
total_msgs_outqueue:0
127.0.0.1:8888>
Info command response introduction:
Server:
Clients:
- version: redis-migrate-tool version number
- os: operating system name
- multiplexing_api: Multiplexing API
- gcc_version: gcc version
- process_id: the process id of redis-migrate-tool
- tcp_port: the tcp port listened by redis-migrate-tool
- uptime_in_seconds: How long redis-migrate-tool runs. The unit is second.
- uptime_in_days: How long redis-migrate-tool runs. Unit days.
- config_file: the name of the redis-migrate-tool configuration file
- connected_clients: the number of currently connected clients
- max_clients_limit: The maximum number of clients allowed to connect at the same time
- total_connections_received: The total number of connections received so far
Group:
- source_nodes_count: the number of nodes of the source redis
- target_nodes_count: the number of nodes of the target redis
Stats:
- all_rdb_received: Whether the rdb of all source nodes has been received.
- all_rdb_parsed: Whether all source node rdb files have been parsed.
- all_aof_loaded: Whether all source node aof files have been loaded.
- rdb_received_count: The number of received source redis node rdb files.
- rdb_parsed_count: The number of rdbs that have been parsed.
- aof_loaded_count: The number of aofs that have been loaded.
- total_msgs_recv: Total number of messages received from source redis.
- total_msgs_sent: The total number of messages sent to the target redis and received.
- total_net_input_bytes: total size of data received from source redis
- total_net_output_bytes: The total size of the data sent to the target redis
- total_net_input_bytes_human: Same as total_net_input_bytes, converted into readable format
- total_net_output_bytes_human: Same as total_net_output_bytes, converted to a readable format
- total_mbufs_inqueue: Cache data from source redis mbufs (excluding rdb data)
- total_msgs_outqueue: The total number of messages to be sent to the target redis and sent waiting for a response.
Close redis-migrate-tool
shutdown [seconds|asap]
This command will do the following things:
- Stop copying data from source redis
- Attempt to send the data in the redis-migrate-tool buffer to the target cluster
- Stop redis-migrate-tool and exit
parameter:
Parameter:
- seconds: Before redis-migrate-tool exits, it sends the buffered data to the target redis for the maximum length of time (in seconds). The default is 10s
- asap: Don't care about the data in the buffer, just exit directly.
For example, you use the shutdown command:
$redis-cli -h 127.0.0.1 -p 8888
127.0.0.1:8888> shutdown
OK
Check migration results (redis_check)
After migrating the data, you can use the redis_check command to check the data of the original cluster and the target cluster.
Use the redis_check command:
$src/redis-migrate-tool -c rmt.conf -o log -C redis_check
Check job is running...
Checked keys: 1000
Inconsistent value keys: 0
Inconsistent expire keys : 0
Other check error keys: 0
Checked OK keys: 1000
All keys checked OK!
Check job finished, used 1.041s
If you want to check more keys, you can use the following command:
$src/redis-migrate-tool -c rmt.conf -o log -C "redis_check 200000"
Check job is running...
Checked keys: 200000
Inconsistent value keys: 0
Inconsistent expire keys : 0
Other check error keys: 0
Checked OK keys: 200000
All keys checked OK!
Check job finished, used 11.962s
Insert some test keys (redis_testinsert)
Use the redis_testinsert command:
$src/redis-migrate-tool -c rmt.conf -o log -C "redis_testinsert"
Test insert job is running...
Insert string keys: 200
Insert list keys : 200
Insert set keys : 200
Insert zset keys : 200
Insert hash keys : 200
Insert total keys : 1000
Correct inserted keys: 1000
Test insert job finished, used 0.525s
If you want to insert more keys:
$src/redis-migrate-tool -c rmt.conf -o log -C "redis_testinsert 30000"
Test insert job is running...
Insert string keys: 6000
Insert list keys : 6000
Insert set keys : 6000
Insert zset keys : 6000
Insert hash keys : 6000
Insert total keys : 30000
Correct inserted keys: 30000
Test insert job finished, used 15.486s
If you only want to insert keys of type string:
$src/redis-migrate-tool -c rmt.conf -o log -C "redis_testinsert string"
Test insert job is running...
Insert string keys: 1000
Insert list keys : 0
Insert set keys : 0
Insert zset keys : 0
Insert hash keys : 0
Insert total keys : 1000
Correct inserted keys: 1000
Test insert job finished, used 0.024s
If you insert some specific type of key:
$src/redis-migrate-tool -c rmt.conf -o log -C "redis_testinsert string|set|list 10000"
Test insert job is running...
Insert string keys: 3336
Insert list keys : 3336
Insert set keys : 3328
Insert zset keys : 0
Insert hash keys : 0
Insert total keys : 10000
Correct inserted keys: 10000
Test insert job finished, used 5.539s