Use redis-migrate-tool for Redis data migration

Use redis-migrate-tool for Redis data migration

advantage

• fast.
• Multithreading.
• Based on redis replication.
• Live migration.
• During the migration process, the source cluster does not affect the external services provided.
• Heterogeneous migration.
• Support Twemproxy cluster, redis cluster cluster, redis single node, rdb file and aof file.
• filter function.
• When the target cluster is Twemproxy, the data will skip Twemproxy and be directly imported to the backend redis.
• Migration status can be displayed.
• Perfect data sampling verification.

Types of supported migrations

Source: Redis single node (single), Redis cluster (redis cluster), rdb file (snapshot), aof file (log), twemproxy cluster

Target: Redis single node (single), Redis cluster (redis cluster), rdb file (snapshot), aof file (log), twemproxy cluster

Specific migration steps

1.git clone [email protected]:vipshop/redis-migrate-tool.git

2. Compile the project, the specific steps are as follows:

cd redis-migrate-tool
autoreconf -fvi
./configure
make
src/redis-migrate-tool -h

3. Modify the configuration file rmt.conf

redis cluster "" redis cluster type type redis cluster redis cluster "" redis single node type type: single redis single node "" redis single node type type single

[source]
type: redis cluster
servers:
- 127.0.0.1:8379

[target]
type: redis cluster
servers:
- 127.0.0.1:7379

[common]
listen: 0.0.0.0:8888

redis cluster cluster >> twemproxy cluster

[source]
type: redis cluster
servers:
- 127.0.0.1:6379

[target]
type: twemproxy
hash: fnv1a_64
hash_tag: "{}"
distribution: ketama
servers:
- 127.0.0.1:6380:1 server1
- 127.0.0.1:6381:1 server2
- 127.0.0.1:6382:1 server3
- 127.0.0.1:6383:1 server4
	
[common]
listen: 0.0.0.0:34345
threads: 8
step: 1
mbuf_size: 512
source_safe: true

4. After the configuration file is modified, the startup tool starts to run ( the command is still executed in the root directory of the project )

src/redis-migrate-tool -c rmt.conf -o log -d

5. Connect the port monitored by redis-migrate-tool through redis-cli to check the status. You can also use the keys * command to check whether the number of keys is close.

$redis-cli -h 127.0.0.1 -p 8888
127.0.0.1:8888> info
# Server
version:0.1.0
os:Linux 2.6.32-573.12.1.el6.x86_64 x86_64
multiplexing_api:epoll
gcc_version:4.4.7
process_id:9199
tcp_port:8888
uptime_in_seconds:1662
uptime_in_days:0
config_file:/ect/rmt.conf

# Clients
connected_clients:1
max_clients_limit:100
total_connections_received:3

# Memory
mem_allocator:jemalloc-4.0.4

# Group
source_nodes_count:32
target_nodes_count:48

# Stats
all_rdb_received:1
all_rdb_parsed:1
rdb_received_count:32
rdb_parsed_count:32
total_msgs_recv:7753587
total_msgs_sent:7753587
total_net_input_bytes:234636318
total_net_output_bytes:255384129
total_net_input_bytes_human:223.77M
total_net_output_bytes_human:243.55M
total_mbufs_inqueue:0
total_msgs_outqueue:0
127.0.0.1:8888>

6. Data verification

Excuting an order

src/redis-migrate-tool -c rmt.conf -o log -C redis_check

Information that can be viewed

Check job is running...

Checked keys: 9467
Inconsistent value keys: 0
Inconsistent expire keys : 0
Other check error keys: 0
Checked OK keys: 1000

All keys checked OK!
Check job finished, used 4.143s

At this point, the redis migration is complete. You can modify the reds address and re-upgrade the service according to your own business scenario.