Redis migrate data migration tool

In the work, you may encounter the problem of single-point Redis migrating data to Redis cluster, but you can't do it with troublesome operation and maintenance. In order to facilitate the research and development of self-migrating data, I wrote a simple Redis migration tool here, hoping to be useful to those in need.

This tool supports:

  • Single-point Redis to single-point Redis migration
  • Single point Redis to Redis cluster migration
  • Redis cluster to Redis cluster migration
  • Redis cluster to single-point Redis migration

The tool has been compiled into a multi-platform command, just download the binary file directly from Github and execute it.
Project address: https://github.com/icowan/redis-tool After pulling down the code, you can directly execute the command make to compile executable files for multiple platforms, and you need to rely on the golang compiler.

  • Windows amd64: redis-tool-windows-amd64.exe
  • MacOS amd64: redis-tool-darwin-amd64
  • Linux amd64: redis-tool-linux-amd64
  • Linux arm64: redis-tool-linux -arm64
    View Usage:
    $ chmod a+x redis-tool-linux-amd64

    $ ./redis-tool-linux-amd64 -h

Supported data types

  • string
  • hash hash list
  • list list
  • sorted-set ordered set

how to use

Download the command and execute it after authorization. /redis-tool-linux-amd64 -h You can view all the functions supported by the tool:

$ ./redis-tool-darwin-amd64 migrate -h
数据迁移命令

Usage:
redis-tool migrate [command]

Examples:

支持命令:
[hash, set, sorted-set, list]

Available Commands:
all         迁移所有
hash       哈希列表迁移
list       列表迁移
 set         redis set 迁移
sorted-set 有序集合迁移

Flags:
 -h, --help                   help for migrate
     --source-auth string     源密码
     --source-database int   源database
     --source-hosts string   源redis地址, 多个ip用','隔开 (default "127.0.0.1:6379")
     --source-prefix string   源redis前缀
     --source-redis-cluster   源redis是否是集群
     --target-auth string     目标密码
     --target-database int   目标database
     --target-hosts string   目标redis地址, 多个ip用','隔开 (default "127.0.0.1:6379")
     --target-prefix string   目标redis前缀
     --target-redis-cluster   目标redis是否是集群

Use "redis-tool migrate [command] --help" for more information about a command.

Parameter Description:

  • --source-auth: source redis password, fill in if there is one
  • --source-database: source database, default is 0
  • --source-hosts: source redis address, multiple ips of the cluster are separated by',' (default "127.0.0.1:6379")
  • --source-prefix: source redis prefix, optional
  • --source-redis-cluster: Whether the source redis is a cluster, the default is false
  • --target-auth: migration target redis password, fill in if there is one
  • --target-database: migration target database, default is 0
  • --target-hosts: Migration target redis address, multiple ips of the cluster are separated by',' (default "127.0.0.1:6379")
  • --target-prefix: migration target redis prefix, optional
  • --target-redis-cluster: Whether the migration target redis is a cluster, default false

Migrate the data of a single key

Here are two examples, the others are not too different.

Hash type

You can view the instructions through the command redis-tool migrate hash -h

$ redis-tool migrate hash helloworld \
  --source-hosts 127.0.0.1:6379 \
  --target-redis-cluster true \
  --target-hosts 127.0.0.1:6379,127.0.0.1:7379 \
  --target-auth 123456

Redis migrate data migration tool

Ordered set

You can use the command redis-tool migrate sorted-set -h to view the instructions
. The amount of data in the ordered set may be relatively large, so the cut is performed in units of 50000. I have tested the migration of nearly 170 million pieces of data, which took more than 40 minutes.

$ redis-tool migrate hash helloworld \
  --source-hosts 127.0.0.1:6379 \
  --target-redis-cluster true \
  --target-hosts 127.0.0.1:6379,127.0.0.1:7379 \
  --target-auth 123456

Redis migrate data migration tool

Migrate all key data to support wildcard filtering

You can view the instructions through the command redis-tool migrate all -h

$ redis-tool migrate all "ipdetect:*" \ 
    --source-hosts 127.0.0.1:6379 \
    --target-redis-cluster true \
    --target-hosts 127.0.0.1:6379,127.0.0.1:7379 \
    --target-auth 123456

This command will compile all the matched keys, and then migrate gradually according to the key type.

tail

A relatively simple tool written in golang, which is mainly used when Redis does not have persistence or when multiple sets of Redis are migrated to one set of Redis.

Hope it is useful to everyone, thank you!

Author: CreditEase Institute of Technology Cong

Guess you like

Origin blog.51cto.com/14159827/2545142