TiDB (6): Data Migration - TiDB Lightning

1 Introduction to TiDB Lightning

TiDB Lightning is a tool for importing full data into TiDB clusters at high speed. It currently supports data sources in Mydumper or CSV output format. You can use Lightning in the following two scenarios:

Quickly import large volumes of new data.

Backup restores all data.

TiDB Lightning mainly consists of two parts:

(1) tidb-lightning ("front end"): mainly completes the adaptation work, reads the data source, builds tables in the downstream TiDB cluster, converts the data into key/value pairs (KV pairs) and sends them to tikv-importer for inspection data integrity etc.

(2) tikv-importer ("backend"): mainly completes the work of importing data into the TiKV cluster, caches, sorts, splits and imports the KV pairs written by tidb-lightning into the TiKV cluster.

2 Prepare the Migration Tool

First use the address download tool

wget https://download.pingcap.org/tidb-enterprise-tools-latest-linux-amd64.tar.gz

wget https://download.pingcap.org/tidb-toolkit-latest-linux-amd64.tar.gz

3 Prepare MySQL data

CREATE DATABASE mytest;
USE mytest;

CREATE TABLE mytest.t1 (
	id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
	c CHAR (32),
	PORT INT
);

insert into t1 values (),(),(),(),(),(),(),();
        insert into t1 (id) select null from t1;
        insert into t1 (id) select null from t1;
        insert into t1 (id) select null from t1;
        insert into t1 (id) select null from t1;
        insert into t1 (id) select null from t1;
        update t1 set c=md5(id), port=@@port;

CREATE TABLE mytest.t2 (
	id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
	c CHAR (32),
	PORT INT
);

insert into t2 values (),(),(),(),(),(),(),();
        insert into t2 (id) select null from t2;
        insert into t2 (id) select null from t2;
        insert into t2 (id) select null from t2;
        insert into t2 (id) select null from t2;
        insert into t2 (id) select null from t2;
        update t2 set c=md5(id), port=@@port;

4 Export data

mkdir -p /data/my_database/

cd  /home/tidb-enterprise-tools-latest-linux-amd64/bin

./mydumper -h 192.168.222.132 -P 3306 -u root -p 123456 -t 16 -F 256 -B mytest -T t1,t2 --skip-tz-utc -o /data/my_database/

cd /data/my_database

in:

-B mytest : Export from the mytest database.
-T t1,t2: Export only the two tables t1 and t2.
-t 16: Use 16 threads to export data.
-F 256: Divide each table into multiple files, each with a size of about 256 MB.
--skip-tz-utc: Adding this parameter will ignore the inconsistency of time zone settings between TiDB and the machine that imports data, and prohibit automatic conversion.

In this way, the full backup data is exported to the /data/my_database directory.

5 start tikv-importer

cd /home/tidb-toolkit-latest-linux-amd64/bin

Modify the configuration file

vim tikv-importer.toml

# TiKV Importer 配置文件模版

# 日志文件。
log-file = "tikv-importer.log"
# 日志等级：trace、debug、info、warn、error、off。
log-level = "info"

[server]
# tikv-importer 监听的地址，tidb-lightning 需要连到这个地址进行数据写入。
addr = "192.168.222.136:8287"

[import]
# 存储引擎文档 (engine file) 的文件夹路径。
import-dir = "/mnt/ssd/data.import/"

start tikv-importer

nohup ./tikv-importer -C tikv-importer.toml > nohup.out &

6 Start tidb-lightning

The startup command is more complicated, it is recommended to use a script, as follows:

vim run.sh

The script content is as follows

#!/bin/bash
nohup ./tidb-lightning \
            --importer 192.168.1.101:8287 \
            -d /data/my_database/ \
            --pd-urls 0.0.0.0:2379 \
            --tidb-host 192.168.1.101 \
            --tidb-user root \
            --log-file tidb-lightning.log \
        > nohup.out &

Modify script permissions

chmod 755 run.sh

startup script

./run.sh

This is to check TiDB, and found that all the data specified in mysql has been synchronized to TiDB