ElasticSearch-index migration

Article Directory

O. Introduction

Introduce two better methods for migrating data from ES clusters. The following is a detailed description of the two methods, which have been verified by practice.

方法1, Use the open source tool elasticdump for migration. The advantage is that the operation is extremely simple, and there is no need to restart the cluster; the disadvantage is that the transmission is slow and it is not suitable for the transmission of large amounts of data. In the process, the http port is used between the clusters, which takes up the cluster bandwidth. Mainly still slow.
方法2, Use ES's own snapshot backup to file storage for migration. The advantage is the speed, because it is written to the disk, copied to the new cluster server, and restored from the snapshot, which is fast. Suitable for scenarios with large amounts of data

One, use elasticdump tool to migrate

0. Install nodejs (please ignore if it is already installed)

carried out

yum install -y nodejs

1. Upgrade nodejs

carried out

npm install -g n
n latest

2. Install elasticdump tool

carried out

npm install elasticdump -g

3. Migrate the settings, mapping, data of the specified index

carried out

elasticdump --input=http://fromhost:9200/indexname --output=http://tohost:9200/indexname --type=settings
elasticdump --input=http://fromhost:9200/indexname --output=http://tohost:9200/indexname --type=mapping
elasticdump --input=http://fromhost:9200/indexname --output=http://tohost:9200/indexname --type=data

2. Use ES's own snapshot backup to file storage for migration

0. The source ES cluster specifies the shared folder

Ensure that the /data/backups directory exists and belongs to the elastic startup user

mkdir /data/backups
chown -R elastic:elastic /data/backups

vim master node elasticsearch.yml add line

path.repo: ["/data/backups"]

Restart the master node to take effect

1. Create a repository in the source ES cluster

Execute instructions

curl -XPUT http://fromhost:9200/_snapshot/my_repository  -d '{
    "type": "fs", 
    "settings": {
        "location": "/data/backups" 
    }
}'

2. Create a snapshot (snapshot backup data)

curl -XPUT http://fromhost:9200/_snapshot/my_repository/snapshot_20191114?wait_for_completion=true  -d '{
    "indices": "indexname"
}'

After the backup is completed, there are backup files in the folder /data/backups

3. Move the snapshot of the source ES cluster to the warehouse of the target ES cluster

Copy /data/backups/my_backup to the new es cluster machine

4. Create a repository for the target ES cluster

curl -XPUT http://tohost:9200/_snapshot/my_repository  -d '{
    "type": "fs", 
    "settings": {
        "location": "/data/backups" 
    }
}'

5. New cluster view snapshot information

curl -XGET http://tohost:9200/_snapshot/my_repository/_all

6. Restore from snapshot

curl -XPOST http://tohost:9200/_snapshot/my_repository/snapshot_20191114/_restore