Actual use of the Elasticdump tool to back up and restore data on the Elasticsearch cluster

Text/Zhu Jiqian

• Contents
  1. Introduction to the Elasticdump tool
  2. Installation of the Elasticdump tool
  3. Use of the Elasticdump tool

Recently, I have done some work involving Elasticsearch mapping structure and data export and import during development. I am afraid that I will forget this process in the future. It can be said that a good memory is not as good as a bad writing, so I recorded it in a blog post.

Children who play Elasticsearch will probably encounter such a problem, how to quickly map the index structure and corresponding data in Elasticsearch to quickly back up and restore data.

At this time, it can be achieved through Elasticsearch's import and export tool Elasticdump, which can index, back up and restore data from different Elasticsearch clusters.

1. Introduction to the Elasticdump tool

On the official English website of npm about Elasticdump, you can see an English introduction about Elasticdump. The logo of this tool is very interesting. It is a tool cart that can move (migrate) things (data). This logo indicates that the Elasticdump tool Can be used to migrate backup and restore data.

What is particularly important when using Elasticdump is that if it is npm install elasticdump -gused follow the instructions, the node version must v10.0.0be above to support it, otherwise an error will occur when executing this instruction.

Elasticdump inputworks by sending aoutput

elasticdump --input SOURCE --output DESTINATION [OPTIONS]

• input SOURCEIndicates reading data source SOURCE
• output DESTINATIONIndicates transferring data source to destination DESTINATION.
• SOURCE/DESTINATIONBoth can be Elasticsearch URLs or files. If it is an Elasticsearch URL, such as http://127.0.0.1/index, it means directly importing or exporting the index from the ES library at the address http://127.0.0.1 related data.
• [OPTIONS]It is the operation option. The more commonly used ones are type and limit. Other operations will not be introduced here.
   

type is the ES data export and import type. The Elasticdum tool supports the import and export of the following data types -

type type	illustrate
mapping	ES index mapping structure data
data	ES data
settings	ES’s index library default configuration
analyzer	ES tokenizer
template	ES template structure data
alias	ES index alias

limitThe number of objects backed up from SOURCE to DESTINATION is 100 by default and can be customized.

2. Elasticdump tool installation

1. Installing the Elasticdum tool online depends on node, so install node v10.0.0 or above first.

[root@zhu opt]# wget https://nodejs.org/dist/v12.18.3/node-v12.18.3-linux-x64.tar.xz
[root@zhu opt]# tar xvf  node-v12.18.3-linux-x64.tar.xz -C /usr/local/
[root@zhu opt]# mv /usr/local/node-v12.18.3-linux-x64 /usr/local/nodejs
[root@zhu opt]# echo export NODEJS_HOME=/usr/local/nodejs >> /etc/profile
[root@zhu opt]# echo export PATH=$PATH:$NODEJS_HOME/bin >> /etc/profile
[root@zhu opt]# echo export NODEJS_PATH=$NODEJS_HOME/lib/node_modules >>/etc/profile
[root@zhu opt]# source /etc/profile
[root@zhu opt]# ln -s /usr/local/nodejs/bin/node /usr/local/bin/node
[root@zhu opt]# ln -s /usr/local/nodejs/bin/npm /usr/local/bin/npm
[root@zhu opt]# npm -v
6.14.6
[root@zhu opt]# node -v
v12.18.3

2. Install elasticdump via npm

[root@zhu opt]# npm install elasticdump -g

After successful installation, go to

[root@zhu opt]#cd /usr/local/nodejs/lib/node_modules/elasticdump/bin

You can see that there are two commands, elasticdump is used to back up a single index, and multielasticdump can be used to back up multiple indexes in parallel:

root@zhu bin]# ll
总用量 20
-rwxr-xr-x. 1 1001 1001  4026 4月   9 14:38 elasticdump
-rwxr-xr-x. 1 1001 1001 14598 10月 26 1985 multielasticdump

3. Use of Elasticdump tool

Use elasticdump to perform a single index backup and restore operation -


- Export the mapping structure of the index test_event:

[root@zhu opt]# elasticdump --input=http://127.0.0.1:9200/test_event  --output=/opt/test_event_mapping.json --type=mapping 

Check the current situation and find that it has been backed up into a json file:

[root@zhu opt]# ll
总用量 14368
-rw-r--r--. 1 root root     6200 4月   9 11:30 ucas_hisevenr_mapping.json

You can also directly import it into another es cluster:

[root@zhu opt]# elasticdump --input=http://127.0.0.1:9200/test_event   --output=http://127.0.0.2:9200/test_event --type=mapping

- Export data of index test_event:

[root@zhu opt]# elasticdump --input=http://127.0.0.1:9200/test_event  --output=/opt/data.json --type=data

In the same way, backup data can be directly imported into another ES cluster:

[root@zhu opt]# elasticdump --input=http://127.0.0.1:9200/test_event   --output=http://127.0.0.2:9200/test_event --type=data

elasticdump performs data restoration operation

-Mapping mapping structure restoration:

[root@zhu opt]# elasticdump --input=/opt/test_event_mapping.json --output http://127.0.0.1:9200/ --type=mapping

- data data restoration

[root@zhu opt]# elasticdump --input=/opt/data.json    --output=http://127.0.0.1:9200/test_event    --type=data

Use elasticdump for multiple index backup operations:

＃将ES索引及其所有类型备份到es_backup文件夹中
multielasticdump direction = dump match ='^.*$'  input = http://127.0.0.1:9200   output =/tmp/es_backup

＃仅备份ES索引以“ -index”（匹配正则表达式）为前缀的结尾。仅备份索引数据。所有其他类型都将被忽略。＃注意：默认情况下会忽略分析器和别名类型
multielasticdump --direction=dump --match='^.*-index$' --input=http://127.0.0.1:9200 --ignoreType='mapping,settings,template'  --output=/tmp/es_backup

Use elasticdump for multiple index restore operations:

multielasticdump --direction=load --input=/tmp/es_backup --output=http://127.0.0.1:9200

According to the introduction of npm's elasticdump English official website, one thing to note here is that even when using multielasticdump, there is a difference between the --direction parameter setting and the --ignoreType parameter setting.

• When backing up, --direction= dumpis the default, it --inputmust be the URL of the ElasticSearch server's base location (i.e. http://localhost:9200), and --outputit must be a directory. A data, mapping and analyzer file is created for each matching index.

• When restoring, to load files dumped from multi-elasticsearch, --directionthis should be set to load, --inputmust be the directory where multi-elasticsearch was dumped, and --outputmust be the Elasticsearch server URL.

• --match` is used to filter which indexes should be dumped/loaded (regular expression).

• --ignoreTypeAllow types to be ignored from dump/load. Six options are supported. data,mapping,analyzer,alias,settings,template. Provides multi-type support, each type must be separated by commas when used, and intervalallows control of the dump/load interval for generating new indexes.

• --includeTypeAllow types to be included in dump/load. Supports six options -  data,mapping,analyzer,alias,settings,template.

All kinds of articles will be organized and posted on the public account [Zhu Jiqian who writes code]——