es data import and export

Use elasticdump to import and export data

1. To install elasticdump,
enter
1 npm install elasticdump -g
-g in the terminal to indicate that it is available globally. Enter elasticdump --version directly in the terminal. If the version information appears, the installation is successful, as follows:
1
2 C:\Users\T470s>elasticdump --version
6.3.3

2. Commonly used import and export methods
　1. Export the data in the index to the local
1 elasticdump --input=http://localhost:9200/demo --output=D:/ES/date/demo.json
　Among them, demo is index.
　2. Import local data into es
1 elasticdump --input=D:/ES/date/demo.json --output=http://localhost:9200/demo1
　3. Import es into another es
1 elasticdump --input =http://ip:9200/demo --output=http://127.0.0.1:9200/demo
　4. Import with word segmentation Preface
: The company's running project needs to set up a test environment, and the ES of the formal environment needs to be The data stored in it is imported into the ES server of the test environment. Use the local windows system to achieve.
1. Environment preparation
Because elasticdump depends on the nodejs environment, you need to download the relevant dependent environment of nodejs. Go to the nodejs official website to install the package directly. I downloaded the msi installer. After installation, the environment variables will be configured directly. Directly in cmd or Powershell Just download elasticdump:
1 npm install elasticdump

2. Export the production environment to the test environment
1) Export the word breaker, pay special attention when exporting the word breaker, we can only import one by one according to the index, not all export, all export will cause an error that the index does not exist: 1 elasticdump
--input =http://ip:9200 --output=http://127.0.0.1:9200/ --type=analyzer --all=true　　
The command for all export and import will be as follows:
1 Error Emitted => {"root_cause ":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: index is missing;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: index is missing missing;”}
　
So you can import them separately by index later:
1
2
3 elasticdump --input=http://ip:9200/applog --output=http://127.0.0.1:9200/applog --type =analyzer
elasticdump --input=http://ip:9200/cms_article --output=http://127.0.0.1:9200/cms_article --type=analyzer
elasticdump --input=http://ip:9200/followup --output=http://127.0.0.1:9200/followup --type=analyzer
pit: I don't know how many indexes there are when I pour it in again , how to determine the name of the index?
Here I directly export all the mappings to the local, then find the name of the index and then import the word segmentation:
1 elasticdump --input=D:/mapping.json --output=http://127.0.0.1:9200/ - -all=true --type=mapping
Doubt: I currently only have three indexes here, what should I do if there are more than one? More than 100?
2) Export mapping
Mapping can be imported and exported directly, direct command:
1 elasticdump --input=http://ip:9200/ --output=http://127.0.0.1:9200/ --all =true --type=mapping
3) Export all data Data
under each index can be imported all at once:
1 elasticdump --input=http://ip:9200/ --output=http://127.0.0.1 :9200/ --all=true --type=data
3, summary
1), what happens if analyzer is not imported?
I have tried that if only data and mapping are imported, the data will be imported and the mapping will change, and the final result has no word segmentation effect.
2), if you do not import mapping, there will be no word segmentation effect.
3) It is best to import in order: analyzer, mapping, and data in this order to prevent the result of word segmentation failure.

Import in order: analyzer, mapping, data

Export of es data

// index export

// Export index mapping to .json file

elasticdump \
  --input=http://production.es.com:9200/demo \
  --output=/data/demo.json \
  --type=mapping

//export all indexes

elasticdump --input=http://ip:9200/ --output=http://127.0.0.1:9200/ --all=true --type=mapping

// data export

// Export all data of index to .json file

elasticdump \
  --input=http://production.es.com:9200/demo \
  --output=/data/demo.json \
  --type=data

//export all data

elasticdump --input=http://ip:9200/ --output=http://127.0.0.1:9200/ --all=true --type=data

// tokenizer export

//Export the tokenizer, pay special attention when exporting the tokenizer, we can only import one by one according to the index, not all of them, and there will be an error that the index does not exist when exporting all of them //Change to import separately by index
:

elasticdump --input=http://ip:9200/applog --output=http://127.0.0.1:9200/applog --type=analyzer
elasticdump --input=http://ip:9200/cms_article --output=http://127.0.0.1:9200/cms_article --type=analyzer
elasticdump --input=http://ip:9200/followup --output=http://127.0.0.1:9200/followup --type=analyzer

es data import

• Import data from local files

elasticdump  --input=/data/demo.json --output=http://localhost:9200/demo1

Import es into another es

elasticdump --input=http://ip:9200/demo --output=http://127.0.0.1:9200/demo

#!/usr/bin/env bash

array=(“full_vehicle_portrait” “latest_car_history_trail_dt” “vehicle_abnormal_monitor” “gantry_transaction-20200914” “vehicle_abnormal_summary”)
for(( i=0;i<KaTeX parse error: Expected '}', got '#' at position 2: {#̲array[@]};i++))…{array[i]}
#创建索引
curl -X PUT ‘http://cdh173:9200/$VAR1?pretty’

elasticdump
–input=http://33.69.6.95:9200/$array[i]--output=http://cdh219:9200/${array[i]}
–type=analyzer
elasticdump
–input=http://33.69.6.95:9200/$array\left[i\right]--output=http://cdh219:9200/${array[i]}
–type=settings
elasticdump
–input=http://33.69.6.95:9200/$array[i]--output=http://cdh219:9200/${array[i]}
–type=mappingcc

elasticdump --size=200
–input=http://33.69.6.95:9200/$array\left[i\right]--output=http://cdh219:9200/${array[i]}
–type=data
done

If there are a lot of indexes in my es, it will still take time to execute them one by one, what should I do? Of course shell scripts come to mind.
The script name is: esExportOrInput.sh
#!/bin/sh
index_name=$1
index_data=$1"_data"
index_settings=$1"_settings"
echo "Start execution"

Export index mapping data

./elasticdump --input=http://source:9200/$inde{x}_{n}ame/--output=./$index_name.json --type=mapping
./elasticdump --input=http://source:9200/$inde{x}_{n}ame/--output=./$index_settings.json --type=settings

Export index data

./elasticdump --input=http://source:9200/$inde{x}_{n}ame/--output=./$index_data.json --type=data --limit=1000

#Import index Mapping data
echo "Delete index start of target service:" $inde{x}_{n}amecurl-XDELETEhttp://target:9200/$ index_name
echo "Delete index end of the target service: "$inde{x}_{n}amesleep3echo"Wait\; three\; seconds..."./elasticdump--input=./$index_settings.json --output=http://target:9200/$inde{x}_{n}ame/--type=settings./elasticdump--input=./$index_name.json --output=http://target:9200/$inde{x}_{n}ame/--type=mapping./elasticdump--input=./$ index_data.json --output=http://target:9200/KaTeX parse error: Expected 'EOF', got '#' at position 39: …a --limit=1000 #̲Clear generated files rm -f ..index_name.json
rm -f ./$inde{x}_{s}ettings.jsonrm-f./$ index_data.json
#Clear generated files
echo "Index" $index_name" execution completed"

One variable needs to be passed during execution, which is the index name index_name

Reminder:
1. This script can only be used on the server where elasticdump is installed. The script directory is in /root/node_modules/elasticdump/bin
2. Export and import a batch of 100 by default, you can add –limit number of entries per page to customize each number of pages.
3. When importing, you must restrict the file import of settings, and execute the import of mapping, otherwise there will be conflicts, because the unique identification of uuid is carried on the settings.
Example: If I want to migrate the data of the bum_user index from the test environment to the stress test server target, just execute the command
./esExportOrInput.sh bum_user

Leave a homework, you can transfer dynamic parameters between the source server and the target server, and you can achieve universal adaptation. If you are interested, rewrite it