[Operation and Maintenance Knowledge Master] Two methods, one-click deployment of ElasticSearch cluster (Shell+Ansible automated deployment)

Enterprise 2023-12-18 01:34:05 views: null

This article will introduce to you how to use Shell and Ansible to deploy ES (ElasticSearch) clusters with one click, realize batch operations in cluster deployment, and enjoy the charm of automated deployment.

Prepare the host

CPU name IP cpu disk memory
Ansible 10.0.0.61 2C 20G 2G
ELK104 10.0.0.104 2C 20G 2G
ELK105 10.0.0.105 2C 20G 2G
ELK106 10.0.0.106 2C 20G 2G

Shell one-click deployment

We prepare the required files on Ansible (ES installation package without JDK and Oracle's JDK), and write Shell scripts in Ansible to achieve one-click deployment of the ES cluster. In fact, there are many ideas for script writing. Here I use Do It First to avoid After completing the key, write a for loop to perform single-point deployment. You can also directly deploy a node and copy it.

If you want to modify the directory, you need to modify the scripts, elasticsearch.yml, and es7.service files, so the configuration files are packaged and placed at the end of the article.

[root@Ansible ~]# cat deploy_ES.sh
#!/bin/bash
#1、先做免密钥，方便文件的推送以及远程执行命令
if [ -f /root/.ssh/id_rsa.pub ];then
	echo "公钥已经存在"
else
	ssh-keygen -t rsa
	echo "公钥不存在，自动创建公钥"
fi

hosts=(
10.0.0.104
10.0.0.105
10.0.0.106
)

for i in ${hosts[*]}
do
	expect <<EOF
      set timeout 10
      spawn ssh-copy-id -i /root/.ssh/id_rsa.pub root@$i
      expect {
        "yes/no" { send "yes\n";exp_continue }
        "password" { send "1\n" }
      }
      expect "password" { send "1\n" }
EOF
done

#2、准备配置文件



# 拼接IP字符串
ip_str=$(printf '", "%s' "${hosts[@]}")
ip_str=${ip_str:3}

# 替换文件中的IP地址
sed -i 's#^discovery.*#discovery.seed_hosts: \['"$ip_str"'"\]#g' elasticsearch.yml

sed -i 's#^cluster.initial_master_nodes.*#cluster.initial_master_nodes: \['"$ip_str"'"\]#g' elasticsearch.yml


#3、搭建ES集群

for q in ${hosts[*]}
do
	scp -r elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz jdk-8u291-linux-x64.tar.gz elasticsearch.yml es7.service jvm.options jdk.sh $q:/root/
	ssh $q "mkdir -p /koten/{softwares,data,logs}"
	ssh $q "tar xf /root/elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz -C /koten/softwares/"
	ssh $q "useradd koten"
	ssh $q "chown koten:koten /koten/softwares/elasticsearch-7.17.5/ -R"
	ssh $q "cat > /etc/security/limits.d/es.conf <<EOF
*          soft    nofile     65535
*          hard    nofile     131070
EOF
"	
	ssh $q "cat > /etc/sysctl.d/es.conf <<EOF
vm.max_map_count=262144
EOF"
	ssh $q "sysctl -f /etc/sysctl.d/es.conf"
	ssh $q "tar xf /root/jdk-8u291-linux-x64.tar.gz -C /koten/softwares/"
	ssh $q "\cp /root/elasticsearch.yml  /koten/softwares/elasticsearch-7.17.5/config/"
	ssh $q "\cp /root/es7.service /usr/lib/systemd/system/"
	ssh $q "\cp /root/jvm.options /koten/softwares/elasticsearch-7.17.5/config/"
	ssh $q "\cp /root/jdk.sh /etc/profile.d/"
	ssh $q "source  /etc/profile.d/jdk.sh"
	ssh $q "install -d /koten/{data,logs}/es7 -o koten -g koten"
done

#3、都部署好后再启动，防止脑裂
for e in ${hosts[*]}
do
	ssh $e "systemctl enable --now es7"
	if [ `ss -ntl | grep 19|wc -l` != 0 ];then
		echo "$e 主机ES启动成功！"
	fi	
done

[root@Ansible ~]# sh deploy_ES.sh
......
Created symlink from /etc/systemd/system/multi-user.target.wants/es7.service to /usr/lib/systemd/system/es7.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/es7.service to /usr/lib/systemd/system/es7.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/es7.service to /usr/lib/systemd/system/es7.service.

[root@ELK104 ~]# netstat -tnulp|grep 19
tcp6       0      0 :::19200                :::*                    LISTEN      2252/java           
tcp6       0      0 :::19300                :::*                    LISTEN      2252/java

 

Ansible batch deployment

1. Initialize the roles directory

[root@Ansible roles]# ansible-galaxy init es
- Role es was created successfully
[root@Ansible roles]# cd es 
[root@Ansible es]# pwd
/ansible/roles/es
[root@Ansible es]# ls
defaults  files  handlers  meta  README.md  tasks  templates  tests  vars

2. Write key-free playbooks for 10.0.0.104, 10.0.0.105, and 10.0.0.106

[root@Ansible es]# cat /etc/ansible/hosts | tail -4 
[es_group]
es104 ansible_ssh_host=10.0.0.104 ansible_ssh_user='root' ansible_ssh_pass='1'
es105 ansible_ssh_host=10.0.0.105 ansible_ssh_user='root' ansible_ssh_pass='1'
es106 ansible_ssh_host=10.0.0.106 ansible_ssh_user='root' ansible_ssh_pass='1'

[root@Ansible es]# cat ssh.yaml 
---
- hosts: es_group
  gather_facts: no
 
  tasks:
  - name: install ssh key
    authorized_key: user=root
                    key="{
   
   { lookup('file','/root/.ssh/id_rsa.pub')}}"
                    state=present

[root@Ansible es]# ansible-playbook ssh.yaml 

PLAY [es_group] ****************************************************************************************

TASK [install ssh key] *********************************************************************************
changed: [es106]
changed: [es105]
changed: [es104]

PLAY RECAP *********************************************************************************************
es104                      : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
es105                      : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
es106                      : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

3. Write a playbook for data push and configuration

If you need to modify the host, you must not only modify the hosts file, but also modify elasticsearch.yml

[root@Ansible roles]# cat hosts | tail -4
[es_group]
10.0.0.104
10.0.0.105
10.0.0.106

[root@Ansible roles]# cat site.yml
- hosts: es_group
  roles:
    - role: es

[root@Ansible roles]# ls es/files/
elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz  es7.service                 jdk.sh
elasticsearch.yml                                jdk-8u291-linux-x64.tar.gz  jvm.options

[root@Ansible roles]# cat es/tasks/main.yml 
- name: mkdir /koten/softwares /koten/data /koten/logs
  file: 
    path: "{
   
   { item }}"
    state: directory
  with_items:
    - /koten/softwares
    - /koten/data
    - /koten/logs

- name: tar xf elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz jdk-8u291-linux-x64.tar.gz
  unarchive: 
    src: "{
   
   { item }}"
    dest: /koten/softwares
  with_items:
    - elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz
    - jdk-8u291-linux-x64.tar.gz

- name: Create koten Group
  group:
    name: koten

- name: Create koten User
  user:
    name: koten

- name: chown -R koten.koten /koten/softwares/elasticsearch-7.17.5/
  file:
    path: /koten/softwares/elasticsearch-7.17.5/
    owner: koten
    group: koten
    recurse: yes

- name: copy content to /etc/security/limits.d/es.conf
  copy:
    content: | 
      *          soft    nofile     65535
      *          hard    nofile     131070
    dest: /etc/security/limits.d/es.conf

- name: copy test content to /etc/sysctl.d/es.conf
  copy:
    content: |
      vm.max_map_count=262144
    dest: /etc/sysctl.d/es.conf

- name: sysctl -f /etc/sysctl.d/es.conf
  command: sysctl -f /etc/sysctl.d/es.conf

- name: push es7.service to /usr/lib/systemd/system/
  copy: 
    src: es7.service
    dest: /usr/lib/systemd/system/

- name: push jvm.options to /koten/softwares/elasticsearch-7.17.5/config/
  copy:
    src: jvm.options
    dest: /koten/softwares/elasticsearch-7.17.5/config/

- name: push elasticsearch.yml to /koten/softwares/elasticsearch-7.17.5/config/
  copy:
    src: elasticsearch.yml
    dest: /koten/softwares/elasticsearch-7.17.5/config/elasticsearch.yml

- name: push jdk.sh to /etc/profile.d/
  copy:
    src: jdk.sh
    dest: /etc/profile.d/

- name: source /etc/profile.d/jdk.sh
  command: bash -c 'source /etc/profile.d/jdk.sh'

- name: mkdir /koten/data/es7 /koten/logs/es7
  file: 
    path: "{
   
   { item }}"
    state: directory
    owner: koten
    group: koten
  with_items:
    - /koten/data/es7
    - /koten/logs/es7

- name: systemctl enable --now es7
  systemd:
    name: es7
    state: started
    enabled: yes

4. Execute data push and configuration playbook, and deploy es cluster

[root@Ansible roles]# ansible-playbook site.yml 

PLAY [es_group] ****************************************************************

TASK [Gathering Facts] *********************************************************
ok: [es105]
ok: [es106]
ok: [es104]

TASK [mkdir /koten/softwares /koten/data /koten/logs] **************************
changed: [es105] => (item=/koten/softwares)
changed: [es104] => (item=/koten/softwares)
changed: [es106] => (item=/koten/softwares)
changed: [es106] => (item=/koten/data)
changed: [es104] => (item=/koten/data)
changed: [es105] => (item=/koten/data)
changed: [es106] => (item=/koten/logs)
changed: [es105] => (item=/koten/logs)
changed: [es104] => (item=/koten/logs)

TASK [es : tar xf elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz jdk-8u291-linux-x64.tar.gz] ***
changed: [es105] => (item=elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz)
changed: [es104] => (item=elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz)
changed: [es106] => (item=elasticsearch-7.17.5-no-jdk-linux-x86_64.tar.gz)
changed: [es105] => (item=jdk-8u291-linux-x64.tar.gz)
changed: [es106] => (item=jdk-8u291-linux-x64.tar.gz)
changed: [es104] => (item=jdk-8u291-linux-x64.tar.gz)

TASK [es : Create koten Group] *************************************************
changed: [es105]
changed: [es106]
changed: [es104]

TASK [es : Create koten User] **************************************************
changed: [es104]
changed: [es105]
changed: [es106]

TASK [chown -R koten.koten /koten/softwares/elasticsearch-7.17.5/] *************
changed: [es104]
changed: [es105]
changed: [es106]

TASK [copy content to /etc/security/limits.d/es.conf] **************************
changed: [es104]
changed: [es106]
changed: [es105]

TASK [copy test content to /etc/sysctl.d/es.conf] ******************************
changed: [es106]
changed: [es104]
changed: [es105]

TASK [sysctl -f /etc/sysctl.d/es.conf] *****************************************
changed: [es106]
changed: [es104]
changed: [es105]

TASK [push es7.service to /usr/lib/systemd/system/] ****************************
changed: [es104]
changed: [es105]
changed: [es106]

TASK [push jvm.options to /koten/softwares/elasticsearch-7.17.5/config/] *******
changed: [es104]
changed: [es105]
changed: [es106]

TASK [push elasticsearch.yml to /koten/softwares/elasticsearch-7.17.5/config/] ***
changed: [es104]
changed: [es106]
changed: [es105]

TASK [es : push jdk.sh to /etc/profile.d/] *************************************
changed: [es104]
changed: [es105]
changed: [es106]

TASK [es : source /etc/profile.d/jdk.sh] ***************************************
changed: [es104]
changed: [es106]
changed: [es105]

TASK [mkdir /koten/data/es7 /koten/logs/es7] ***********************************
changed: [es104] => (item=/koten/data/es7)
changed: [es105] => (item=/koten/data/es7)
changed: [es106] => (item=/koten/data/es7)
changed: [es105] => (item=/koten/logs/es7)
changed: [es104] => (item=/koten/logs/es7)
changed: [es106] => (item=/koten/logs/es7)

TASK [systemctl enable --now es7] **********************************************
changed: [es106]
changed: [es104]
changed: [es105]

PLAY RECAP *********************************************************************
es104                      : ok=16   changed=15   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
es105                      : ok=16   changed=15   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
es106                      : ok=16   changed=15   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   



# 查看es的状态
[root@ELK104 ~]# systemctl status es7.service 
● es7.service - es7
   Loaded: loaded (/usr/lib/systemd/system/es7.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2023-10-27 14:01:54 CST; 42s ago
 Main PID: 3578 (java)
   CGroup: /system.slice/es7.service
           ├─3578 /koten/softwares/jdk1.8.0_291/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=6...
           └─3711 /koten/softwares/elasticsearch-7.17.5/modules/x-pack-ml/platform/linux-x86_64/bin/c...

Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,716][INFO ][o.e.p.PluginsService ...wn]
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,716][INFO ][o.e.p.PluginsService ...ql]
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,716][INFO ][o.e.p.PluginsService ...ck]
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,717][INFO ][o.e.p.PluginsService ...re]
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,717][INFO ][o.e.p.PluginsService ...de]
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,749][INFO ][o.e.p.PluginsService ...er]
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,750][INFO ][o.e.p.PluginsService ...ded
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,958][INFO ][o.e.e.NodeEnvironment...fs]
Oct 27 14:02:19 ELK104 elasticsearch[3578]: [2023-10-27T14:02:19,959][INFO ][o.e.e.NodeEnvironment...ue]
Oct 27 14:02:20 ELK104 elasticsearch[3578]: [2023-10-27T14:02:20,074][INFO ][o.e.n.Node           ...st]
Hint: Some lines were ellipsized, use -l to show in full.

 

deploy_es_shell package file download link: https://pan.baidu.com/s/1l22I0aSB-50NXlh4lYZArw?pwd=umbm

deploy_es_ansible package file download link: https://pan.baidu.com/s/1lIfYWMnhsg7yoyzgkBSEDA?pwd=8hpl

My name is Koten. I have 10 years of operation and maintenance experience. I continue to share operation and maintenance tips. Thank you for reading and paying attention!

Guess you like

Origin blog.csdn.net/qq_37510195/article/details/130855070
Recommended
TIOBE May list: Fortran “resurrected” into Top 10
GCC 14.1 released
Ranking
B. Little Girl and Game【1300 / 回文字符串 博弈论】
CIKERS Shane 20190613
"Javascript advanced programming" study notes - the constructor and prototype
beeline hiveserver2 start
springboot - Automatically backup mysql data every day
Data Storage Full Solution--Detailed Persistence Technology
Detailed Explanation of Spring Web MVC DispatcherServlet—Official Original
TCP / IP protocol layers structure and function
Command type literal pos: unknown； Fallback type literal pos: unknown] with root cause
Design of multifunctional curtain controller with indoor anti-theft alarm
Daily
More
2024-05-08(18)
2024-05-07(34)
2024-05-06(6)
2024-05-05(0)
2024-05-04(18)
2024-05-03(8)
2024-05-02(0)
2024-05-01(4)
2024-04-30(36)
2024-04-29(5)

Code World

© 2018 codetd.com