Zookeeper installation and tuning deployment reference documents (Linux)

1. Installation environment preparation

1.1 Host environment preparation

1.1.1. Close selinux

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0

1.1.2. Software download

apache-zookeeper-3.6.1-bin.tar.gz: download link

1.1.3. Deployment planning

Software installation path /usr/local/zookeeper
port planning 2192

1.1.4. System host time, time zone, system language

 This section requires operations based on actual conditions
 Modify time zone

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

 Modify the system language environment

echo 'LANG="en_US.UTF-8"' >> /etc/profile && source /etc/profile

 Configure host NTP time synchronization

yum -y install ntp
systemctl enable ntpd && systemctl start ntpd
echo 'server ntp1.aliyun.com' >> /etc/ntp.conf
echo 'server ntp2.aliyun.com' >> /etc/ntp.conf

2. Zookeeper installation and deployment

2.1 Zookeeper dependent installation and deployment

 Add users and user groups (please define your own user name)

groupadd -r middleware && useradd -s /sbin/nologin -r -M -g middleware middleware

 JDK installation and deployment

tar -zxvf jdk-8u231-linux-x64.tar.gz -C /usr/local/
cat >>/etc/profile<<EOF
export JAVA_HOME=/usr/local/jdk1.8.0_231
export JRE_HOME=\${JAVA_HOME}/jre
export CLASSPATH=.:\${JAVA_HOME}/lib:\${JRE_HOME}/lib
export PATH=\${JAVA_HOME}/bin:\$PATH
EOF
source /etc/profile
java -version

 Download the apache-zookeeper-3.6.1-bin.tar.gz installation package, and unzip and install

yum -y install gcc gcc-c++ automake autoconf libevent-devel libevent make wget net-tools
cd /opt
wget https://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.6.1/apache-zookeeper-3.6.1-bin.tar.gz
tar -zxvf apache-zookeeper-3.6.1-bin.tar.gz -C /usr/local/
cd /usr/local/
mv apache-zookeeper-3.6.1-bin zookeeper
mkdir -p zookeeper/data/zookeeper
mkdir zookeeper/dataLog
cd zookeeper/conf
cp zoo_sample.cfg zoo.cfg

 Modify the zookeeper data storage path and connection port

vi zoo.cfg
dataDir=/usr/local/zookeeper/data/zookeeper
dataLogDir=/usr/local/zookeeper/dataLog
clientPort=2192
chown -R middleware:middleware /usr/local/zookeeper

 Configure Zookeeper environment variables

cat >>/etc/profile<< EOF
export PATH="\$PATH:/usr/local/zookeeper/bin"
EOF
source /etc/profile

2.2 Configure zookeeper system service

2.2.1. Add system services for 6 systems

1. Add firewall policy
(1) All machines are accessible

iptables -A INPUT -p tcp --dport 2192 -j ACCEPT
service iptables save

(2) The specific IP 192.168.31.130 can access port 2192 of the machine

iptables -A INPUT -p tcp -s 192.168.31.130 --dport 2192 -j ACCEPT
service iptables save

2. Add the zookeeper system service startup script

cd /usr/local/zookeeper/bin/
sed -i '77aJAVA_HOME="/usr/local/jdk1.8.0_231"' zkEnv.sh
vi /etc/init.d/zookeeper
#!/bin/bash
#
# zookeeper  start/stop the zookeeper daemon
#
# chkconfig: 345 80 20
# description: zookeeper is a message server.
#
ZOOKEEPER_HOME=/usr/local/zookeeper
PIDFILE=/usr/local/zookeeper/data/zookeeper/zookeeper_server.pid

case $1 in
    start)
        if [ -f $PIDFILE ]
        then
                echo "$PIDFILE exists, process is already running"
        else
                echo "Starting zookeeper server..."
                sudo -u middleware $ZOOKEEPER_HOME/bin/zkServer.sh start
        fi
        ;;
    stop)
        if [ ! -f $PIDFILE ]
        then
                echo "$PIDFILE does not exist, process is not running"
        else
                sudo -u middleware $ZOOKEEPER_HOME/bin/zkServer.sh stop
        fi
        ;;
    status)
        if [ ! -f $PIDFILE ]
        then
                echo "$PIDFILE does not exist, process is not running"
        else
                sudo -u middleware $ZOOKEEPER_HOME/bin/zkServer.sh status
                echo "Zookeeper service is running..."
        fi
        ;;
    restart)
        sudo -u middleware $ZOOKEEPER_HOME/bin/zkServer.sh restart
        ;;
    *)
        echo "Please use start|stop|status|restart as first argument"
        ;;
esac

3. Configure zookeeper system service and self-start

chmod +x /etc/init.d/zookeeper
chkconfig --add zookeeper && chkconfig zookeeper on
chkconfig --list zookeeper

4. Start and stop zookeeper service

service zookeeper start
ps -ef|grep zookeeper
service zookeeper stop

2.2.2. Add system services for 7 systems

1. Add firewall policy
(1) All machines are accessible

firewall-cmd --permanent --zone=public --add-port=2192/tcp
firewall-cmd --reload

(2) The specific IP 192.168.31.130 can access port 2192 of the machine

firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.31.130" port protocol="tcp" port="2192" accept"
firewall-cmd --reload

(3) The specific IP segment 192.168.142.0/24 can access port 2192 of this machine

firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.142.0/24" port protocol="tcp" port="2192" accept"
firewall-cmd --reload

2. Add the zookeeper system service startup script to
obtain the current server PATH path information, and add this information to the zookeeper system service

echo $PATH
/usr/local/jdk1.8.0_231/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
cat >/usr/lib/systemd/system/zookeeper.service<<EOF
[Unit]
Description=Zookeeper
After=network.target

[Service]
Type=forking
Environment=ZOO_LOG_DIR=/usr/local/zookeeper/logs
Environment=PATH=/usr/local/jdk1.8.0_231/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
PIDFile=/usr/local/zookeeper/data/zookeeper/zookeeper_server.pid
ExecStart=/usr/local/zookeeper/bin/zkServer.sh start
ExecStop=/usr/local/zookeeper/bin/zkServer.sh stop
ExecRestart=/usr/local/zookeeper/bin/zkServer.sh restart
User=middleware
Group=middleware

[Install]
WantedBy=multi-user.target
EOF

3. Configure zookeeper system service and self-start

systemctl daemon-reload
systemctl enable zookeeper.service

4. Start and stop zookeeper service

systemctl start zookeeper
ps -ef|grep zookeeper
systemctl stop zookeeper

3. Zookeeper reinforcement

3.1 Start by user with minimal permissions

 Please define your own user name

groupadd -r middleware && useradd -s /sbin/nologin -r -M -g middleware middleware

3.2 Prevent DOS***

 Limit the maximum number of connections of the zookeeper client.

vi /usr/local/zookeeper/conf/zoo.cfg
maxClientCnxns=60

3.3 Modify the default port 2181

 By default, zookeeper uses port 2181 by default, please modify the default listening port, for example, 2192 is used in this document

vi /usr/local/zookeeper/conf/zoo.cfg
clientPort=2192

3.4 Disable the management console

 If you don’t need to use zookeeper's management console, it is recommended to disable it (zookeeper's management console is started by jetty, the default is http, which has certain information leakage and security risks.)
 Operation guidance:
in the bin/zkServer.sh file Will be as follows

vi /usr/local/zookeeper/bin/zkServer.sh
start)
    echo  -n "Starting zookeeper ... "
    if [ -f "$ZOOPIDFILE" ]; then
      if kill -0 `cat "$ZOOPIDFILE"` > /dev/null 2>&1; then
         echo $command already running as process `cat "$ZOOPIDFILE"`.
         exit 1
      fi
    fi
    nohup "$JAVA" $ZOO_DATADIR_AUTOCREATE "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" \
"-Dzookeeper.log.file=${ZOO_LOG_FILE}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \
修改为（即在nohup这一行，添加 "-Dzookeeper.admin.enableServer=false"）
start)
    echo  -n "Starting zookeeper ... "
    if [ -f "$ZOOPIDFILE" ]; then
      if kill -0 `cat "$ZOOPIDFILE"` > /dev/null 2>&1; then
         echo $command already running as process `cat "$ZOOPIDFILE"`.
         exit 1
      fi
    fi
    nohup "$JAVA" $ZOO_DATADIR_AUTOCREATE "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" \
    "-Dzookeeper.log.file=${ZOO_LOG_FILE}" "-Dzookeeper.admin.enableServer=false" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \

3.5 Log cleanup

 It is recommended to set up a regular cleanup function for zookeeper logs, and clean up the log policy in the configuration file, as shown below:

vi /usr/local/zookeeper/conf/zoo.cfg
autopurge.snapRetainCount=10
autopurge.purgeInterval=24
参数说明：
autopurge.snapRetainCount=10  //保留多少个快照
autopurge.purgeInterval=24     //多少小时清理一次

3.6 Configure transaction log and snapshot log separation

vi /usr/local/zookeeper/conf/zoo.cfg
dataDir=/usr/local/zookeeper/data/zookeeper
dataLogDir=/usr/local/zookeeper/dataLog

3.7 Add authorized access to the specified IP of zookeeper

 Zookeeper allows unauthorized access by any client by default, which poses a great security risk. The specific connection instructions are as follows:

/usr/local/zookeeper/bin/zkCli.sh -server 127.0.0.1:2192
WatchedEvent state:SyncConnected type:None path:null    //敲回车

 Waiting for input operation instructions, such as creating user, authorization, etc.

[zk: 127.0.0.1:2192(CONNECTED) 0]

 getAcl / means to view the current permissions quit means to exit the client connection

[zk: 127.0.0.1:2192(CONNECTED) 3] getAcl /
'world,'anyone
: cdrwa

 Add accessible IP, a group of accessible IPs are separated by symbols, the format is as follows

[zk: 127.0.0.1:2192(CONNECTED) 3] 
setAcl / ip:192.168.31.130:cdrwa,ip:127.0.0.1:cdrwa

 Check whether the permissions are added successfully

[zk: 127.0.0.1:2192(CONNECTED) 3] getAcl /
'ip,'192.168.31.130
: cdrwa
'ip,'127.0.0.1
: cdrwa

 Rollback method

[zk: 127.0.0.1:2192(CONNECTED) 3] setAcl / world:anyone:cdrwa

 There are 4 ways to authenticate zookeeper identity:
(1) world: the default method, equivalent to the world can access
(2) auth: on behalf of the authenticated user (cli can add the current context through addauth digest user: pwd the authorized user)
(. 3) Digest: i.e. username: password authentication in this way, which is the most common service system, with username: password MD5 string to generate a string, then the string is used as an ACL ID , The authentication is carried out by sending username:password in plain text. When used in ACL, the expression is username:base64, base64 is the encoding of the SHA1 digest of the password;
(4)ip: use IP address authentication

 ID authorization object ID refers to the user or an entity given the authority, for example: IP address or machine, authorization mode authorization objects are:
(1) IP: usually an IP address or IP segment, such as "192.168.29.100" or "192.168.29.100/110"
(2) Digest: Custom, usually "username:BASE64(SHA-1(username:password))", for example "foo:kWN6aNsbjcKWpqjiV7cg0N24raU="
(3)Word has only one ID: "anyone "
(4) Super: Consistent with Digest mode
 Zookeeper supports five types of permissions (where delete refers to the delete permissions on child nodes, and the other four permissions refer to the operating permissions on its own node)

cdrwa:
        create: 可以创建子节点；
        read: 可以获取节点数据以及当前节点的子节点列表；
        write: 可以为节点设置数据；
        delete: 可以删除子节点；
        admin: 可以为节点设置权限。

3.8 Account and authentication

1、通过zkCli.cmd 进入zookeeper客户端
/usr/local/zookeeper/bin/zkCli.sh -server 127.0.0.1:2192
WatchedEvent state:SyncConnected type:None path:null    //敲回车
2、使用auth方式加密，添加用户名crm和密码pwd
addauth digest crm:crm#pwd
3、授予/dubbo auth权限
setAcl /dubbo auth:crm:crm#pwd:rwadc
4、查看目录加密后的权限
getAcl /dubbo

3.9 Configure firewall policy

 According to different operating systems, refer to chapter 2.2 (note that if you are configuring a specific IP address for access, you must also add the specified IP added in chapter 3.7)

3.10 Regular upgrades

 Use the latest official stable version

4. Zookeeper optimization

4.1 Optimizing kernel parameters

cat >>/etc/sysctl.conf<<EOF
fs.file-max = 6815744
net.ipv4.tcp_tw_reuse = 1 
net.ipv4.tcp_tw_recycle = 1 
net.ipv4.tcp_fin_timeout = 30 
net.ipv4.ip_local_port_range = 10000 65000 
net.ipv4.tcp_max_syn_backlog = 8192 
net.ipv4.tcp_max_tw_buckets = 10000
net.core.somaxconn=4000
net.ipv4.tcp_syncookies = 1
net.core.netdev_max_backlog = 262144
net.ipv4.tcp_max_orphans = 262144
EOF
sysctl -p

4.2 System resource limit

cat >>/etc/security/limits.conf<<EOF
* soft nofile 65525
* hard nofile 65525
* soft nproc 65525
* hard nproc 65525
EOF

5. End