Linux cluster installation and set up Hadoop3.1.2 (6) - to build a fully distributed mode of operation

Chapter 7: fully distributed mode of operation (development focus)

7.1.1 virtual machine is ready

	Host computer	Slave	Slave
CPU name	hadoop104	hadoop105	hadoop106
hostname	hadoop104	hadoop105	hadoop106
IP addresses	192.168.153.104	192.168.153.105	192.168.153.106

7.1.2 cluster distribution write scripts xsync

1. scp (secure copy) secure copy
   (1) scp defined:
   SCP data copy may be implemented between the server and the server. (From server1 to server2)
   (2) Basic syntax

scp -r $pdir/$fname $user@hadoop$host:$pdir/$fname

Recursive command to copy the file path / name of the destination user @ host: the purpose of path / name
(3) the practical operation case
(a) in hadoop104, software hadoop104 in / opt in / module to copy the directory hadoop105.

[zpark@hadoop104 /]$ scp -r hadoop104:/opt/module  hadoop105:/opt/module

(B) on hadoop106, under software / opt on hadoop104 server / module to copy the directory hadoop106.

[zpark@hadoop106 opt]$sudo scp -r hadoop104:/opt/module hadoop106:/opt/module

(C) in the operating software hadoop104 hadoop103 in / opt in / module to copy the directory hadoop104.

[zpark@hadoop103 opt]$ scp -r zpark@hadoop104:/opt/module root@hadoop104:/opt/module

Note: The copied the / opt / module directory, do not forget hadoop104, hadoop105, modify, owner and group owner of all files on hadoop106.

sudo chown zpark:zpark -R /opt/module

And (d) the copies hadoop104 / etc / profile file to hadoop102 the / etc / profile.

[zpark@hadoop104 ~]$ sudo scp hadoop104:/etc/profile 

hadoop102: / etc / profile
(E) hadoop104 in the / etc / profile hadoop103 files are copied to the / etc / profile on.

[zpark@hadoop104 ~]$ sudo scp hadoop104:/etc/profile

hadoop103: / etc / profile
(F) hadoop104 in the / etc / profile hadoop104 files are copied to the / etc / profile on.

[zpark@hadoop104 ~]$ sudo scp hadoop104:/etc/profile 

hadoop104: / etc / profile
Note: Do not forget to copy over the configuration file at the source / etc / Profile
2. Remote synchronization tool rsync
rsync is mainly used for backup and mirroring. Has the speed, avoid copying the contents of the same advantages and supports symbolic links.
rsync and scp difference: do with rsync to copy files faster than scp, rsync only difference file to do the update. scp is to copy all the files in the past.
(1) The basic syntax

rsync -av $pdir/$fname $user@hadoop$host:$pdir/$fname

Command option parameter you want to copy the file path / name of the destination user @ host: destination path / name
Option Parameter Description

Options	Features
-a	Archive copies
-v	The copy process

(2) Case gymnastics
/ opt / directory under the root user (a) the / opt on hadoop104 machine / software directory synchronization server to hadoop102

[zpark@hadoop104 opt]$ rsync -av /opt/software/

hadoop102: / opt / Software
3. XSync cluster distribution scripts
(1) Requirements: cycling copy the file to the same directory for all nodes
(2) needs analysis:
(A) the original copy rsync command:

rsync -av /opt/module  root@hadoop103:/opt/

(B) expected script:
file name xsync to be synchronized
(c) Note: / home / zpark / bin directory script stored, zpark user can execute directly anywhere in the system.
(3) script to achieve
(a) create xsync directory in the root directory, and xsync create a file in the bin directory, file reads as follows:

[zpark@hadoop102 ~]$ vi xsync

Write the following code in the file

#!/bin/bash
#1 获取输入参数个数，如果没有参数，直接退出
pcount=$#
if ((pcount==0)); then
echo no args;
exit;
fi

#2 获取文件名称
p1=$1
fname=`basename $p1`
echo fname=$fname

#3 获取上级目录到绝对路径
pdir=`cd -P $(dirname $p1); pwd`
echo pdir=$pdir

#4 获取当前用户名称
user=`whoami`

#5 循环
for((host=103; host<105; host++)); do
        echo ------------------- hadoop$host --------------
        rsync -av $pdir/$fname $user@hadoop$host:$pdir
done

(B) modify the script has execute permissions xsync

[zpark@hadoop102 ~]$ chmod +x xsync

(C) to distribute the xsync hadoop105, hadoop106

(D) view hadoop105, hadoop106


(E) to copy the Hadoop3.1.2 hadoop105, hadoop106 in

[zhangyong@hadoop104 module]$ xsync hadoop-3.1.2/

(F) to copy the jdk1.8.0_181 hadoop105, hadoop106 in

[zhangyong@hadoop104 module]$ xsync jdk1.8.0_181/

(G) the configuration file to copy hadoop105, hadoop106 in

View hadoop105, hadoop106 if configured, if good, certify that the above configuration is correct, if not please see step by step