Ali cloud has four hands, just to build a cluster, the data on a single node configuration should just copy the past.
Write cluster distribution script xsync
1. scp (secure copy) copy of the security
(1) scp defined:
scp data copy may be implemented between the server and the server. (From server1 to server2)
(2) The basic syntax
Linux scp command is used to copy files and directories between Linux.
scp is a secure copy of the abbreviation, scp remote file copy is based on ssh command for secure login linux system.
scp is encrypted, rcp is not encrypted, scp is an enhanced version of rcp.
grammar
scp [-1246BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file] [-l limit] [-o ssh_option] [-P port] [-S program] [[user@]host1:]file1 [...] [[user@]host2:]file2
Easy writing:
SCP [Optional Parameters] file_source file_target
Parameter Description:
- -1: scp force command protocol ssh1
- -2: scp force command protocol ssh2
- -4: Forced scp command only use IPv4 addressing
- -6: Forced scp the only IPv6 addressing
- -B: use batch mode (without asking transmission during transmission password or phrase)
- -C: Allow compression. (The -C flag to ssh, thereby opening the compression)
- -p: retention of the original file modification times, access times and access rights.
- -q: does not display the transmission progress bar.
- -r: recursive copy the entire directory.
- -v: display output mode in detail. scp and ssh (1) will show the whole process of debugging information. This information is used for debugging connection, authentication, and configuration problems.
- -c cipher: cipher to encrypt the data transmission, this option is passed directly to ssh.
- -F ssh_config: ssh specify an alternate configuration file, this parameter is directly transmitted to ssh.
- -i identity_file: using the read key file transfer from the specified file, this parameter is directly transmitted to ssh.
- -l limit: the user can use a defined bandwidth, to Kbit / s units.
- -o ssh_option: If the parameter passing are accustomed to using ssh_config (5),
- -P port: note the capital P, port is the port number used to specify the data transfer
- -S program: a program specified encrypted transmission is used. This program must be able to understand ssh (1) options.
Format:
scp local_file remote_username@remote_ip:remote_folder 或者 scp local_file remote_username@remote_ip:remote_file 或者 scp local_file remote_ip:remote_folder 或者 scp local_file remote_ip:remote_file
- 1st, 2nd specifies the user name, need to re-enter the password after the command is executed, the first one specifies only a remote directory, file name change, the second specifies the file name;
- The first three or four do not specify a user name, you need to enter a user name and password after the command is executed, the third specifies only a remote directory, file name unchanged, the fourth specifies the file name;
Applications:
scp /home/space/music/1.mp3 [email protected]:/home/root/others/music scp /home/space/music/1.mp3 [email protected]:/home/root/others/music/001.mp3 scp /home/space/music/1.mp3 www.runoob.com:/home/root/others/music scp /home/space/music/1.mp3 www.runoob.com:/home/root/others/music/001.mp3
(3) the practical operation case
(A) on hadoop101, the hadoop101 in / opt / software module in the directory copy to other nodes.
[root@hadoop001 hadoop]# scp -r /opt/module root@hadoop002:/opt/module The authenticity of host 'hadoop002 (47.111.251.212)' can't be established. ECDSA key fingerprint is SHA256:YNsodxgiTBhRi6BaO1LqQrkOeudAVMaRisZJQdaJdAA. ECDSA key fingerprint is MD5:b5:82:88:93:92:6a:01:ed:4d:6f:80:bc:68:1a:e8:c9. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'hadoop002,47.111.251.212' (ECDSA) to the list of known hosts. root@hadoop002's password: Permission denied, please try again. root@hadoop002's password: THIRDPARTYLICENSEREADME.txt 100% 142KB 6.2MB/s 00:00 LICENSE 100% 11KB 1.9MB/s 00:00 NOTICE 100% 13KB 2.2MB/s 00:00 3RDPARTY 100% 11KB 1.8MB/s 00:00 dblook.bat 100% 1387 236.1KB/s 00:00
(B) copy hadoop101 in / etc / profile file to other nodes in the / etc / profile.
[root@hadoop001 hadoop-2.7.2]# scp /etc/profile root@hadoop002:/etc/profile root@hadoop002's password: profile 100% 2012 342.8KB/s 00:00 [root@hadoop001 hadoop-2.7.2]# scp /etc/profile root@hadoop003:/etc/profile root@hadoop003's password: profile 100% 2012 529.6KB/s 00:00 [root@hadoop001 hadoop-2.7.2]# scp /etc/profile root@hadoop004:/etc/profile
Note: Do not forget to copy over the configuration file at the source / etc / profile ,.
2. rsync remote synchronization tool
rsync mainly used for backup and mirroring. Has the speed, avoid copying the contents of the same advantages and supports symbolic links.
rsync and scp difference: do with rsync to copy files faster than scp, rsync only difference file to do the update. scp is to copy all the files in the past.
(1) The basic syntax
rsync -rvl $pdir/$fname $user@hadoop$host:$pdir/$fname
Command option parameter you want to copy the file path / name of the destination user @ host: destination path / name
Options Parameter Description
Table 2-2
| Options |
Features |
| -r |
Recursion |
| -v |
The copy process |
| -l |
Copy symbolic link |
(2) Case gymnastics
/ Opt / directory (a) of the / opt / module hadoop101 directory on the server machine synchronized to the root user hadoop003
[root@hadoop001 hadoop]# vim /etc/hosts [root@hadoop001 hadoop]# rsync -rvl /opt/module/ root@hadoop003:/opt/module The authenticity of host 'hadoop003 (47.111.232.53)' can't be established. ECDSA key fingerprint is SHA256:YKGJjhv5Q/fneno2xR5HWQjFBkeZL09SF4RwtWI8tC4. ECDSA key fingerprint is MD5:30:ad:2b:59:1c:a8:37:d3:bf:69:6b:7d:8f:ee:3b:b8. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'hadoop003,47.111.232.53' (ECDSA) to the list of known hosts. root@hadoop003's password: sending incremental file list created directory /opt/module ./ hadoop-2.7.2/ hadoop-2.7.2/LICENSE.txt hadoop-2.7.2/NOTICE.txt hadoop-2.7.2/README.txt hadoop-2.7.2/hdfs-site.xml hadoop-2.7.2/bin/ hadoop-2.7.2/bin/container-executor hadoop-2.7.2/bin/hadoop
3. xsync cluster distribution script
(1) Requirements: cycling copy the file to the same directory for all nodes
(2) Requirements Analysis:
(A) rsync original copy command:
rsync -rvl /opt/module root@hadoop103:/opt/
(B) expected script:
To synchronize the file name xsync
(3) script to achieve
(A) in the / usr / local / bin directory xsync create a file, the file contents are as follows:
[@ hadoop002 the root Module1] # CD / usr / local / bin [@ hadoop002 the root bin] # LL Total. 4 -rw-R & lt - r--. 1 the root the root 498 Jan 14 22:57 XSync [the root @ hadoop002 bin] # 777 XSync the chmod [@ hadoop002 the root bin] # LL Total. 4 -rwxrwxrwx the root the root. 1 Jan 14 22:57 498 XSync [the root @ hadoop002 bin] # pwd / usr / local / bin [the root @ hadoop002 bin] # CAT XSync # ! / bin / the bash # . 1 acquires the number of input parameters, without parameters, exit Pcount = $ # IF ((== 0 Pcount)); The then echo NO args; Exit; Fi # 2 acquires the file name P1 = $. 1 fname = `the basename $ P1 ` echo fname = $ fname # . 3 acquires the parent directory to the absolute path PDir = `CD -P $ (dirname $ P1 ); pwd` echo PDir = $ PDir # . 4 acquires the current user name user = `whoami` # . 5 cycles for ((= Host. 3; Host <. 5; Host ++)); do echo ---------- hadoop --------- $ Host -------------- rsync -rvl $ pdir / $ fname $ the User@hadoop00$host:$pdir done [root@hadoop002 bin]#