[Big learning research data] 2. Using VirtualBox Linux cluster simulation

 

1. Set on the host HOST Macbook

Before the instrument has a static IP address of the virtual machine set up, you can log on later by ip address. For convenience, however, was set about, first modify the hosts file in the Mac, so do not enter the ip address when ssh.

sudo vim /etc/hosts

or 

sudo vim /private/etc/hosts

 

This file is actually a two, it is by doing link link. Note To add sudo, run as an administrator, or can not save.

 

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting. Do not change this entry.
##
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost
50.116.33.29 sublime.wbond.net
127.0.0.1 windows10.microdone.cn
# Added by Docker Desktop
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal

192.168.56.100 hadoop100
192.168.56.101 hadoop101
192.168.56.102 hadoop102
192.168.56.103 hadoop103
192.168.56.104 hadoop104
# End of section

 

2. Copy the virtual machine

Then we need a good last time with this virtual machine, out of more than one copy, in order to form a cluster. First off the virtual, in the top right-click, select copy, the following dialog box appears, select all the card I have to re-generate the Mac address, in order to simulate different environments calculator.

 

 

3. Modify each of HOST, IP address

After the copy is complete, remember to log on to the virtual machine, in accordance with the aforementioned method modify a static IP address, so the IP address conflict.

vi /etc/sysconfig/network-scripts/ifcfg-enp0s3

vi /etc/sysconfig/network-scripts/ifcfg-enp0s8

 

In addition, preferably also each Linux virtual machine also set about HOSTNAME, in order to communicate with each other when the virtual machine before they can use the hostname. Need to turn to several machines hostname are set.

[root@hadoop101 ~]# hostnamectl set-hostname hadoop107
[root@hadoop101 ~]# hostname
hadoop107

 

4. xcall let the server cluster run command

Because we have several machines at the same time, if one by one by one login up operation, it is inevitable trouble, you can write a shell script, from which after a launch command, so that all machines are executed much more convenient. Here is an example. I have hadopp100, hadopp101, hadopp102, hadopp103, hadopp104 the five virtual machines. I hope to hadopp100 fortress, unified control of all the other machines. Create a xcall under / user / local / bin file, as follows:

 

touch /user/local/bin/xcall

chmod +x /user/local/bin/xcall

vi /user/local/bin/xcall

 

#!/bin/bash
pcount=$#
if((pcount==0));then
echo no args;
exit;
fi

echo ---------running at localhost--------
$@
for((host=101;host<=104;host++));do
echo ---------running at hadoop$host-------
ssh hadoop$host $@
done
~

 

For example, I use this xcall script calls pwd name on all machines, view the current directory, in turn prompts to enter the password.

[root@hadoop100 ~]# xcall pwd
---------running at localhost--------
/root
---------running at hadoop101-------
root@hadoop101's password:
/root
---------running at hadoop102-------
root@hadoop102's password:
/root
---------running at hadoop103-------
root@hadoop103's password:
/root
---------running at hadoop104-------
root@hadoop104's password:
/root
[root@hadoop100 ~]#

 

5. scp and rsync

Then we talk about scp tool. scp remote copy data between linux. If you want to copy the entire directory, add -r on it.

[root@hadoop100 ~]# ls
anaconda-ks.cfg
[root@hadoop100 ~]# scp anaconda-ks.cfg hadoop104:/root/
root@hadoop104's password:
anaconda-ks.cfg 100% 1233 61.1KB/s 00:00
[root@hadoop100 ~]#

 

 

Also you can use rsync, scp is the case regardless of the target machine, in order to have a copy. rsync is to compare, and then change the copy. If you want something remote copy is relatively large, with rsync faster. Rsync is better not installed by default on centOS, need to install it. In the previous article , we have virtual machines can be networked, so the line is installed on it.

[root@hadoop100 ~]# xcall sudo yum install -y rsync

For example, the Java SDK synchronous machine on hadoop100 up to 102:

[root@hadoop100 /]# rsync -r /opt/modules/jdk1.8.0_121/ hadoop102:/opt/modules/jdk1.8.0_121/

 

 

Well, now the basic tools and build up a clustered environment, you can start hadoop behind the study.