First, prepare the host
In order to train and deploy Hadoop eco tuning techniques, you need three virtual machine deployment Hadoop cluster environment, to ensure HA, that is, there is no single point of failure main services, perform basic functions, parameter adjustments are small memory model.
1.1, ready to install package
The host operating system Win7 SP1, memory 16G, 256G of disk to disk SSD
1) virtual machine: VMware® Workstation 15 Pro 15.5.0 build-14665864 Chinese version
2) Linux system: CentOS-8-x86_64-1905-dvd1.iso (Centos 8.0)
1.2, configure the network
After installing VMware, in addition to the host a total of three wireless network card, namely:
1) local network card, 192.168.3.127, mask 255.255.255.0
2) Vmnet1 (used to support host only mode), 192.168.121.1, mask 255.255.255.0
3) Vmnet8 (for supporting the NAT [. 1] mode), 192.168.154.1, 255.255.255.0 mask
In this study, we deploy the virtual machine using NAT mode network.
Start VMware Workstation Pro, click Edit -> Virtual Network Editor ...
1.2.1 View NAT settings
Click "NAT Settings (S) ...", you can view the NAT settings
I.e. NAT gateway is 192.168.154.2, set in the virtual machine 192.168.154 IP segment in the mask is set 255.255.255.0 Gateway 192.168.154.2, virtual machine can access the external network.
1.2.2 Changing settings
If the virtual LAN and IP Vmnet1 modify Wmnet8, the need to modify the information and vmware pro workstation card "Change Settings"
You can click Change settings, note that this requires administrator can set up and will pop up "User Account Control" dialog box for confirmation.
You can click on the "NAT Settings ..." Set Gateway
1.3, server configuration time
The virtual machine host as a time server.
- Right-click "Computer" on the desktop, select "Manage" and then select "service." Will automatically (delayed start) Windows startup type Time Service
- "Start" -> "Run" -> type "regedit" to open the registry.
- HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Services \ W32Time \ TimeProviders \ NtpServer NetSerVer the setting values of 1 Enabled, if no increase Enabled Enabled name, type REG_DWORD, item value of 1.
- HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Services \ W32Time \ Config \ AnnounceFlags set to 5 (5 represents a stable source, default 10)
- If the notebook is unplugged network cable, because the "media probe (Media Sense)" Win7 technology will automatically shut down network service, you can modify the registry to deceive OS continue to provide services
- HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters,添加子项
- 名称:DisableDHCPMediaSense
- Data Type: REG_DWORD (Boolean)
- Value: 1
- Side effects, the notebook hibernation, from a wireless network environment to another wireless network environment, you cheated because DHCP, does not automatically apply the new notebook wake IP, resulting in can not connect wireless network, you need to manually start it and prohibit wireless network, other DHCP network will also need similar operations.
- HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters,添加子项
- Restart Windows Time service, you can use the host as an NTP server.
- Note that the default Windows Time service is Automatic (Delayed Start), which applies to desktop and notebook models, if a Windows server, to modify the "auto" mode.
- Foreign only provide services only time physical NIC, Linux time server IP must correspond to the physical IP network card, in this case the time server IP address is 192.168.3.127
1.4, Centos8 commonly used commands
root run
# Systemctl start foo # run a service
# Systemctl stop foo # stop a service
# Systemctl restart foo # restart a service
# Systemctl status foo # display a service (whether running or not) state
# Systemctl enable foo # enable a service at boot
# Systemctl disable foo # disable a service at boot
Are # systemctl is-enabled foo # View service boot
# Reboot # reboot the host
# Init 0 # immediate shutdown
Second, the installation Centos8.0
2.1, ready to configure the virtual machine foundation
Select "Custom" and click "Next"
Click "Next"
Select "install the operating system later" and click "Next"
Select "Red Hat Enterprise Linux 8 64 Bits", click "Next"
According to individual needs, enter the name of the virtual machine, you can also select the directory to store the virtual machine, click "Next"
Select the number of processors and the number of nuclear host according to the situation and your specific plan, click "Next"
Select or enter the memory for a virtual machine using the host according to the situation and your specific plan, click "Next"
According to the specific needs of select network mode, in this case you need to access the Internet through the host, but also prohibited outside the network to access the virtual machine, so the use of NAT mode, click "Next"
Recommended by automatic mode, click "Next"
Recommended by automatic mode, click "Next"
Select "Create a new virtual disk", click "Next"
Depending on the needs of the disk, enter the size of the disk, because the clone as a seed, be careful not to choose to allocate all the disk space immediately. In order to facilitate copying, select "See Split virtual disk into multiple files." Click "Next"
Click "Next"
Optimize, delete unnecessary hardware USB, sound card, printer, and so on.
Click the "Custom Hardware"
Stepwise selection USB controller, sound card, printer, click "Remove"
Setting a display to save resources, cancel "accelerated 3D graphics", set a resolution of 1280 × 1024
Set the drive using the ISO file Centos8
Click "Close" Revert
Click "Finish." The end of the preparatory phase of the virtual machine
2.2, start the virtual machine
Connect the text, click on the "open this virtual machine"
Note: switch a virtual machine interface and host of characters to press "Ctrl" + "Alt" key
Select the cursor to "Install CentOS Linux 8.0.1905", carriage return
After a quick scroll through a lot of text messages, graphical interface.
Click "Continue", displays summary information node, in this interface, we need to set up a collection of software, general information about disk partitions, date, network and so on.
2.2.1, set up the network
Drag the button, select the network status to "On"
At this point the IP set to DHCP mode, click on the "Configure", set manually assigned IP
"General" page, select the option "Automatically Connect to this network ...", ensure that the automatic start this network settings.
Set in IPV4 Settings in the network to manually specify static IP, according to earlier agreement, the gateway is specified as 192.168.154.2, DNS specify the circumstances the country, such as Tianjin Unicom DNS is 202.99.96.68
Click "Save" to return to the previous interface
The contents of the input box after "Host Name" is planning to modify the server domain name, such as "Hadoop.centos8.cn", click on "Apply"
Click "Done" return to the previous screen.
2.2.2, set the date
At this time, the network had been connected, you can set the time and the time the server at the same time.
First select the correct time zone. Can be selected on the map by selecting the drop-down menu with the mouse.
Now "Network Time" in the "On" state, click the Settings button next to
Increase the time server host, and to cancel the default server centos.pool.ntp.org
Note that the status "Working", if it is , represents the normal time and can access the server, and vice versa if it is necessary to check and set the time server network settings.
Select "OK" to confirm the setting network time server.
Select "Done" Revert
2.2.3, configure disk space
Virtual machines can automatically select the partition by its relatively simple rules, divided into boot, root, swap, main space in the root partition. If it is a physical machine or have special needs, you can choose "Custom"
2.2.4, select the package installed
As a server, generally do not select the GUI, to conserve system resources occupied.
In order to develop and debug, usually selected "Performance Tools" and "Development Tools" can be, other follow-up package can be installed when required by yum.
Click "Done" Revert
2.2.5 to start the installation
Click "Begin Installation" to begin the installation
2.2.6, set the user name and password
Note that if the root password is too simple, it requires two confirmation ( "click Down"), as shown below.
You can then create a normal user
Similarly, if the password is too weak, you need to click twice "Done"
Continue to wait for the completion of the installation progress
2.2.7, the installation is complete restart
Click "Reboot" to restart
2.3 General Configuration
2.3.1, Configuring SecureCRT
Create a file named hadoop of Session, IP address 192.168.154.8, double-click the connection to the server
The first connection dialog box, note: each virtual machine has a separate Hash values.
Select "Aceept & Save"
Enter the correct user name and password. Note: Try not to log in as root user.
Session options can be set after a successful login. Menu "Options" -> "Session Options ..."
Color scheme and configure the rolling area of the buffer
Configuring Character Sets and display fonts
2.3.2, set the ISO or DVD software source source software
2.3.2.1, set the DVD source software
2.3.2.1.1 loading optical drive
Ensure that the virtual machine settings in the device status is "Connected."
# Switch to the root user
its
#Create a directory
mkdir /mnt/centos8
# Mount
mount /dev/sr0 /mnt/centos8/
# Uninstall command
umount /mnt/centos8
2.3.2.1.2, automatically mount cdrom
vi / etc / fstab
Add the line
/dev/sr0 /mnt/centos8 iso9660 auto,ro 0 0
2.3.2.1.3, create a local source
cd /etc/yum.repos.d/
ll
-rw-r--r--. 1 root root 731 Aug 14 14:42 CentOS-AppStream.repo
-rw-r--r--. 1 root root 712 Aug 14 14:42 CentOS-Base.repo
-rw-r--r--. 1 root root 798 Aug 14 14:42 CentOS-centosplus.repo
-rw-r--r--. 1 root root 1320 Aug 14 14:42 CentOS-CR.repo
-rw-r--r--. 1 root root 668 Aug 14 14:42 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root 756 Aug 14 14:42 CentOS-Extras.repo
-rw-r--r--. 1 root root 338 Aug 14 14:42 CentOS-fasttrack.repo
-rw-r--r--. 1 root root 928 Aug 14 14:42 CentOS-Media.repo
-rw-r--r--. 1 root root 736 Aug 14 14:42 CentOS-PowerTools.repo
-rw-r--r--. 1 root root 1382 Aug 14 14:42 CentOS-Sources.repo
-rw-r--r--. 1 root root 74 Aug 14 14:42 CentOS-Vault.repo
Check which sources are available for use
grep -lrn "enabled=1"
CentOS-AppStream.repo
CentOS-Base.repo
CentOS-Extras.repo
The renamed source can be used
mv CentOS CentOS Base.repo Base.repo.bak
mv CentOS-AppStream.repo CentOS-AppStream.repo.bak
mv CentOS-Extras.repo CentOS-Extras.repo.bak
Editing Base and App Stream source
vi CentOS Base.repo
[BaseOS]
name=CentOS-$releasever - Base
baseurl = file: /// mnt / centos8 / BaseOS
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
enabled=1
Save and exit
I saw CentOS-AppStream.repo
[AppStream]
name=CentOS-$releasever - AppStream
baseurl = file: /// mnt / centos8 / AppStream
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
enabled=1
Save and exit
Check local source
yum clean all
yum list | grep gcc style list is shown below
gcc.x86_64 8.2.1-3.5.el8 @AppStream
gcc-c++.x86_64 8.2.1-3.5.el8 @AppStream
gcc-gdb-plugin.x86_64 8.2.1-3.5.el8 @AppStream
libgcc.x86_64 8.2.1-3.5.el8 @anaconda
gcc-gdb-plugin.i686 8.2.1-3.5.el8 AppStream
gcc-gfortran.x86_64 8.2.1-3.5.el8 AppStream
gcc-offload-nvptx.x86_64 8.2.1-3.5.el8 AppStream
libgcc.i686 8.2.1-3.5.el8 BaseOS
2.3.2.2, setting the software source ISO
2.3.2.2.1 loading iso file
#Create a directory
mkdir /mnt/centos8
# Copy iso file to a specific directory, such as /opt/kvm/data/iso/CentOS-8-x86_64-1905-dvd1.iso
# Mount
mount -o loop /opt/kvm/data/iso/CentOS-8-x86_64-1905-dvd1.iso /mnt/centos8
# Uninstall command
umount /mnt/centos8
2.3.2.2.2, automatically mount iso file
vi / etc / fstab
Add the line
/opt/kvm/data/iso/CentOS-8-x86_64-1905-dvd1.iso /mnt/centos8 iso9660 auto,ro 0 0
2.3.2.2.3, create a local source
With 2.3.2.1.3, create a local source
2.3.3 Check time synchronization
The installation process has been set up time and time synchronization server, where only check the configuration file and whether to start service
cat /etc/chrony.conf see the following line:
server 192.168.3.127 iburst
The following command can be Centos8 of chrony time the server is restarted, set to start automatically, view the status
systemctl restart chronyd
systemctl enable chronyd
systemctl status chronyd
Check synchronization and time server Linux case
chronyc sourcestats –v
2.3.4 simplify security settings
Turn off the firewall
# Check the firewall status
systemctl status firewalld.service
# Turn off the firewall
systemctl stop firewalld.service
# Disable the firewall boot
systemctl disable firewalld.service
Close SELINUX
# vi /etc/selinux/config
Edits:
# SELINUX = enforcing # comment out
SELINUX = disabled # increase
Save and exit, take effect after the next restart
If you want to make the configuration take effect immediately, execute
setenforce 0
2.3.5, the management pack mounting portion
yum install expect redhat-lsb-core -y
verification
lsb_release –a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 8.0.1905 (Core)
Release: 8.0.1905
Codename: Core
Core edition
uname –a
Linux hadoop.centos8.cn 4.18.0-80.el8.x86_64 #1 SMP Tue Jun 4 09:19:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
2.3.6, Samba deployment
yum install samba samba-client -y
It has closed the firewall so the most complex issues will not appear
we /etc/samba/smb.conf
The default shared user directory
Also because there is no printer installed, it deletes the shared printer's configuration
Delete [printers] [print $]
Start Service
systemctl enable smb
systemctl enable nmb
systemctl restart smb
systemctl restart nmb
Setting an Access Password
smbpasswd -a ansible
Set the host win7
C:\Windows\System32\drivers\etc\hosts
192.168.154.8 hadoop
192.168.154.11 hadoop11.centos8.cn hadoop11
192.168.154.12 hadoop12.centos8.cn hadoop12
192.168.154.13 hadoop13.centos8.cn hadoop13
Access hadoop in win7
You can access the home directory of the ansible
Third, the clone
3.1, clone node
The shutdown command init 0, after shutdown completes start cloning
Select the virtual machine hadoop, click menu -> Virtual Machine -> Administration -> Clone ...
Click "Next"
Click "Next" to continue
If the new virtual machine running on the machine, in order to save disk, you can select "create a linked clone," whereas if you want to copy to a different server, be sure to "create a full clone."
Click "Next" to continue
Enter a name for the cloned virtual machine, click "Finish" to start cloning the virtual machine.
Click "Close"
3.2, modify the IP address
Start cloning the resulting virtual machine hadoop11
Set the IP and hostname
IP 在vi /etc/sysconfig/network-scripts/ifcfg-ens160
vmware does not regenerate UUID, avoid duplication, need to update this UUID
UUID="…"
Call uuidgen produced a new uuid, replace the original string
Such as:
uuidgen
bd83dded-76d2-4d23-8799-134f8b7eb984
Modify the IP address
IPADDR="192.168.154.8"
for
IPADDR="192.168.154.11"
3.3, modify the host name IP
vi / etc / hostname
hadoop.centos8.cn
change into
hadoop11.centos8.cn
3.4 Restart
Repeat 3.1-3.4 complete hadoop12, cloning and disposed hadoop13
Fourth, the main difference Centos8 and Centos7
4.1, network services
Centos8 / RHEL8 network management-related services has been transferred to NetworkManager, not installed by default network services.
systemctl status NetworkManager can check the service status.
Note: Modifying IP, if only to modify the network card configuration, and then restart NetworkManager, it is not feasible. The right way is to restart the network card:
1, nmcli connection down ens160 stop card
2, nmcli connection up ens160 start card
Note: At this point do not pay attention to the card is a remote management card that way.
Modify the configuration file, there are three
1, we
2, nmcli connection modify ens160 (ens160 network card name)
3, nmtui character editing interface
If, as Centos7 as the use of network services can be installed with yum
yum install network-scripts
But according to official information, future network services will be completely removed from the distribution package.
[1] NAT: Network Address - Search.com, network address translation, can achieve a bandwidth sharing and security.