Dark Horse Big Data Study Notes 0-Environment Configuration

Video p3-p5
https://www.bilibili.com/video/BV1WY4y197g7?p=3

Set up VMware networking

The network segment is set to 192.168.88.0
and the gateway is set to 192.168.88.2

CentOS operating system

Download CentOS-7-x86_64-DVD-1810-7.6.ios
https://pan.baidu.com/s/1_TOxbkMXw4JGAO6rOBCRdA?pwd=9987
bigdata 123


clone 3 copies





of the three virtual machines that pop up when they start Prompt, ignore it,,,

Three virtual machine system configuration: host name, fixed IP, SSH password-free login

The operation of node1, node2, and node3 is the same

open terminal

switch to root user

su -

Modify the hostname of the three virtual machines respectively

hostnamectl set-hostname node1

hostnamectl set-hostname node2

hostnamectl set-hostname node3

restart terminal

Modify IP address

vim /etc/sysconfig/network-scripts/ifcfg-ens33

If the opening is blank, pay attention to check whether the command is wrong

Press i to enter insert mode

After entering, change BOOT
PROTO="dhcp" to BOOTPROTO="static"
and add the following content to the three virtual machines respectively

IPADDR="192.168.88.101"
NETMASK="255.255.255.0"
GATEWAY="192.168.88.2"
DNS1="192.168.88.2"

IPADDR="192.168.88.102"
NETMASK="255.255.255.0"
GATEWAY="192.168.88.2"
DNS1="192.168.88.2"

IPADDR="192.168.88.103"
NETMASK="255.255.255.0"
GATEWAY="192.168.88.2"
DNS1="192.168.88.2"

esc to exit edit
: wq to save and exit

restart network card

systemctl restart network

View IP address

ifconfig

Configure hostname mapping

Windows system C:\Windows\System32\drivers\etc Open the hosts file and add content

192.168.88.101 node1
192.168.88.102 node2
192.168.88.103 node3

The same operation of the three virtual machines
For convenience, use finalshell to remotely control the three virtual machines



Click to modify the hosts file after entering successfully

vim /etc/hosts

Add to

192.168.88.101 node1
192.168.88.102 node2
192.168.88.103 node3

Configuring SSH password-free login

Same operation for three virtual machines

Excuting an order

ssh-keygen -t rsa -b 4096

then all the way back

cd .ssh/ then ll to view private and public keys

implement

ssh-copy-id node1
ssh-copy-id node2
ssh-copy-id node3

After the execution is completed, the password-free login between node1, node2, and node3 will be completed between root users

Execute ssh node1 for password-free login, exit exit

Create hadoop users and configure password-free login

The root user has too much authority, for the sake of safety, create an ordinary user to operate

The three virtual machines operate the same

create hadoop user

useradd hadoop

set password

passwd hadoop

switch to hadoop user

su - hadoop

create ssh key

ssh-keygen -t rsa -b 4096

implement

ssh-copy-id node1
ssh-copy-id node2
ssh-copy-id node3

Note: The following operations are performed under the root user

JDK1.8 environment deployment

1. Create a folder to deploy JDK, install and deploy both JDK and Tomact to: /export/server

mkdir -p /export/server

2. Upload the jdk file

rz

3. View the file list

ll

4. Unzip the JDK file

tar -zxvf jdk-8u361-linux-x64.tar.gz -C /export/server

5. Enter the path /export/server

cd /export/server/

6. View list

ls

7. Configure the soft link of JDK

ln -s /export/server/jdk1.8.0_361 /export/server/jdk

8. View

ls -l

9. Edit the /etc/profile file

vim /etc/profile

10. Add the following content (configure the JAVA_HOME environment variable, add the $JAVA_HOME/bin folder to the PATH environment variable)

export JAVA_HOME=/export/server/jdk
export PATH=$PATH:$JAVA_HOME/bin

11. Effective environment variables

source /etc/profile

12. Configure the soft link executed by java
Delete the java program that comes with the system

rm -f /usr/bin/java

13. Soft link to the java program we installed ourselves

ln -s /export/server/jdk/bin/java /usr/bin/java

14. Perform verification

java -version

javac -version

15. Copy jdk to node2 and node3 (this step is equivalent to 2 and 4)

scp -r jdk1.8.0_361 node2:`pwd`/

scp -r jdk1.8.0_361 node3:`pwd`/

Firewall, SELinux, time synchronization

Clustered software needs to communicate with each other through ports. In order to avoid network failure, we can simply turn off the firewall inside the cluster.
turn off firewall

systemctl stop firewalld

kill autostart

systemctl disable firewalld

Linux has a security module SELinux, which is used to limit the relevant permissions of users and programs to ensure the security and stability of the system.
close SELinux

vim /etc/sysconfig/selinux

Change the seventh line SELINUX=enforcing to

SELINUX=disabled

reboot

init 6

Enter
to view the status

systemctl status firewalld

install ntp software

yum install -y ntp

update time zone

rm -f /etc/localtime;sudo ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

check time

date

synchronised time

ntpdate -u ntp.aliyun.com

Start the ntp service and configure the boot to start automatically

systemctl start ntpd

systemctl enable ntpd

check status

systemctl status ntpd

set snapshot

shutdown ->