linux cluster server construction
-
Cloning of virtual machines
- A full clone must be used to clone an existing virtual machine in the shutdown state
- Full clone means that the contents of the two machines are exactly the same, but some things cannot be the same
- Everything in Linux is a file For any attribute modification, the final modification is the permanent effect of the file. The file must be modified
#ip /etc/sysconfig/network-scripts/ifcfg-eth0 1、删除mac地址：HWADDR 2、修改IP： IPADDR #网卡MAC地址 1、关机状态下，右键-->设置-->网络适配器-->高级-->生成一个新的mac地址 2、删除mac配置文件 该文件在下次重启的时候 如果不存在 系统就会根据设置自动重新生成一个 rm -rf /etc/udev/rules.d/70-persistent-net.rules #主机名 hostname vim /etc/sysconfig/network hostname=node-2 reboot重启机器 -
Detailed explanation of IP configuration file
/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0 #网卡的设备名称 TYPE=Ethernet UUID=26e46d47-e806-4beb-918d-bee29df0b235 ONBOOT=yes #网卡服务是否开机启动 NM_CONTROLLED=yes BOOTPROTO=none #ip配置方式 dhcp static none HWADDR=00:0C:29:9E:B4:51 #网卡的mac地址 此文件中该属性意义不大 可以删除 IPADDR=192.168.227.151 #ip地址 PREFIX=24 # 11111111 11111111 11111111 00000000 子网掩码 # NETMASK=255.255.255.0 GATEWAY=192.168.227.2 # 网关 DNS1=192.168.227.2 # dns1 网关进行解析 DOMAIN=114.114.114.114 # dns2 公网的dns进行解析 DEFROUTE=yes IPV4_FAILURE_FATAL=yes IPV6INIT=no NAME="System eth0" -
Linux service opening and closing and self-starting settings
service 服务名 start|stop|status 启动|关闭|服务状态 #service命令是单次控制服务状态 并不能影响下次开启是否启动 chkconfig 服务名 on|off 开机自启|开机不启 -
Firewall(All 3 servers need to be executed)
#防火墙iptables service iptables stop chkconfig iptables off #防火墙selinux 修改配置文件之后 在下次重启的时候生效 vim /etc/selinux/config SELINUX=disabled -
Host name host mapping (hosts file) (All 3 servers need to be executed)
-
When we enter the host name or domain name, the system willFirst go to the local hosts file to find the mapping relationship See what the corresponding ip is
If there is a mapping relationship, access the machine according to the corresponding ip
-
If the local hosts file is not mapped, the system will visitDNS server for domain name resolution
If yes, visit according to the corresponding ip
-
If the above two steps are not resolved, it means that it can only be accessed through ip.
#linux hosts文件： /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.227.151 node-1 192.168.227.152 node-2 192.168.227.153 node-3 #windosws hosts文件 C:\Windows\System32\drivers\etc\hosts 192.168.227.151 node-1 192.168.227.152 node-2 192.168.227.153 node-3 -
-
Password-free login between linux systems
- Knowledge point: SSH protocol (one-way verification process)
- Tools: linux system built-in software: openSSH (sshd)
#step1 在node-1上使用命令生成一对公钥和私钥 ssh-keygen 四下回车 #查看一下生成的公钥和私钥 [root@node-1 .ssh]# ll total 12 -rw------- 1 root root 1675 May 13 22:12 id_rsa #私钥 -rw-r--r-- 1 root root 393 May 13 22:12 id_rsa.pub #公钥 -rw-r--r-- 1 root root 404 May 13 22:05 known_hosts [root@node-1 .ssh]# pwd #本地公钥保存的路径 /root/.ssh #step2 在node-1上把公钥拷贝给其他机器 该机器就是你需要远程免密登录的机器 ssh-copy-id node-2 #step3 后续登录node-2就不需要输入密码了 ssh node-2 #如果退出返回本机 输入命令 exit-
Note: You also need to configure the password-free login when you log in without password.
[root@node-1 ~]# ssh-copy-id node-1 root@node-1's password: -
It is required in the course to at least get through the password-free login in the following directions
node-1----->node-1 node-1----->node-2 node-1----->node-3
-
Remote copy files-scp
- If the password-free login is not configured, you need to enter the password when scp is configured. If the password-free login is configured, the password is omitted.
#把文件从本机文件系统拷贝到其他机器上 scp 1.txt root@node-2:/root/ scp -r test/ root@node-2:/root/ #把文件从其他机器上拷贝到本机文件系统 scp root@node-2:/root/1.txt ./ #注意事项 1、之所以在scp的时候不需要输入密码 原因：是因为配置了机器间免密登录 否则的话是需要输入密码的 2、之所以在sco的时候可以写node-1 node-2主机名 原因：是以为我们配置了hosts文件。否则输入IP -
Linux time synchronization problem
#查看linux系统当前时间 date #使用命令把本机时间和国家授时中心的时间同步 #step1 使用yum命令在线安装 yum install -y ntpdate #step2 和服务器进行时间同步 ntpdate ntp3.aliyun.com #阿里云免费的ntp授时地址 https://eallion.com/aliyun-ntp/-
note:There are two forms of time synchronization: ntpdate ntpd
ntpdate 是一种命令 需要自己手动的执行 跟服务器进行同步 后面如果有了偏差继续手动执行 ntpd 是一种软件服务 配置好之后 可以跟服务器进行通信 不断修正时间 保持一致 -
Recommendation: Use ntpdate for time synchronization in the course. The effect of immediate synchronization will be executed again if the time is inconsistent.
-
If the real server is in the enterprise, you can consider configuring the ntpd service to run continuously to correct the time.
-
Installation of linux commonly used software
-
JDK（3 machines installed）
#Centos系统自带了OpenJDK 把其卸载 安装Oracle JDK（甲骨文） #step1:从已经安装的rpm包中找出jdk相关的包 [root@node-2 ~]# rpm -qa |grep java java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el6_8.x86_64 tzdata-java-2016j-1.el6.noarch java-1.7.0-openjdk-1.7.0.131-2.6.9.0.el6_8.x86_64 #step2:卸载openJDK相关的rpm包 rpm -e --nodeps java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el6_8.x86_64 tzdata-java-2016j-1.el6.noarch java-1.7.0-openjdk-1.7.0.131-2.6.9.0.el6_8.x86_64 #step3:统一制定安装软件的相关目录 mkdir -p /export/servers #软件安装 mkdir -p /export/software #安装包 mkdir -p /export/data #数据存放 #step4: 安装oraclejdk 上传安装包到/export/servers jdk-8u65-linux-x64.tar.gz #要求JDK必须是1.8版本 涉及多台机器安装同一个软件 常见的做法是在一台机器安装配置 然后scp安装包给其他机器。 #step5:解压安装包到当前路径下 tar -zxvf jdk-8u65-linux-x64.tar.gz #step6:配置jdk的环境变量 vim /etc/profile export JAVA_HOME=/export/servers/jdk1.8.0_65 export PATH=$PATH:$JAVA_HOME/bin export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar #step7:使用source命令加载环境变量文件 让其立即生效 也可以重启生效 source /etc/profile #step8:把jdk安装包从node-1 scp给其他机器 scp -r /export/servers/jdk1.8.0_65/ root@node-2:/export/servers/ scp -r /export/servers/jdk1.8.0_65/ root@node-3:/export/servers/ #step9:别忘了其他机器上也需要配置jdk的环境变量 scp /etc/profile root@node-2:/etc/ source /etc/profile #最终验证是否安装成功 [root@node-2 servers]# java -version java version "1.8.0_65" Java(TM) SE Runtime Environment (build 1.8.0_65-b17) Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode) [root@node-2 servers]# -
mysql(Install on node-1)
Username: root Password: hadoop
#在线安装mysql yum install -y mysql mysql-server mysql-devel #首次启动mysql /etc/init.d/mysqld start #登录mysql的控制台 初始化设置root用户的密码 授予远程登录的权限 mysql mysql>; USE mysql; mysql>; UPDATE user SET Password=PASSWORD('hadoop') WHERE user='root'; mysql>GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'hadoop' WITH GRANT OPTION; mysql>; FLUSH PRIVILEGES; #以后mysql的启动关闭 使用下述命令 service mysqld start|stop|status #建议：把mysqld服务设置为开机自启动 chkconfig mysqld on #在大数据中 不需要特意把mysql的编码改为utf-8 后面涉及软件 谁需要中文支持 给其创建对应的数据库 支持utf-8
rpm、yum
-
rpm
-
RPM is the RH package manager responsible for viewing, installing and uninstalling the software.
-
The biggest drawback:First download the rpm package and install it manually. You need to resolve the dependencies between the software yourself.
如果某个软件有依赖 需要先把依赖安装好 再安装本软件。 -
This method is rarely used in enterprises for software installation. It is recommended to use yum for online installation.
-
Use the command
#查看rpm包的安装信息 rpm -qa |grep 软件名称 rpm -qi rpm包名 [root@node-1 ~]# rpm -qi mysql-server-5.1.73-8.el6_8.x86_64 #rpm的安装 rpm -ivh rpm包名 #rpm的卸载 忽略依赖卸载 只卸载自己本身 rpm -e --nodeps rpm包名
-
-
yum
-
Introduction
Yum（全称为 Yellow dog Updater, Modified）是一个在Fedora和RedHat以及CentOS中的Shell前端软件包管理器。基于RPM包管理，能够从指定的服务器自动下载RPM包并且安装，可以自动处理依赖性关系，并且一次安装所有依赖的软件包，无须繁琐地一次次下载、安装。 -
advantage:Automatic installation to resolve dependencies。
-
How to realize this function?
#yum是否强大取决于yum源。 1、有很多rpm包 2、rpm包之间的依赖 #yum源分类 1、公网免费开源的yum源。 不需要配置 Centos中内置了yum源镜像列表。 联网就可以搜索到国内免费yum源。 断网就凉凉。 2、自己配置本地yum源。 -
yum command
#安装软件 yum install -y 软件名 #-y表示不提示自动直接安装 #其他命令 yum remove -y 软件名 #卸载 yum repolist all #查看当前可用的yum源信息 yum clean all #清楚yum缓存信息
-
-
Make a local yum source (Understand the principle, you can not build it yourself）
-
Need raw materials: rpm package, dependency between rpm package
-
In the installation image of CentOS
- There are packages----->packages
- There are package dependencies---->repodata
You can use the mirror file to temporarily make a local yum source.
#创建一个文件夹 用于存放rpm包和依赖 mkdir /dev/centios #挂载路径 用于解析光驱中的数据 mkdir /mnt/local_yum #本地yum源的文件夹 #挂载光驱中的镜像文件 挂载到/dev/centios mount -o loop /dev/cdrom /dev/centios #把挂载之后的文件复制到本地yum源文件中 cp -r /dev/centios/* /mnt/local_yum/ #修改yum的配置文件 cd /etc/yum.repos.d/ #在这个路径下 所有.repo结尾的文件都是yum有效配置文件 里面记录了yum源地址 rename .repo .repo.bak *.repo cp CentOS-Media.repo.bak CentOS-Local.repo vim CentOS-Local.repo [local] name=this is my local yum yuan baseurl=file:///mnt/local_yum/ gpgcheck=1 enabled=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6Note: After the exercise, don't forget to restore the configuration of the network yum source.
rename .repo.bak .repo *.bak -
-
Expansion: making local LAN yum source
- Core: Making yum sources on one machine can only meet the needs of this machine. Other machines in the local area network still need to be connected to the Internet before they can be used.Use a web server to expose the local yum source through the http protocol so that other machines in the LAN can also access it.
#安装web服务器 httpd(Apache Server 轻量级web服务器) 注意把防火墙关了 yum install -y httpd #启动 service httpd start #httpd 项目根目录/var/www/html ln -s /mnt/local_yum/ /var/www/html/local #在其他的机器上配置的yum文件 [local] name=this is my local yum yuan baseurl=http://node-2/local/ gpgcheck=1 enabled=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 -
New disk for virtual machine (understand)
软件右键选择增加一个新的磁盘 确定大小 分区 格式化 挂载 详细步骤可以参考附件资料。 -
crontab scheduled scheduling
- It is a linux gadget that helps us execute the specified shell script in a regular cycle.
#语法 crontab -e 编辑定时任务 -l 查看定时任务 -r 删除定时任务 -u 指定给哪个用户配置 如果不指定 给自己配置 [root@node-1 ~]# crontab -l no crontab for root #当前root用户没有定时任务 * * * * * shell command 分 时 日 月 周 待调度的shell命令或脚本 * * * * * date >> /root/cron.txt #每隔一分钟执行后面的脚本或者命令- Scheduled tasks are bound to be involved in big data projects. Usually use mature open source scheduling software for task scheduling.
Apache oozie、 azkaban