Big Data learning early experience: Linux Shell to deploy learning basic programming + + hadoop cluster

Blog has been nine days since the last time, simply record the learning process these days

2020-02-15 10:38:47

---

A, Linux learning

About Linux commands, before I had learned the part, so more learning linux this time is to learn Linux system installation and configuration a little more, some of the more commonly used commands enough to record the following installation considerations when configuring Linux system.

 

 

 

 Here configured virtual machine memory for 4g

CentOS-6.5-x86_64-minimal.iso use mapping files

Upon entering linux system, the host name of the VM needs to be modified to the name you want, but also the preparation of a good network and IP

View the vi editor to modify #cat

cat /etc/sysconfig/network

vi /etc/sysconfig/network

 

 

 

 Network Setting

 

 

 

vi /etc/sysconfig/network-scripts/ifcfg-eth0

Need to modify several places

 

 

 

 

• ONBOOT = yes: this card represents the start
• BOOTPROTO = static: Specifies a static routing protocol, can be given a fixed IP
• HWADDR: MAC address represents a virtual machine, the virtual machine needs to be consistent with the current MAC address
• IPADDR: indicates the IP address of the virtual machine
• GATEWAY: Gateway represents a virtual machine, usually the last digit IP address becomes 2
• NETMASK: subnet mask is a virtual machine, usually 255.255.255.0
• DNS1: represents a DNS resolver, where the use of Google's free DNS server 8.8.8.8 (w corresponding to the DNS can also be set to the PC computer)

Find locations are as follows HWADDR

 

 

 

 This will copy the MAC address can be.

 

Next, find the IP address range

 

 

 

 My IP address can only be in 192.168.42.128 - 192.168.42.254 this section

GATEWAY values ​​usually xxx.xxx.xx.2 me is 192.168.42.2

After good preparation, execute reboot command to restart the virtual machine, use the command ifconfig to view ip information

 

Last connection WLAN network share on the local network

To access the network

 

 

 

 输入命令 ping www.baidu.com

 

 此时，就是虚拟机就可以上网了。

二、Shell编程

这里推荐几个网址进行shell学习

https://www.runoob.com/linux/linux-shell.html

https://blog.csdn.net/weixin_45093060/article/details/94594130

https://blog.csdn.net/happiness_llz/article/details/82809789

http://c.biancheng.net/shell/

三、hadoop集群部署

第一次接触hadoop，部署了两天才部署成功，此过程是相当煎熬的(T＿T)/~~

这里也只是简单写出我认为比较麻烦的步骤与需要注意的地方。

1.ssh免密登录功能的配置

！需要在每台虚拟机上都操作一遍

ssh-keygen -t rsa　　#创建密钥

ssh-copy-id hadoop02　　#分享到免密登录的虚拟机

2.在执行一个jar包时，可能会出现系统内存和资源分配不足的情况而无法将任务执行完，卡死

或者是nodemanager节点无法启动 || 启动后自动结束的情况

在 yarn-site.xml 中添加

 

    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>2048</value>
    </property>


    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>2048</value>
    </property>


    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>1</value>
    </property>

已上这些代码，可以解决这些问题。