HADOOP/HBase dynamically adds nodes

1. Check whether the JDK version is the same, copy the JDK, and configure the JAVA_HOME and CLASSPATH properties of the /etc/profile file

2. Configure the /etc/hosts file, add the computer name of the newly added machine, and modify the file, including the original machine and the new machine

3. Copy hadoop-2.6.0 and hbase-1.0.1.1, and change the file owner and file group of hadoop-2.6.0 and hbase-1.0.1.1 folders to hadoop

4. Configure passwordless login from Master to Slave

5. Set the number of machine handles, modify /etc/security/limits.conf, see http://beyond3518.blog.51cto.com/1360525/1764190

6. Install snappy

    6.1 Install software: Gcc c++, autoconf, automake, libtool, Java 6, JAVA_HOME set, Maven 3, glibc

           Installation method: yum install xxx

    6.2 Install glibc method: 

            6.2.1 Copy the file from /data/server/glibc-2.14.tar.gz to 192.168.9.110, and copy the file to the /usr/local directory

            6.2.2 Enter the /usr/local directory, cd /usr/local

            6.2.3 解压glibc-2.14.tar.gz,  tar -zxvf glibc-2.14.tar.gz

            6.2.4 Create a new directory /usr/local/glibc, mkdir /usr/local/glibc

            6.2.5 Enter /usr/local/glibc directory, cd /usr/local/glibc

            6.2.6 Run the command export CFLAGS="-g -O2"

            6.2.7 运行命令 /usr/local/glibc-2.14/configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin

            6.2.8 Run the command make

            6.2.9 Run the command make install

            6.2.10 Verify that the installation was successful: 

                        Run the command: ll /lib64/libc.so.6 , if the result is displayed /lib64/libc.so.6 -> libc-2.14.so , the installation is successful

 

7. Modify the hadoop slaves configuration file, add new nodes, and send the configuration file to all other nodes

8. Add Datanode

For newly added Datanode nodes, the datanode process needs to be started to add it to the cluster

    8.1 On the newly added node, run sbin/hadoop-daemon.sh start datanode

    8.2 Then check the cluster status through hdfs dfsadmin -report on the namenode

    8.3 Finally , it is necessary to set balancing for the hdfs load, because the default data transmission bandwidth is relatively low, which can be set to 64M, that is, hdfs dfsadmin -setBalancerBandwidth 67108864

    8.4 默认balancer的threshold为10%,即各个节点与集群总的存储使用率相差不超过10%,我们可将其设置为5%

    8.5 然后启动Balancer,sbin/start-balancer.sh -threshold 5,等待集群自均衡完成即可

9. 添加Nodemanager

由于Hadoop 2.X引入了YARN框架,所以对于每个计算节点都可以通过NodeManager进行管理,同理启动NodeManager进程后,即可将其加入集群

    9.1 在新增节点,运行sbin/yarn-daemon.sh start nodemanager即可

    9.2 在ResourceManager,通过yarn node -list查看集群情况

10. 修改hbase的regionservers配置文件,添加新增节点.并把该配置文件发送到所有其他节点

11. 在新增的节点上,启动regionserver. 运行hbase-daemon.sh start regionserver即可

 

12. 修改InputService/StoreService服务器的/etc/hosts文件,添加新节点的域名

 

Guess you like

Origin http://10.200.1.11:23101/article/api/json?id=326831249&siteId=291194637