Question guide:
1. How to offline a datanode node?
2. What should I do if the disk of a datanode node is broken?
3. What should I do if the NameNode server fails?
4. How does Hadoop perform operating system tuning?
Hadoop daily operation and maintenance problems and their solutions
1. How to offline a datanode?
When the server where a datanode is located fails or is about to be decommissioned, you need to offline this node in Hadoop. The process of offline a datanode is as follows.
(1) Modify the hdfs-site.xml file
Find the namenode node configuration file /etc/hadoop/conf/hdfs-site.xml with the following options:
|
1
2
3
4
|
<property>
<name>dfs.hosts.exclude</name>
<value>/etc/hadoop/conf/hosts-exclude</value>
</property>
|
(2) Modify the hosts-exclude file
Perform the following operations to add the datanode host name that needs to be offline in hosts-exclude:
vi /etc/hadoop/conf/hosts-exclude
172.16.213.188
(3) Refresh configuration
Execute the following command as hadoop user on namenode to refresh hadoop configuration
[hadoop @ purposeodemaster ~] $ hdfs dfsadmin-refreshNodes
(4) Check whether the offline is completed
Execute the following command to check whether the offline is completed:
[hadoop@namenodemaster ~]$hdfs dfsadmin-report
You can also access the Web interface through the 50070 port of the NameNode to view the HDFS status. You need to focus on the number of decommissioned nodes, as well as the number of replicated blocks and the progress.
2. How to add a DataNode node to the Hadoop cluster?
When the cluster resources are insufficient, you need to add several machines to join the cluster. This is one of the most common processing methods for Hadoop operation and maintenance. So how to add the newly added server to the Hadoop cluster mainly includes the following steps.
(1) Deploy Hadoop environment on new nodes
After the new node is installed, a series of operations must be performed, such as basic system optimization settings, Hadoop environment deployment and installation, and JDK installation. These basic tasks need to be completed in advance.
(2) Modify the hdfs-site.xml file
Check the /etc/hadoop/conf/hdfs-site.xml file on the NameNode and find the following:
|
1
2
3
4
|
<property>
<name>dfs.hosts</name>
<value>/etc/hadoop/conf/hosts</value>
</property>
|
(3) Modify the hosts file
Modify the /etc/hadoop/conf/hosts file on the NameNode and add the new node host name, as follows:
vi /etc/hadoop/conf/hosts
slave0191.iivey.cloud
Finally, synchronize the configuration to the machines of all DataNode nodes.
(4) Make the configuration effective
After adding a new node, in order for the NameNode to recognize the new node, you need to refresh the configuration on the NameNode and perform the following operations:
[hadoop @ purposeodemaster ~] $ hdfs dfsadmin-refreshNodes
(5) Start the dn service on the new node
After the NameNode is configured, you need to start the DataNode service on the new node, and perform the following operations:
[[email protected] ~]$ hdfs--daemon start datanode
In this way, a new node is added to the cluster. This mechanism of Hadoop can add or delete any node without affecting the operation of the existing cluster, which is very convenient.
3. What should I do if the disk of a datanode is broken?
If the disk of a datanode node fails, the node will not be able to perform write operations and cause the datanode process to exit. For this problem, you can solve this problem as follows:
First, check the corresponding dfs.datanode.data.dir parameter setting in the /etc/hadoop/conf/hdfs-site.xml file on the failed node, and remove the directory mount point corresponding to the failed disk;
Then, check the corresponding yarn.nodemanager.local-dirs parameter setting in the /etc/hadoop/conf/yarn-site.xml file on the failed node, and remove the directory mount point corresponding to the failed disk;
Finally, restart the DataNode service and NodeManager service of the node.
4.What should I do if Hadoop enters safe mode?
When Hadoop is just started, because the verification and startup of each service has not been completed, Hadoop will enter the safe mode at this time. At this time, the content of the file system is not allowed to be modified or deleted. This state will continue until the end of the safe mode. until.
This security mode is mainly for checking the validity of each DataNode data block when the system is started, and performing necessary copying or deleting of some data blocks according to the strategy.
If the startup and verification of Hadoop are normal, then just wait a while and Hadoop will automatically end the safe mode.
Of course, you can also manually end safe mode by executing the following commands:
[hadoop@namenodemaster conf]$ hdfs dfsadmin -safemode leave
5. What should I do if NodeManager has a Java heapspace error?
This kind of error is generally caused by insufficient JVM memory, so you need to modify the JVM memory size of all DataNodes and NodeManagers. As for the specific memory setting, it depends on the actual environment of the server.
If the set JVM value is already very large, but the problem still occurs, you need to check the NodeManager running log. What is the cause? You need to analyze the specific problem. Of course, the most direct method is to restart the NodeManager service of this node.
6. What is the reason for the Too manyfetch-failures error on the DataNode node?
The main reason for this problem is that the connectivity between the DataNode nodes is not smooth, or the network environment is not stable.
You can find the reason from the following aspects, and then you can basically judge the problem:
Check the network delay between the DataNode node and the NameNode node;
Use the Nslookup command to test the DNS resolution of the host name;
Check /etc/hosts and the corresponding host name information;
Check the SSH one-way trust from the NameNode to the DataNode node.
7. What should I do if No route to host appears?
This problem usually occurs when the DataNode cannot connect to the NameNode, which causes the DataNode to fail to start. When the problem occurs, you can see the following similar information in the DataNode log:
This problem is usually caused by the local firewall, the local network, or the selinux of the system, so you can turn off the local firewall or selinux, and then check the connectivity between the local machine and the NameNode, so you can determine the crux of the problem Where.
8. What should I do if the NameNode server fails?
In an HDFS cluster, all metadata information is stored on the NameNode host. Once this information is lost, all data on HDFS will be unusable.
So when the NameNode server fails to start, there are two ways to solve it:
(1) When the NameNode is a highly available service, after the primary NameNode fails, the NameNode service will automatically switch to the standby NameNode. This process is automatic and does not require manual intervention;
(2) If the Namenode does not provide a high-availability service, you can use the SecondaryNameNode service to find the metadata information in the SecondaryNameNode host, and then directly start the Namenode service on this node; because the SecondaryNameNode implements the cold backup of the Namenode, this The method may not be able to retrieve all the data, and some data will still be lost.
From this, it can be seen that the disaster recovery backup of the NameNode is very important. In a production environment, I recommend using standby NameNode to implement a highly available hot backup of the NameNode.
9. Why are cluster nodes marked as unhealthy by Yarn?
After the Yarn cluster runs tasks for a long time, some nodes will suddenly be marked as unhealthy nodes and removed from the Yarn cluster. After that, no more tasks will be submitted to this node.
So under what circumstances will a node be marked as unhealthy?
In the Yarn configuration, there is a parameter yarn.nodemanager.local-dirs, which is used to store the intermediate results of the NodeManager application running; there is another parameter yarn.nodemanager.log-dirs, which is used to specify the list of directories where NodeManager log files are stored. . Both of these parameters can configure multiple directories, and use commas to separate multiple directories.
Yarn will periodically check the disk status. If the free space of the directories specified by these two parameters is lower than the threshold specified by Yarn, NodeManager will not start any new containers on these nodes.
The local directory health check mainly involves the following two parameters:
yarn.nodemanager.disk-health-checker.min-healthy-disks
The default value of this parameter is 0.25, which means the proportion of normal directories in the general directory. If it is lower than 0.25, it is determined that the node is in an abnormal state. For example, if twelve directories (disks) are specified, at least three of them are in a normal state, and NodeManager will start a new container on that node.
yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage
The default value of this parameter is 90 (you can also set this parameter to any number between 0 and 100). It means that when the disk usage of the path under the yarn.nodemanager.local-dirs configuration item or the path under the yarn.nodemanager.log-dirs configuration item reaches more than 90%, the nodemanager on this machine will be marked as unhealthy.
The solution is simple: clean up the temporary data in the corresponding directory to reduce the disk usage rate to below 90%; modify the default parameter value of 90 to reset the disk usage rate.
I have an experience to share with you: It is best not to put Yarn's log directory or intermediate result directory on the same disk as the HDFS data storage directory. This can reduce a lot of unnecessary trouble.
10. How to solve the unbalanced disk storage of datanode nodes?
In HDFS clusters, disk damage is commonplace. After a disk failure, our general strategy is to replace a new hard disk. After the new hard disk is replaced, only new data will be written to this hard disk, and the old data will not automatically balance the data. .
If this continues, the more hard disks are replaced, the data between nodes and between the disks of each node will become more and more imbalanced; in addition, adding new data nodes to the cluster will also cause data imbalance in HDFS.
So how to make the HDFS cluster reach a balanced state again? You can use the Balancer program provided by Hadoop, and execute the command as follows:
[hadoop@namenodemaster sbin]$$HADOOP_HOME/bin/start-balancer.sh -t 5%
or:
[hadoop@namenodemaster sbin]$ hdfs balancer-threshold 5
The -t parameter in this command is followed by the disk usage deviation value when HDFS reaches a balanced state. If the disk usage deviation between nodes is less than 5%, then we think that the HDFS cluster has reached a balanced state.
11. What should I do if unbalanced task distribution is found in Yarn cluster?
Sometimes, when you run a data analysis task through the Yarn cluster, you will find such a problem: the load of each node will be uneven (that is, the number of tasks is different), some nodes have many tasks in execution, and some nodes have no tasks Execution, then how to balance the number of tasks running on each node?
The occurrence of this problem is closely related to the Yarn resource scheduling strategy you adopt.
If this is the case, the reason should be the default capacity scheduling strategy (Capacity Scheduler). Capacity scheduling will allocate tasks to nodes with resources as much as possible, regardless of task balance factors. Therefore, in this case, I recommend setting it as a fair scheduling strategy. This scheduling mode can evenly distribute tasks to each node in the cluster.
In fact, from the perspective of Hadoop cluster utilization, the probability of this problem is relatively low. Because under normal circumstances, tasks will continue to be submitted to the cluster, the cluster will be in a busy state at all times, and there will be no situation where the nodes have been idle, so uneven task distribution is difficult to occur.
12. There are missing blocks in HDFS, how to solve it?
The missing blocks error in HDFS clusters is a frequently occurring problem, and once it occurs, it often means that metadata is lost or damaged. It is very difficult or even impossible to restore it if you want to restore it.
So our solution is often not to restore data, but to delete related files. The specific solution is as follows, execute the following commands:
[hadoop@namenodemaster sbin]$ hdfs fsck/blocks-path/
This command will check the status of all blocks under HDFS and will show you which files have lost or damaged blocks.
Then execute the following command to delete these files:
[hadoop@namenodemaster sbin]$ hdfs fsck -fshdfs://bigdata/logs/mv.log -delete
The file mv.log on HDFS was deleted above. Because the metadata of this file is lost and cannot be recovered, it can only be deleted.
Operating system tuning for Hadoop tuning
1. Adjust the upper limit of the operating system to open file descriptors
Hadoop task analysis often needs to read and write a large number of files, so the upper limit of open file descriptors needs to be increased. You can use ulimit -n to view the upper limit of open file descriptors in the current system. The default value of CentOS 7 system is 1024, this value is too small, it is recommended to modify it to 655360 or greater.
Through the command "ulimit -a", you can see all system resource parameters. The key settings here are "open files" and "max user processes", and others can be set as appropriate.
To permanently set resource parameters, it is mainly achieved through the following files:
/etc/security/limits.conf
/etc/security/limits.d/90-nproc.conf(centos6.x)
/etc/security/limits.d/20-nproc.conf(centos7.x)
Add the following content to /etc/security/limits.conf, then exit the shell and log in again to take effect.
copy
* soft nproc 204800
* hard nproc 204800
* soft nofile 655360
* hard nofile 655360
* soft memlock unlimited
* hard memlock unlimited
It should be noted that: In CentOS 6.x, there is a 90-nproc.conf file; in CentOS 7.x, there is a 20-nproc.conf file. Since the maximum number of user processes has been configured by default, for this The two settings are unnecessary, so just delete these two files directly.
2. Modify the net.core.somaxconn parameter
The specific file path corresponding to this kernel parameter is /proc/sys/net/core/somaxconn, which is used to set the upper limit of the backlog of the socket monitor (listen).
What is backlog? It is the listening queue of Socket. When a request is not processed or established, it will enter the backlog. The socket server can process all requests in the backlog at one time, and the processed requests are no longer in the listening queue.
If the server is slow to process requests so that when the listening queue is full, new requests will be rejected, so this value must be increased. The default value of this parameter is 128. As a basic optimization of network parameters, it is recommended to modify the following values:
echo 4096 >/proc/sys/net/core/somaxconn
3. Adjust the proportion of operating system using swap
Swap was originally used as an extension of physical memory, but nowadays the memory is generally sufficient, and swap is rarely used. In addition, data exchange to swap causes operation timeouts, which affects Hadoop's reading and writing and data analysis performance. Therefore, the above two points have led to fewer and fewer scenarios for using swap today.
We can adjust the ratio of swap using the system kernel parameter /proc/sys/vm/swappiness. When swappiness=0, it means that the physical memory is used to the maximum and then the swap space; when swappiness=100, it means that the swap partition is actively used and the data on the memory is moved to the swap space in time.
Linux is basically set to 60 by default, which means that when your physical memory reaches 100-60=40%, the swap partition will be used. For servers with high memory requirements (such as Hadoop, Redis, and HBase machines), the Linux value needs to be set small enough (between 0 and 10) to maximize the use of physical memory.
4. Disable THP (TransparentHuge Pages) function
The original intention of THP is to improve the performance of the memory, but in the Hadoop environment, it was found that this function will increase the CPU usage and affect the performance of Hadoop, so it is recommended to turn it off.
First check the activation status of THP:
|
1
2
3
4
|
[root@localhost ~]# cat /sys/kernel/mm/transparent_hugepage/defrag
[always] madvise never
[root@localhost ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
|
Here always shows that THP is currently enabled. To disable THP, open the /etc/rc.d/rc.local file and add the following:
|
1
2
3
4
5
6
|
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
|
Then save and exit.
Finally, grant execution permissions to the rc.local file and execute the following commands:
[root@localhost ~]# chmod +x/etc/rc.d/rc.local
[root@localhost ~]# source /etc/rc.local
At this time, the THP function has been disabled.