版权声明：采菊东篱下，悠然现南山! https://blog.csdn.net/ChaosJ/article/details/52920342
1. NM异常日志 (/data0/hadoop/log/yarn/yarn-mapred-nodemanager-yz5202.hadoop.data.sina.com.cn.log)
ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs:
//ns1/user/huali1/.staging/job_1470311300058_64103/libjars/tez-mapreduce-0.5.0-incubating-SNAPSHOT.jar, 1470452069832, FILE, null },pending,[],1163127461340,DOWNLOADING} with non-zero refcount
2016
-
10
-
25
00
:
07
:
59
,
843
ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs:
//ns1/user/huali1/.staging/job_1470311300058_64103/libjars/snappy-java-1.0.4.1.jar, 1470452070141, FILE, null },pending,[],1163127475524,DOWNLOADING} with non-zero refcount
2016
-
10
-
25
00
:
07
:
59
,
843
ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs:
//ns1/user/huali1/.staging/job_1470311300058_64103/libjars/hadoop-mapreduce-client-core-2.4.0.jar, 1470452070471, FILE, null },pending,[],1163127488563,DOWNLOADING} with non-zero refcount
2016
-
10
-
25
00
:
07
:
59
,
843
ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs:
//ns1/user/huali1/.staging/job_1470311300058_64103/libjars/hadoop-mapreduce-client-common-2.4.0.jar, 1470452071821, FILE, null },pending,[],1163127500891,DOWNLOADING} with non-zero refcount
2016
-
10
-
25
00
:
07
:
59
,
843
ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs:
//ns1/user/huali1/.staging/job_1470311300058_64103/libjars/guice-3.0.jar, 1470452070637, FILE, null },pending,[],1163127516204,DOWNLOADING} with non-zero refcount
2016
-
10
-
25
00
:
07
:
59
,
843
ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs:
//ns1/user/huali1/.staging/job_1470311300058_64103/libjars/tez-dag-0.5.0-incubating-SNAPSHOT.jar, 1470452072079, FILE, null },pending,[],1163127534824,DOWNLOADING} with non-zero refcount
2016
-
10
-
25
00
:
07
:
59
,
843
ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs:
//ns1/user/huali1/.staging/job_1470311300058_64103/libjars/tez-runtime-internals-0.5.0-incubating-SNAPSHOT.jar, 1470452069673, FILE, null },pending,[],1163127555160,DOWNLOADING} with non-zero refcount
2016
-
10
-
25
00
:
07
:
59
,
843
ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource: { { hdfs:
//ns1/user/huali1/.staging/job_1470311300058_64103/libjars/jsr305-2.0.3.jar, 1470452070933, FILE, null },pending,[],1163127573133,DOWNLOADING} with non-zero refcount
|
2.异常问题分析
If a container is in the process of localizing when it is stopped/killed then resources are left in the DOWNLOADING state. If no other container comes along and requests these resources they linger around with no reference counts but aren't cleaned up during normal cache cleanup scans since it will never delete resources in the DOWNLOADING state even if their reference count is zero
container 在localizing 过程中如果被kill或者stopped，使得资源处于DOWNLOADING 状态，并且没有其他的container使用这些资源即rsrc.getRefCount() <= 0；
在进行资源清理的时候，处于DOWNLOADING 状态的资源不会被清理，即使RefCount=0；
在上述状态下会出现异常：ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Attempt to remove resource
异常代码位置
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
public
boolean
remove(LocalizedResource rem, DeletionService delService) {
// current synchronization guaranteed by crude RLS event for cleanup
LocalizedResource rsrc = localrsrc.get(rem.getRequest());
if
(
null
== rsrc) {
LOG.error(
"Attempt to remove absent resource: "
+ rem.getRequest()
+
" from "
+ getUser());
return
true
;
}
if
(rsrc.getRefCount() >
0
|| ResourceState.DOWNLOADING.equals(rsrc.getState()) || rsrc != rem) {
// internal error
LOG.error(
"Attempt to remove resource: "
+ rsrc
+
" with non-zero refcount"
);
return
false
;
}
else
{
// ResourceState is LOCALIZED or INIT
localrsrc.remove(rem.getRequest());
if
(ResourceState.LOCALIZED.equals(rsrc.getState())) {
delService.delete(getUser(), getPathToDelete(rsrc.getLocalPath()));
}
decrementFileCountForLocalCacheDirectory(rem.getRequest(), rsrc);
LOG.info(
"Removed "
+ rsrc.getLocalPath() +
" from localized cache"
);
return
true
;
}
}
|
3.解决办法
过滤条件中把ResourceState.DOWNLOADING.equals(rsrc.getState()) 条件去掉，patch如下，patch 内容见Reference；
Index: LocalResourcesTrackerImpl.java
===================================================================
--- LocalResourcesTrackerImpl.java (revision
270924
)
+++ LocalResourcesTrackerImpl.java (working copy)
@@ -
230
,
8
+
230
,
7
@@
+
" from "
+ getUser());
return
true
;
}
-
if
(rsrc.getRefCount() >
0
- || ResourceState.DOWNLOADING.equals(rsrc.getState()) || rsrc != rem) {
+
if
(rsrc.getRefCount() >
0
|| rsrc != rem) {
// internal error
LOG.error(
"Attempt to remove resource: "
+ rsrc
+
" with non-zero refcount"
);
|
4.Reference
JIRA 地址 ：https://issues.apache.org/jira/browse/YARN-2902
patch 地址：https://issues.apache.org/jira/secure/attachment/12685562/YARN-2902.patch