1. Problem Description: In the process of debugging mapreduce secondary sort (secondary sort), run the program always error
Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewOutputCollector@1398c56
java.lang.NullPointerException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:157)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1269)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:74)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1491)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2021)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
8860 [Thread-6] INFO org.apache.hadoop.mapred.LocalJobRunner - map task executor complete.
8872 [Thread-6] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local1887386767_0001
java.lang.Exception: java.lang.NullPointerException
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:157)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1269)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:74)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1491)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2. Analysis:
(1) initially determined by calling a null pointer;
(2) click WritableComparator.java:157 jump to a given place this.buffer.reset (b1, s1, l1);
public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
try {
this.buffer.reset(b1, s1, l1);
this.key1.readFields(this.buffer);
this.buffer.reset(b2, s2, l2);
this.key2.readFields(this.buffer);
this.buffer.reset((byte[])null, 0, 0);
} catch (IOException var8) {
throw new RuntimeException(var8);
}
（3）也就是说WritableComparator buffer没有初始化
public class WritableComparator the implements RawComparator, the Configurable {
Private static Final of ConcurrentHashMap <Class, WritableComparator> Comparators = new new of ConcurrentHashMap ();
Private the Configuration the conf;
Private Final Class keyClass <the extends WritableComparable?>;
Private Final WritableComparable key1;
Private Final WritableComparable key2;
Private Final DataInputBuffer buffer;
defined in (4) used in the program to look for related classes WritableComparator class is relatively FirstSecondComparator first field and packet type ReducerGroupingComparator no default constructor
static class ReducerGroupingComparator the extends WritableComparator public
{ // Be sure to add the default constructor, and be sure to add super, otherwise they will be reported abnormal Ignoring Exception During use Close for java.lang.NullPointerException org.apache.hadoop.mapred.MapTask$NewOutputCollector@1398c56
public ReducerGroupingComparator ()
{
Super (IntPair.class, to true);
}
@Override
public int Compare (WritableComparable A, B WritableComparable) {
IntPair Pairal = (IntPair) A;
IntPair pairB = (IntPair) B;
the same year // return value is divided into a set of 0
(int) return (pairA.getFirst () - pairB.getFirst ());
}
}
// default same functions defined FirstSecondComparator contrast function, the contrast setting for displaying the class
public static class FirstSecondComparator extends WritableComparator
{
//
// Be sure to add the default constructor, and be sure to add super, otherwise it will report an exception java.lang.NullPointerException Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewOutputCollector@1398c
FirstSecondComparator public ()
{
Super (IntPair.class, to true);
}
@Override
public int Compare (WritableComparable A, B WritableComparable) {
IntPair Pairal = (IntPair) A;
IntPair pairB = (IntPair) B;
int CMP = (int) (pairA.getFirst () - pairA.getFirst ());
IF (CMP = 0!)
{
return CMP;
}
Double pairA.getSecond A = ();
Double pairB.getSecond B = ();
return - ((A = B =) 0: (A> B. 1:?? -1));
}
}
(. 5) Why did not have to add this message java.lang.NullPointerException exception? Opening the super (IntPair.class, true);
It will jump to WritableComparator constructor
WritableComparator protected (Class keyClass, boolean createInstances <the extends WritableComparable?>) {
the this (keyClass, (the Configuration) null, createInstances);
}
After clicking this, a further jump to another overloaded constructor, see here, we should understand name the above reported abnormal place, that did not apply for these variables memory space
protected WritableComparator(Class<? extends WritableComparable> keyClass, Configuration conf, boolean createInstances) {
this.keyClass = keyClass;
this.conf = conf != null ? conf : new Configuration();
if (createInstances) {
this.key1 = this.newKey();
this.key2 = this.newKey();
this.buffer = new DataInputBuffer();
} else {
this.key1 = this.key2 = null;
this.buffer = null;
}
}
Himself developed an intelligent stock analysis software, very powerful, you need to click on the link below to obtain:
https://www.cnblogs.com/bclshuai/p/11380657.html