参考
Effective Testing Strategies for MapReduce Applications：https://stackoverflow.com/questions/27136752/powermock-throws-nosuchmethoderror-setmockname
PowerMock throws NoSuchMethodError (setMockName)：https://stackoverflow.com/questions/27136752/powermock-throws-nosuchmethoderror-setmockname
Intellij IDEA run coverage之覆盖率测试：https://www.jianshu.com/p/ca3b91e85fea
Idea maven项目不能新建package和class的解决：https://blog.csdn.net/qq_24949727/article/details/52097838
Hadoop现有测试框架探幽：https://blog.csdn.net/azhao_dn/article/details/8065679
Hadoop Herriot测试框架之旅：https://www.linuxidc.com/Linux/2011-10/44984.htm
@Override is not allowed when implementing interface method：https://stackoverflow.com/questions/15402615/override-is-not-allowed-when-implementing-interface-method
测试方式
mrunit
mrunit 已经被退休了，很多人建议不要再使用这个进行测试
千遍一律的WordCount
public class WordCountMRUnitTest {
MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable> mapReduceDriver;
MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
@Before
public void setup() {
WordCountMapper mapper = new WordCountMapper();
WordCountReducer reducer = new WordCountReducer();
mapDriver = new MapDriver<LongWritable, Text, Text, IntWritable>();
mapDriver.setMapper(mapper);
reduceDriver = new ReduceDriver<Text, IntWritable, Text, IntWritable>();
reduceDriver.setReducer(reducer);
mapReduceDriver = new MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable>();
mapReduceDriver.setMapper(mapper);
mapReduceDriver.setReducer(reducer);
Configuration conf = new Configuration();
// add config here as needed
mapReduceDriver.setConfiguration(conf);
reduceDriver.setConfiguration(conf);
mapDriver.setConfiguration(conf);
}
@Test
public void testMapper() throws IOException {
mapDriver.withInput(new LongWritable(1), new Text("cat cat dog"));
mapDriver.withOutput(new Text("cat"), new IntWritable(1));
mapDriver.withOutput(new Text("cat"), new IntWritable(1));
mapDriver.withOutput(new Text("dog"), new IntWritable(1));
mapDriver.runTest();
}
@Test
public void testReducer() throws IOException {
List<IntWritable> values = new ArrayList<IntWritable>();
values.add(new IntWritable(1));
values.add(new IntWritable(1));
reduceDriver.withInput(new Text("cat"), values);
reduceDriver.withOutput(new Text("cat"), new IntWritable(2));
reduceDriver.runTest();
}
@Test
public void testMapReduce() throws IOException {
mapReduceDriver.withInput(new LongWritable(1), new Text("cat cat dog"));
mapReduceDriver.addOutput(new Text("cat"), new IntWritable(2));
mapReduceDriver.addOutput(new Text("dog"), new IntWritable(1));
mapReduceDriver.runTest();
}
}
junit
对map，reduce和Driver分别测试，主要用到了Mockito
public class WordCountMapperTest {
private WordCountMapper mapper;
private Mapper.Context context;
private IntWritable one;
@Before
public void init() throws IOException, InterruptedException {
mapper = new WordCountMapper();
context = mock(Mapper.Context.class); //
mapper.word = mock(Text.class);
one = new IntWritable(1);
}
@Test
public void testSingleWord() throws IOException, InterruptedException {
mapper.map(new LongWritable(1L), new Text("foo"), context);
InOrder inOrder = inOrder(mapper.word, context);
assertCountedOnce(inOrder, "foo");
}
@Test
public void testMultipleWords() throws IOException, InterruptedException {
mapper.map(new LongWritable(1L), new Text("one two three four"), context);
InOrder inOrder = inOrder(mapper.word, context, mapper.word, context, mapper.word, context, mapper.word, context);
assertCountedOnce(inOrder, "one");
assertCountedOnce(inOrder, "two");
assertCountedOnce(inOrder, "three");
assertCountedOnce(inOrder, "four");
}
private void assertCountedOnce(InOrder inOrder, String w) throws IOException, InterruptedException {
inOrder.verify(mapper.word).set(eq(w));
inOrder.verify(context).write(eq(mapper.word), eq(one));
}
}
public class WordCountReducerTest {
private WordCountReducer reducer;
private Reducer.Context context;
@Before
public void init() throws IOException, InterruptedException {
reducer = new WordCountReducer();
context = mock(Reducer.Context.class);
}
@Test
public void testSingleWord() throws IOException, InterruptedException {
List<IntWritable> values = Arrays.asList(new IntWritable(1), new IntWritable(4), new IntWritable(7));
reducer.reduce(new Text("foo"), values, context);
verify(context).write(new Text("foo"), new IntWritable(12));
}
}
public class WordCountDriverTest {
private Configuration conf;
private Path input;
private Path output;
private FileSystem fs;
@Before
public void setup() throws IOException {
conf = new Configuration();
conf.set("fs.default.name", "file:///");
conf.set("mapred.job.tracker", "local");
input = new Path("src/test/resources/input");
output = new Path("target/output");
fs = FileSystem.getLocal(conf);
fs.delete(output, true);
}
@Test
public void test() throws Exception {
WordCount wordCount = new WordCount();
wordCount.setConf(conf);
int exitCode = wordCount.run(new String[] {input.toString(), output.toString()});
assertEquals(0, exitCode);
validateOuput();
}
private void validateOuput() throws IOException {
InputStream in = null;
try {
in = fs.open(new Path("target/output/part-r-00000"));
BufferedReader br = new BufferedReader(new InputStreamReader(in));
assertEquals("five\t1", br.readLine());
assertEquals("four\t1", br.readLine());
assertEquals("one\t3", br.readLine());
assertEquals("six\t1", br.readLine());
assertEquals("three\t1", br.readLine());
assertEquals("two\t2", br.readLine());
} finally {
IOUtils.closeStream(in);
}
}
}
在hadoop本身测试基础上
hadoop说在0.21版本之前，利用MiniDFSCluster模拟hdfs集群环境，看源码可以看到很多例子，比如TeraSort的tests；
之后，hadoop发行版中引入了一个新的测试框架，Large-Scale Automated Test Framework，该框架跟以前的测试框架不同之处在于，基于它之上的测试的开发是基于真正的集群环境的系统层面的，取名叫做Herriot。
将TeraSort的代码和相关测试抽出来尝试其测试过程，过程中发现很多类通过maven引不进来，比如MiniDFSCluster，后来就直接导入相关jar包：hadoop-hdfs-2.8.3-tests.jar，该jar包在安装的hadoop目录的share文件中可以找到。
package TeraSort;
import java.io.File;
import java.io.IOException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapred.FileAlreadyExistsException;
import org.apache.hadoop.mapred.HadoopTestCase;
import org.apache.hadoop.util.ToolRunner;
import org.junit.After;
import org.junit.Test;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.fail;
public class TestTeraSort extends HadoopTestCase {
private static Log LOG = LogFactory.getLog(TestTeraSort.class);
public TestTeraSort()
throws IOException {
super(LOCAL_MR, LOCAL_FS, 1, 1);
}
@After
public void tearDown() throws Exception {
getFileSystem().delete(TEST_DIR, true);
super.tearDown();
}
// Input/Output paths for sort
private static final Path TEST_DIR = new Path(new File(
System.getProperty("test.build.data", "/tmp"), "terasort")
.getAbsoluteFile().toURI().toString());
private static final Path SORT_INPUT_PATH = new Path(TEST_DIR, "sortin");
private static final Path SORT_OUTPUT_PATH = new Path(TEST_DIR, "sortout");
private static final Path TERA_OUTPUT_PATH = new Path(TEST_DIR, "validate");
private static final String NUM_ROWS = "100";
private void runTeraGen(Configuration conf, Path sortInput) throws Exception {
String[] genArgs = {NUM_ROWS, sortInput.toString()};
// Run TeraGen
assertEquals(ToolRunner.run(conf, new TeraGen(), genArgs), 0);
}
private void runTeraSort(Configuration conf, Path sortInput, Path sortOutput) throws Exception {
// Setup command-line arguments to 'sort'
String[] sortArgs = {sortInput.toString(), sortOutput.toString()};
// Run Sort
assertEquals(ToolRunner.run(conf, new TeraSort(), sortArgs), 0);
}
private void runTeraValidator(Configuration job,Path sortOutput, Path valOutput) throws Exception {
String[] svArgs = {sortOutput.toString(), valOutput.toString()};
// Run Tera-Validator
assertEquals(ToolRunner.run(job, new TeraValidate(), svArgs), 0);
}
@Test
public void testTeraSort() throws Exception {
// Run TeraGen to generate input for 'terasort'
runTeraGen(createJobConf(), SORT_INPUT_PATH);
// Run teragen again to check for FAE
try {
runTeraGen(createJobConf(), SORT_INPUT_PATH);
fail("Teragen output overwritten!");
} catch (FileAlreadyExistsException fae) {
LOG.info("Expected exception: ", fae);
}
// Run terasort
runTeraSort(createJobConf(), SORT_INPUT_PATH, SORT_OUTPUT_PATH);
// Run terasort again to check for FAE
try {
runTeraSort(createJobConf(), SORT_INPUT_PATH, SORT_OUTPUT_PATH);
fail("Terasort output overwritten!");
} catch (FileAlreadyExistsException fae) {
LOG.info("Expected exception: ", fae);
}
// Run tera-validator to check if sort worked correctly
runTeraValidator(createJobConf(), SORT_OUTPUT_PATH,
TERA_OUTPUT_PATH);
}
@Test
public void testTeraSortWithLessThanTwoArgs() throws Exception {
String[] args = new String[1];
assertEquals(new TeraSort().run(args), 2);
}
}
测试率覆盖
idea本身自带有测试覆盖率的，run test时选择run “*Test*” with Coverage
学习过程中遇到的问题
编码错误
@Override is not allowed when implementing interface method Project Structure -> Modules -> Language level 选择6
powermockito和mockito版本问题
错误1：java.lang.NoSuchMethodError:
org.mockito.internal.creation.MockSettingsImpl.setMockName(Lorg/mockito/mock/MockName;)Lorg/mockito/internal/creation/settings/CreationSettings;
错误2：java.lang.NoSuchMethodError:
org.powermock.reflect.internal.WhiteboxImpl.getOriginalUnmockedType(Ljava/lang/Class;)Ljava/lang/Class;| Mockito | PowerMock |
|---|---|
| 2.0.0-beta - 2.0.42-beta | 1.6.5+ |
| 1.10.19 | 1.6.4 |
| 1.10.8 - 1.10.x | 1.6.2+ |
| 1.9.5-rc1 - 1.9.5 | 1.5.0 - 1.5.6 |
| 1.9.0-rc1 & 1.9.0 | 1.4.10 - 1.4.12 |
| 1.8.5 | 1.3.9 - 1.4.9 |
| 1.8.4 | 1.3.7 & 1.3.8 |
| 1.8.3 | 1.3.6 |
| 1.8.1 & 1.8.2 | 1.3.5 |
| 1.8 | 1.3 |
| 1.7 | 1.2.5 |
自己建的目录不能new java类
点击该目录，右键 -> Mark Directory as -> Sources Root