凌云的博客

行胜于言

命令行直接运行 MRUnit 用例

分类：hadoop| 发布时间：2018-04-23 20:28:00

前言

在上一篇文章 MRUnit 入门 中我们介绍了如何使用 MRUnit 编写以及运行 MapReduce 的测试用例。

通过使用 maven 进行项目的构建以及管理显得很方便，我们只需要写好代码然后使用 mvn test 命令即可运行测试。 由于很好奇 mvn test 究竟做了什么以及是否能不使用 mvn 而直接通过 java 命令运行 mrunit 的测试用例， 我做了一些测试以及探索并将相应的过程和结果记录为本文。

本文的目标主要是如何只通过 java 运行 mrunit 用例。

运行 junit 单元测试

由于 mrunit 需要依赖大量的外部库，在搞懂如何运行 mrunit 之前，先从如何直接运行 junit 单元测试是一个很好的着手点。

假设我们有如下代码需要测试：

// Calculator.javaa
public class Calculator {
    public int evaluate(String expression) {
        int sum = 0;
        for (String summand: expression.split("\\+"))
            sum += Integer.valueOf(summand);

        return sum;
    }
}

测试代码如下：

// Calculator.javaa
import static org.junit.Assert.assertEquals;
import org.junit.Test;

public class CalculatorTest {
    @Test
    public void evaluatesExpression() {
        Calculator calculator = new Calculator();
        int sum = calculator.evaluate("1+2+3");
        assertEquals(6, sum);
    }
}

要运行 junit 需要下载相关的 jar，在 https://github.com/junit-team/junit4/wiki/Download-and-Install 这里下载 junit 和 hamcrest-core.jar。 本文假设你将其下载到了当前用户的 home 目录，你可以需要根据实际更改下面的命令：

首先先编译相关的代码和测试代码（注意需要使用全路径）：

% javac javac CalculatorTest.java
% javac -cp .:/home/ubuntu/junit.jar:/home/ubuntu/hamcrest-core-1.3.jar CalculatorTest.java

使用如下命令即可运行单元测试：

% java -cp .:/home/ubuntu/junit.jar:/home/ubuntu/hamcrest-core-1.3.jar org.junit.runner.JUnitCore CalculatorTest
JUnit version 4.12
.
Time: 0.004

OK (1 test)

可以看到单元测试成功运行了。

运行 MRUnit 单元测试

理论上来说直接运行 mrunit 和 junit 是一样的。不同的是要编译以及运行 mrunit 需要依赖的库要比上面简单的 junit 测试用例要复杂得多。

我最终放弃了一个一个地分析其中的依赖树并手动将其下载到本地，转而分析 mvn test 命令的输出得到其完整的以来列表。

以上一篇文章的代码为例，在上一篇文章的代码目录执行以下命令：

% mvn test -X >debug.log 2>&1

通过查看 debug.log 发现其完整依赖列表为：

classpathElements = [
/home/ubuntu/src/wordcount-mrunit/target/test-classes,
/home/ubuntu/src/wordcount-mrunit/target/classes,
/home/ubuntu/.m2/repository/org/apache/hadoop/hadoop-core/1.0.3/hadoop-core-1.0.3.jar,
/home/ubuntu/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar,
/home/ubuntu/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar,
/home/ubuntu/.m2/repository/commons-httpclient/commons-httpclient/3.0.1/commons-httpclient-3.0.1.jar,
/home/ubuntu/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar,
/home/ubuntu/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar,
/home/ubuntu/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar,
/home/ubuntu/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar,
/home/ubuntu/.m2/repository/commons-lang/commons-lang/2.4/commons-lang-2.4.jar,
/home/ubuntu/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar,
/home/ubuntu/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar,
/home/ubuntu/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar,
/home/ubuntu/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar,
/home/ubuntu/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar,
/home/ubuntu/.m2/repository/org/mortbay/jetty/servlet-api/2.5-20081211/servlet-api-2.5-20081211.jar,
/home/ubuntu/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar,
/home/ubuntu/.m2/repository/tomcat/jasper-runtime/5.5.12/jasper-runtime-5.5.12.jar,
/home/ubuntu/.m2/repository/tomcat/jasper-compiler/5.5.12/jasper-compiler-5.5.12.jar,
/home/ubuntu/.m2/repository/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar,
/home/ubuntu/.m2/repository/org/mortbay/jetty/servlet-api-2.5/6.1.14/servlet-api-2.5-6.1.14.jar,
/home/ubuntu/.m2/repository/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar,
/home/ubuntu/.m2/repository/ant/ant/1.6.5/ant-1.6.5.jar, /home/ubuntu/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar,
/home/ubuntu/.m2/repository/net/java/dev/jets3t/jets3t/0.7.1/jets3t-0.7.1.jar,
/home/ubuntu/.m2/repository/net/sf/kosmosfs/kfs/0.3/kfs-0.3.jar, /home/ubuntu/.m2/repository/hsqldb/hsqldb/1.8.0.10/hsqldb-1.8.0.10.jar,
/home/ubuntu/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar,
/home/ubuntu/.m2/repository/org/eclipse/jdt/core/3.1.1/core-3.1.1.jar,
/home/ubuntu/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.0.1/jackson-mapper-asl-1.0.1.jar,
/home/ubuntu/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.0.1/jackson-core-asl-1.0.1.jar,
/home/ubuntu/.m2/repository/org/apache/mrunit/mrunit/1.1.0/mrunit-1.1.0-hadoop1.jar,
/home/ubuntu/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar,
/home/ubuntu/.m2/repository/org/mockito/mockito-core/1.9.5/mockito-core-1.9.5.jar,
/home/ubuntu/.m2/repository/org/objenesis/objenesis/1.0/objenesis-1.0.jar,
/home/ubuntu/.m2/repository/com/google/guava/guava/11.0.2/guava-11.0.2.jar,
/home/ubuntu/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar,
/home/ubuntu/.m2/repository/org/powermock/powermock-core/1.5.1/powermock-core-1.5.1.jar,
/home/ubuntu/.m2/repository/org/powermock/powermock-reflect/1.5.1/powermock-reflect-1.5.1.jar,
/home/ubuntu/.m2/repository/org/javassist/javassist/3.18.0-GA/javassist-3.18.0-GA.jar,
/home/ubuntu/.m2/repository/org/powermock/powermock-api-mockito/1.5.1/powermock-api-mockito-1.5.1.jar,
/home/ubuntu/.m2/repository/org/powermock/powermock-api-support/1.5.1/powermock-api-support-1.5.1.jar,
/home/ubuntu/.m2/repository/org/powermock/powermock-module-junit4/1.5.1/powermock-module-junit4-1.5.1.jar,
/home/ubuntu/.m2/repository/org/powermock/powermock-module-junit4-common/1.5.1/powermock-module-junit4-common-1.5.1.jar,
/home/ubuntu/.m2/repository/junit/junit/4.10/junit-4.10.jar,
/home/ubuntu/.m2/repository/org/hamcrest/hamcrest-core/1.1/hamcrest-core-1.1.jar,
/home/ubuntu/.m2/repository/org/hamcrest/hamcrest-all/1.3/hamcrest-all-1.3.jar]

有了上述信息我们就可以手动编译以及执行 wordcount-mrunit 了。

在这里我只演示如何运行 WordCountMapperTest.java，其他的测试用例类似。

  • 首先编译 WordCountMapper.java
% cd src/main/java
% javac -cp /home/ubuntu/.m2/repository/org/apache/hadoop/hadoop-core/1.0.3/hadoop-core-1.0.3.jar WordCountMapper.java
  • 然后编译 WordCountMapperTest.java
% javac -cp /home/ubuntu/src/wordcount-mrunit/src/main/java/:\
/home/ubuntu/.m2/repository/org/apache/hadoop/hadoop-core/1.0.3/hadoop-core-1.0.3.jar:\
/home/ubuntu/.m2/repository/org/apache/mrunit/mrunit/1.1.0/mrunit-1.1.0-hadoop1.jar:\
/home/ubuntu/.m2/repository/junit/junit/4.10/junit-4.10.jar WordCountMapperTest.java
  • 最后通过以下命令运行单元测试 这里的命令所使用的 classpath 除了前两项有所变化外其他的都是上面 mvn test 得到的路径。
% java -cp /home/ubuntu/src/wordcount-mrunit/src/main/java/:\
/home/ubuntu/src/wordcount-mrunit/src/test/java/:\
/home/ubuntu/.m2/repository/org/apache/hadoop/hadoop-core/1.0.3/hadoop-core-1.0.3.jar:\
/home/ubuntu/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:\
/home/ubuntu/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:\
/home/ubuntu/.m2/repository/commons-httpclient/commons-httpclient/3.0.1/commons-httpclient-3.0.1.jar:\
/home/ubuntu/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:\
/home/ubuntu/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:\
/home/ubuntu/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:\
/home/ubuntu/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:\
/home/ubuntu/.m2/repository/commons-lang/commons-lang/2.4/commons-lang-2.4.jar:\
/home/ubuntu/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:\
/home/ubuntu/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:\
/home/ubuntu/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:\
/home/ubuntu/.m2/repository/commons-net/commons-net/1.4.1/commons-net-1.4.1.jar:\
/home/ubuntu/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar:\
/home/ubuntu/.m2/repository/org/mortbay/jetty/servlet-api/2.5-20081211/servlet-api-2.5-20081211.jar:\
/home/ubuntu/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:\
/home/ubuntu/.m2/repository/tomcat/jasper-runtime/5.5.12/jasper-runtime-5.5.12.jar:\
/home/ubuntu/.m2/repository/tomcat/jasper-compiler/5.5.12/jasper-compiler-5.5.12.jar:\
/home/ubuntu/.m2/repository/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar:\
/home/ubuntu/.m2/repository/org/mortbay/jetty/servlet-api-2.5/6.1.14/servlet-api-2.5-6.1.14.jar:\
/home/ubuntu/.m2/repository/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar:\
/home/ubuntu/.m2/repository/ant/ant/1.6.5/ant-1.6.5.jar:\
/home/ubuntu/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:\
/home/ubuntu/.m2/repository/net/java/dev/jets3t/jets3t/0.7.1/jets3t-0.7.1.jar:\
/home/ubuntu/.m2/repository/net/sf/kosmosfs/kfs/0.3/kfs-0.3.jar:\
/home/ubuntu/.m2/repository/hsqldb/hsqldb/1.8.0.10/hsqldb-1.8.0.10.jar:\
/home/ubuntu/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar:\
/home/ubuntu/.m2/repository/org/eclipse/jdt/core/3.1.1/core-3.1.1.jar:\
/home/ubuntu/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.0.1/jackson-mapper-asl-1.0.1.jar:\
/home/ubuntu/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.0.1/jackson-core-asl-1.0.1.jar:\
/home/ubuntu/.m2/repository/org/apache/mrunit/mrunit/1.1.0/mrunit-1.1.0-hadoop1.jar:\
/home/ubuntu/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:\
/home/ubuntu/.m2/repository/org/mockito/mockito-core/1.9.5/mockito-core-1.9.5.jar:\
/home/ubuntu/.m2/repository/org/objenesis/objenesis/1.0/objenesis-1.0.jar:\
/home/ubuntu/.m2/repository/com/google/guava/guava/11.0.2/guava-11.0.2.jar:\
/home/ubuntu/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:\
/home/ubuntu/.m2/repository/org/powermock/powermock-core/1.5.1/powermock-core-1.5.1.jar:\
/home/ubuntu/.m2/repository/org/powermock/powermock-reflect/1.5.1/powermock-reflect-1.5.1.jar:\
/home/ubuntu/.m2/repository/org/javassist/javassist/3.18.0-GA/javassist-3.18.0-GA.jar:\
/home/ubuntu/.m2/repository/org/powermock/powermock-api-mockito/1.5.1/powermock-api-mockito-1.5.1.jar:\
/home/ubuntu/.m2/repository/org/powermock/powermock-api-support/1.5.1/powermock-api-support-1.5.1.jar:\
/home/ubuntu/.m2/repository/org/powermock/powermock-module-junit4/1.5.1/powermock-module-junit4-1.5.1.jar:\
/home/ubuntu/.m2/repository/org/powermock/powermock-module-junit4-common/1.5.1/powermock-module-junit4-common-1.5.1.jar:\
/home/ubuntu/.m2/repository/junit/junit/4.10/junit-4.10.jar:\
/home/ubuntu/.m2/repository/org/hamcrest/hamcrest-core/1.1/hamcrest-core-1.1.jar:\
/home/ubuntu/.m2/repository/org/hamcrest/hamcrest-all/1.3/hamcrest-all-1.3.jar org.junit.runner.JUnitCore WordCountMapperTest

JUnit version 4.10
..
Time: 0.273

OK (2 tests)

可以看出如此简单的一个 mrunit 用例都要依赖如此多的库，如果真要手动处理好所有的库版本以及其依赖将是无法想象的。 这不仅会使得构建变得低效而且容易出错。

参考

Copyright © 2013-2024 cs-cjl.com