sbt-assembly在Spark应用架包冲突的解决办法

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/yiifaa/article/details/81952245

在利用sbt-assmebly对Spark应用打包时，提示如下错误：

java.lang.RuntimeException: deduplicate: different file contents found in the following:
[error] C:\Users\didi\.ivy2\cache\javax.inject\javax.inject\jars\javax.inject-1.jar:javax/inject/Inject.class
[error] C:\Users\didi\.ivy2\cache\org.glassfish.hk2.external\javax.inject\jars\javax.inject-2.4.0-b34.jar:javax/inject/Inject.class
[error] deduplicate: different file contents found in the following:
[error] C:\Users\didi\.ivy2\cache\javax.inject\javax.inject\jars\javax.inject-1.jar:javax/inject/Named.class
[error] C:\Users\didi\.ivy2\cache\org.glassfish.hk2.external\javax.inject\jars\javax.inject-2.4.0-b34.jar:javax/inject/Named.class

从内容可以看出，问题出现在有重复的class文件，所以解决的思路也很简单，哪里重复就合并哪里！

具体的做法是：
1. 找到出错的包名；
2. 将包名替换为PathList(x, y, xs @ _*)，添加在“case”列表中；

以上面的错误为例：
1. 找到包名，“javax/inject”；
2. 替换为PathList(“javax”, “inject”)，添加在“case”列表中，如下：

lazy val root = (project in file("."))
  .settings(
    sparkSettings,
    name := "spark-assembly",
    mainClass in assembly := Some("com.mirana.spark.ExampleApp"),
    assemblyJarName in assembly := "scala-spark-final.jar",
    assemblyMergeStrategy in assembly := {
      //    需要合并的内容
      case PathList("javax", "inject", xs @ _*) => MergeStrategy.last
      //    保持不变
      case x =>
        val oldStrategy = (assemblyMergeStrategy in assembly).value
        oldStrategy(x)
    }
 )

完整的配置内容及项目，请参见我的git示例-spark-assembly。

参考文档

1. sbt-assembly
2. sbt公布assembly解决jar包冲突 deduplicate: different file contents found in the following