Spark Shell 启动NullPointerException提示getZipEntry
1. 软件版本：
| 软件 | 版本 |
|---|---|
| CDH | CDH-5.7.3-1.cdh5.7.3.p0.5 |
| Spark | Standalone模式，1.6.0 |
| JDK | 1.8.0_131 |
2. 问题描述：
在使用Spark Shell的过程中，CDH其中的一个节点，忽然执行不了spark-shell命令，报类似如下的错误：
Exception in thread "main" java.lang.NullPointerException
at java.util.zip.ZipFile.getZipEntry(ZipFile.java:566)
at java.util.zip.ZipFile.access$900(ZipFile.java:60)
at java.util.zip.ZipFile$ZipEntryIterator.next(ZipFile.java:524)
at java.util.zip.ZipFile$ZipEntryIterator.nextElement(ZipFile.java:499)
at java.util.zip.ZipFile$ZipEntryIterator.nextElement(ZipFile.java:480)
at scala.reflect.io.FileZipArchive.iterator(ZipArchive.scala:135)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.reflect.io.AbstractFile.foreach(AbstractFile.scala:91)
at scala.tools.nsc.util.DirectoryClassPath.traverse(ClassPath.scala:308)
at scala.tools.nsc.util.DirectoryClassPath.x$16$lzycompute(ClassPath.scala:317)
at scala.tools.nsc.util.DirectoryClassPath.x$16(ClassPath.scala:317)
at scala.tools.nsc.util.DirectoryClassPath.packages$lzycompute(ClassPath.scala:317)
at scala.tools.nsc.util.DirectoryClassPath.packages(ClassPath.scala:317)
at scala.tools.nsc.util.DirectoryClassPath.packages(ClassPath.scala:297)
at scala.tools.nsc.util.MergedClassPath$$anonfun$packages$1.apply(ClassPath.scala:375)
at scala.tools.nsc.util.MergedClassPath$$anonfun$packages$1.apply(ClassPath.scala:375)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.tools.nsc.util.MergedClassPath.packages$lzycompute(ClassPath.scala:375)
at scala.tools.nsc.util.MergedClassPath.packages(ClassPath.scala:370)
at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:243)
at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:194)
at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1231)
at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:240)
at scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:59)
at scala.tools.nsc.Global.rootMirror(Global.scala:57)
at scala.tools.nsc.Global.rootMirror(Global.scala:37)
at scala.reflect.internal.Definitions$DefinitionsClass.<init>(Definitions.scala:166)
at scala.reflect.internal.Definitions$definitions$.<init>(Definitions.scala:20)
at scala.reflect.internal.SymbolTable.definitions$lzycompute(SymbolTable.scala:13)
at scala.reflect.internal.SymbolTable.definitions(SymbolTable.scala:13)
at scala.tools.nsc.Global$Run.<init>(Global.scala:1290)
at org.apache.spark.repl.SparkIMain.org$apache$spark$repl$SparkIMain$$_initialize(SparkIMain.scala:198)
at org.apache.spark.repl.SparkIMain.initializeSynchronous(SparkIMain.scala:227)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:990)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1064)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
初步怀疑是classpath的问题，经查询：https://stackoverflow.com/questions/14993621/how-to-debug-nullpointerexception-occurred-in-library-code，确实是classpath的问题。
3. 问题解决思路
- 初步怀疑可能是当前节点的spark-assemble.jar损坏，拷贝其他节点相应jar到此节点，无效；
- 对比/opt/cloudera/parcels/CDH-5.7.3-1.cdh5.7.3.p0.5/lib/spark/conf 目录下面的classpath.txt （此文件是classpath的加载路径，会在进行初始化的时候读取），发现当前节点多了一些记录，故此判断是多的记录引起的，把多余的记录删掉，发现启动成功；
4. 后续
解决后，要还原此问题，发现：
- 有多2条记录，其中一个是mysql jdbc jar；所以把mysql jdbc jar添加到classpath发现仍不会报错；
- 由于另一个记录是一个自定义的jar包，所以是否是因为这个自定义的jar包不存在而导致的。经试验，在classpath.txt中随便加一个不存在的jar包记录，发现仍能启动；故此，添加的Jar包应该是存在的。
- 既然Jar包存在，找到其Jar包，并查看其内容，出现如下所示的异常：
[root@server1 ~]# jar vtf demo.Driver.jar
java.lang.NullPointerException
at java.util.zip.ZipFile.getZipEntry(ZipFile.java:566)
at java.util.zip.ZipFile.access$900(ZipFile.java:60)
at java.util.zip.ZipFile$ZipEntryIterator.next(ZipFile.java:524)
at java.util.zip.ZipFile$ZipEntryIterator.nextElement(ZipFile.java:499)
at java.util.zip.ZipFile$ZipEntryIterator.nextElement(ZipFile.java:480)
at sun.tools.jar.Main.list(Main.java:1118)
at sun.tools.jar.Main.run(Main.java:293)
at sun.tools.jar.Main.main(Main.java:1288)
所以确认是这个Jar包的问题，经确认，这个Jar包损坏。