Spark2.4-Azure HDI 缺包问题汇总

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/u012551524/article/details/87915536

---

以下缺包问题基于已经替换掉spark2.4相应hadoop包，hive包，hbase包，出现的缺包问题

---

1、

Caused by: java.lang.NoClassDefFoundError: org/apache/parquet/hadoop/ParquetOutputFormat$JobSummaryLevel

parquet-hadoop-1.10.0.jar

2、

org.apache.spark.sql.AnalysisException: Failed to find data source: kafka. Please deploy the application as per the deployment section of "Structured Streaming + Kafka Integration Guide".;

spark-sql-kafka-0-10_2.11-2.4.0.jar

3、

java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArrayDeserializer

kafka_2.11-0.10.0.1.jar

kafka-clients-0.10.0.1.jar

4、

Caused by: java.lang.NoClassDefFoundError: co/cask/tephra/TransactionSystemClient.   (版本基于phoenix、hbase版本)

tephra-api-0.7.0.jar

tephra-core-0.7.0.jar

tephra-hbase-compat-1.1-0.7.0.jar

5、

java.lang.NoClassDefFoundError: org/apache/twill/zookeeper/ZKClient

twill-common-0.8.0.jar

twill-discovery-api-0.8.0.jar

twill-discovery-core-0.8.0.jar

twill-zookeeper-0.8.0.jar

6、

phoenix缺包：

phoenix-core-4.7.0.2.6.5.3005-27.jar

phoenix-spark2-4.7.0.2.6.5.3005-27.jar