Spark配置
在${SPARK_HOME}/conf/目录下新建hive-site.xml文件，并在文件中做如下配置：
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- SparkSQL on Hive config. Start -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://hadoopSvr3:9083</value>
</property>
<!-- SparkSQL on Hive config. End -->
</configuration>
spark集群中所有节点均做同样配置
启动Hive的metastore服务
nohup hive --service metastore &
重启spark集群
验证配置是否成功
在spark集群的任一节点输入如下命令：
spark-sql --master yarn
如果看到如下信息，表示配置成功
2019-03-12 15:15:03 INFO metastore:376 - Trying to connect to metastore with URI thrift://hadoopSvr3:9083
2019-03-12 15:15:03 INFO metastore:472 - Connected to metastore.
2019-03-12 15:15:04 INFO SessionState:641 - Created local directory: /tmp/825db99e-682a-4eae-b7ae-51f84ab85acf_resources
2019-03-12 15:15:04 INFO SessionState:641 - Created HDFS directory: /tmp/hive/root/825db99e-682a-4eae-b7ae-51f84ab85acf
2019-03-12 15:15:04 INFO SessionState:641 - Created local directory: /tmp/root/825db99e-682a-4eae-b7ae-51f84ab85acf
2019-03-12 15:15:04 INFO SessionState:641 - Created HDFS directory: /tmp/hive/root/825db99e-682a-4eae-b7ae-51f84ab85acf/_tmp_space.db
2019-03-12 15:15:04 INFO SparkContext:54 - Running Spark version 2.4.0
2019-03-12 15:15:04 INFO SparkContext:54 - Submitted application: SparkSQL::10.62.124.41
2019-03-12 15:15:04 INFO SecurityManager:54 - Changing view acls to: root
2019-03-12 15:15:04 INFO SecurityManager:54 - Changing modify acls to: root
2019-03-12 15:15:04 INFO SecurityManager:54 - Changing view acls groups to:
2019-03-12 15:15:04 INFO SecurityManager:54 - Changing modify acls groups to:
2019-03-12 15:15:04 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
2019-03-12 15:15:05 INFO Utils:54 - Successfully started service 'sparkDriver' on port 55772.
2019-03-12 15:15:05 INFO SparkEnv:54 - Registering MapOutputTracker
2019-03-12 15:15:05 INFO SparkEnv:54 - Registering BlockManagerMaster
2019-03-12 15:15:05 INFO BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2019-03-12 15:15:05 INFO BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2019-03-12 15:15:05 INFO DiskBlockManager:54 - Created local directory at /data/spark/tmp/blockmgr-82285884-46c9-4206-8c55-e59cfa8c0e22
2019-03-12 15:15:05 INFO MemoryStore:54 - MemoryStore started with capacity 366.3 MB
2019-03-12 15:15:05 INFO SparkEnv:54 - Registering OutputCommitCoordinator
2019-03-12 15:15:05 INFO log:192 - Logging initialized @8913ms
2019-03-12 15:15:05 INFO Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2019-03-12 15:15:05 INFO Server:419 - Started @9123ms
2019-03-12 15:15:05 INFO AbstractConnector:278 - Started ServerConnector@2eba55cb{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2019-03-12 15:15:05 INFO Utils:54 - Successfully started service 'SparkUI' on port 4040.
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7b676112{/jobs,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@ed91d8d{/jobs/json,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@446626a7{/jobs/job,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4a2929a4{/jobs/job/json,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@cda6019{/stages,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@797c3c3b{/stages/json,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4012d5bc{/stages/stage,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4f5b08d{/stages/stage/json,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@529c2a9a{/stages/pool,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3c87fdf2{/stages/pool/json,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@26bbe604{/storage,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@fe34b86{/storage/json,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3c98781a{/storage/rdd,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3f736a16{/storage/rdd/json,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4601203a{/environment,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@53abfc07{/environment/json,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2c8c16c0{/executors,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@80bfa9d{/executors/json,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@47c40b56{/executors/threadDump,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4b039c6d{/executors/threadDump/json,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7f5b9db{/static,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@329bad59{/,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@862f408{/api,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@432f521f{/jobs/job/kill,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2d7a9786{/stages/stage/kill,null,AVAILABLE,@Spark}
2019-03-12 15:15:05 INFO SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://hadoopSvr1:4040
2019-03-12 15:15:05 INFO Utils:54 - Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
2019-03-12 15:15:05 INFO RMProxy:98 - Connecting to ResourceManager at hadoopSvr3/10.62.124.43:8032
2019-03-12 15:15:06 INFO Client:54 - Requesting a new application from cluster with 3 NodeManagers
2019-03-12 15:15:06 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
2019-03-12 15:15:06 INFO Client:54 - Will allocate AM container, with 896 MB memory including 384 MB overhead
2019-03-12 15:15:06 INFO Client:54 - Setting up container launch context for our AM
2019-03-12 15:15:06 INFO Client:54 - Setting up the launch environment for our AM container
2019-03-12 15:15:06 INFO Client:54 - Preparing resources for our AM container
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/JavaEWAH-0.3.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/RoaringBitmap-0.5.11.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/ST4-4.0.4.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/activation-1.1.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/aircompressor-0.10.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/antlr-2.7.7.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/antlr-runtime-3.4.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/antlr4-runtime-4.7.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/aopalliance-1.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/aopalliance-repackaged-2.4.0-b34.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/apache-log4j-extras-1.2.17.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/apacheds-i18n-2.0.0-M15.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/apacheds-kerberos-codec-2.0.0-M15.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/api-asn1-api-1.0.0-M20.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/api-util-1.0.0-M20.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/arpack_combined_all-0.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/arrow-format-0.10.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/arrow-memory-0.10.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/arrow-vector-0.10.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/automaton-1.11-8.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/avro-1.8.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/avro-ipc-1.8.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/avro-mapred-1.8.2-hadoop2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/bonecp-0.8.0.RELEASE.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/breeze-macros_2.11-0.13.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/breeze_2.11-0.13.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/calcite-avatica-1.2.0-incubating.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/calcite-core-1.2.0-incubating.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/calcite-linq4j-1.2.0-incubating.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/chill-java-0.9.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/chill_2.11-0.9.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-beanutils-1.7.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-beanutils-core-1.8.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-cli-1.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-codec-1.10.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-collections-3.2.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-compiler-3.0.9.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-compress-1.8.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-configuration-1.6.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-crypto-1.0.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-dbcp-1.4.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-digester-1.8.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-httpclient-3.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-io-2.4.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-lang-2.6.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-lang3-3.5.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-logging-1.1.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-math3-3.4.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-net-3.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/commons-pool-1.5.4.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/compress-lzf-1.0.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/core-1.1.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/curator-client-2.7.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/curator-framework-2.7.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/curator-recipes-2.7.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/datanucleus-api-jdo-3.2.6.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/datanucleus-core-3.2.10.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/datanucleus-rdbms-3.2.9.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/derby-10.12.1.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/eigenbase-properties-1.1.5.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/flatbuffers-1.2.0-3f79e055.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/generex-1.0.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/gson-2.2.4.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/guava-14.0.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/guice-3.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/guice-servlet-3.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-annotations-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-auth-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-client-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-common-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-hdfs-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-mapreduce-client-app-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-mapreduce-client-common-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-mapreduce-client-core-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-mapreduce-client-jobclient-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-mapreduce-client-shuffle-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-yarn-api-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-yarn-client-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-yarn-common-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-yarn-server-common-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hadoop-yarn-server-web-proxy-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hive-beeline-1.2.1.spark2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hive-cli-1.2.1.spark2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hive-exec-1.2.1.spark2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hive-jdbc-1.2.1.spark2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hive-metastore-1.2.1.spark2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hk2-api-2.4.0-b34.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hk2-locator-2.4.0-b34.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hk2-utils-2.4.0-b34.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/hppc-0.7.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/htrace-core-3.1.0-incubating.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/httpclient-4.5.6.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/httpcore-4.4.10.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/ivy-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jackson-annotations-2.6.7.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jackson-core-2.6.7.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jackson-core-asl-1.9.13.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jackson-databind-2.6.7.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jackson-dataformat-yaml-2.6.7.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jackson-jaxrs-1.9.13.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jackson-mapper-asl-1.9.13.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jackson-module-jaxb-annotations-2.6.7.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jackson-module-paranamer-2.7.9.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jackson-module-scala_2.11-2.6.7.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jackson-xc-1.9.13.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/janino-3.0.9.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/javassist-3.18.1-GA.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/javax.annotation-api-1.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/javax.inject-1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/javax.inject-2.4.0-b34.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/javax.servlet-api-3.1.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/javax.ws.rs-api-2.0.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/javolution-5.5.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jaxb-api-2.2.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jcl-over-slf4j-1.7.16.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jdo-api-3.0.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jersey-client-2.22.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jersey-common-2.22.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jersey-container-servlet-2.22.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jersey-container-servlet-core-2.22.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jersey-guava-2.22.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jersey-media-jaxb-2.22.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jersey-server-2.22.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jetty-6.1.26.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jetty-util-6.1.26.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jline-2.14.6.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/joda-time-2.9.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jodd-core-3.5.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jpam-1.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/json4s-ast_2.11-3.5.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/json4s-core_2.11-3.5.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/json4s-jackson_2.11-3.5.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/json4s-scalap_2.11-3.5.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jsp-api-2.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jsr305-1.3.9.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jta-1.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jtransforms-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/jul-to-slf4j-1.7.16.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/kryo-shaded-4.0.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/kubernetes-client-3.0.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/kubernetes-model-2.0.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/leveldbjni-all-1.8.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/libfb303-0.9.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/libthrift-0.9.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/log4j-1.2.17.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/logging-interceptor-3.8.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/lz4-java-1.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/machinist_2.11-0.6.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/macro-compat_2.11-1.1.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/mesos-1.4.0-shaded-protobuf.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/metrics-core-3.1.5.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/metrics-graphite-3.1.5.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/metrics-json-3.1.5.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/metrics-jvm-3.1.5.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/minlog-1.3.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/netty-3.9.9.Final.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/netty-all-4.1.17.Final.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/objenesis-2.5.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/okhttp-3.8.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/okio-1.13.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/opencsv-2.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/orc-core-1.5.2-nohive.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/orc-mapreduce-1.5.2-nohive.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/orc-shims-1.5.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/oro-2.0.8.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/osgi-resource-locator-1.0.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/paranamer-2.8.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/parquet-column-1.10.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/parquet-common-1.10.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/parquet-encoding-1.10.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/parquet-format-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/parquet-hadoop-1.10.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/parquet-hadoop-bundle-1.6.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/parquet-jackson-1.10.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/protobuf-java-2.5.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/py4j-0.10.7.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/pyrolite-4.13.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/scala-compiler-2.11.12.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/scala-library-2.11.12.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/scala-parser-combinators_2.11-1.1.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/scala-reflect-2.11.12.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/scala-xml_2.11-1.0.5.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/shapeless_2.11-2.3.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/slf4j-api-1.7.16.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/snakeyaml-1.15.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/snappy-0.2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/snappy-java-1.1.7.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-catalyst_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-core_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-graphx_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-hive-thriftserver_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-hive_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-kubernetes_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-kvstore_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-launcher_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-mesos_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-mllib-local_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-mllib_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-network-common_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-network-shuffle_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-repl_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-sketch_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-sql_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-streaming_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-tags_2.11-2.4.0-tests.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-tags_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-unsafe_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spark-yarn_2.11-2.4.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spire-macros_2.11-0.13.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/spire_2.11-0.13.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/stax-api-1.0-2.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/stax-api-1.0.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/stream-2.7.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/stringtemplate-3.2.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/super-csv-2.2.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/univocity-parsers-2.7.3.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/validation-api-1.1.0.Final.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/xbean-asm6-shaded-4.8.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/xercesImpl-2.9.1.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/xmlenc-0.52.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/xz-1.5.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/zjsonpatch-0.3.0.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/zookeeper-3.4.6.jar
2019-03-12 15:15:06 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://hadoopSvr1:8020/spark-jars/zstd-jni-1.3.2-2.jar
2019-03-12 15:15:06 INFO Client:54 - Uploading resource file:/data/spark/tmp/spark-238dbffc-ee58-4416-814a-8e919168f601/__spark_conf__4342402280708496926.zip -> hdfs://hadoopSvr1:8020/user/root/.sparkStaging/application_1552122052739_0004/__spark_conf__.zip
2019-03-12 15:15:06 INFO SecurityManager:54 - Changing view acls to: root
2019-03-12 15:15:06 INFO SecurityManager:54 - Changing modify acls to: root
2019-03-12 15:15:06 INFO SecurityManager:54 - Changing view acls groups to:
2019-03-12 15:15:06 INFO SecurityManager:54 - Changing modify acls groups to:
2019-03-12 15:15:06 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
2019-03-12 15:15:08 INFO Client:54 - Submitting application application_1552122052739_0004 to ResourceManager
2019-03-12 15:15:08 INFO YarnClientImpl:273 - Submitted application application_1552122052739_0004
2019-03-12 15:15:08 INFO SchedulerExtensionServices:54 - Starting Yarn extension services with app application_1552122052739_0004 and attemptId None
2019-03-12 15:15:09 INFO Client:54 - Application report for application_1552122052739_0004 (state: ACCEPTED)
2019-03-12 15:15:09 INFO Client:54 -
client token: N/A
diagnostics: AM container is launched, waiting for AM container to Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.dev
start time: 1552374908119
final status: UNDEFINED
tracking URL: http://hadoopSvr3:8088/proxy/application_1552122052739_0004/
user: root
2019-03-12 15:15:10 INFO Client:54 - Application report for application_1552122052739_0004 (state: ACCEPTED)
2019-03-12 15:15:11 INFO Client:54 - Application report for application_1552122052739_0004 (state: ACCEPTED)
2019-03-12 15:15:12 INFO Client:54 - Application report for application_1552122052739_0004 (state: ACCEPTED)
2019-03-12 15:15:13 INFO Client:54 - Application report for application_1552122052739_0004 (state: ACCEPTED)
2019-03-12 15:15:14 INFO Client:54 - Application report for application_1552122052739_0004 (state: ACCEPTED)
2019-03-12 15:15:15 INFO Client:54 - Application report for application_1552122052739_0004 (state: ACCEPTED)
2019-03-12 15:15:16 INFO Client:54 - Application report for application_1552122052739_0004 (state: RUNNING)
2019-03-12 15:15:16 INFO Client:54 -
client token: N/A
diagnostics: N/A
ApplicationMaster host: 10.62.124.43
ApplicationMaster RPC port: -1
queue: root.dev
start time: 1552374908119
final status: UNDEFINED
tracking URL: http://hadoopSvr3:8088/proxy/application_1552122052739_0004/
user: root
2019-03-12 15:15:16 INFO YarnClientSchedulerBackend:54 - Application application_1552122052739_0004 has started running.
2019-03-12 15:15:16 INFO Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40408.
2019-03-12 15:15:16 INFO NettyBlockTransferService:54 - Server created on hadoopSvr1:40408
2019-03-12 15:15:16 INFO BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2019-03-12 15:15:16 INFO BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, hadoopSvr1, 40408, None)
2019-03-12 15:15:16 INFO BlockManagerMasterEndpoint:54 - Registering block manager hadoopSvr1:40408 with 366.3 MB RAM, BlockManagerId(driver, hadoopSvr1, 40408, None)
2019-03-12 15:15:16 INFO BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, hadoopSvr1, 40408, None)
2019-03-12 15:15:16 INFO BlockManager:54 - external shuffle service port = 7337
2019-03-12 15:15:16 INFO BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, hadoopSvr1, 40408, None)
2019-03-12 15:15:16 INFO YarnClientSchedulerBackend:54 - Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> hadoopSvr3, PROXY_URI_BASES -> http://hadoopSvr3:8088/proxy/application_1552122052739_0004), /proxy/application_1552122052739_0004
2019-03-12 15:15:16 INFO JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /jobs, /jobs/json, /jobs/job, /jobs/job/json, /stages, /stages/json, /stages/stage, /stages/stage/json, /stages/pool, /stages/pool/json, /storage, /storage/json, /storage/rdd, /storage/rdd/json, /environment, /environment/json, /executors, /executors/json, /executors/threadDump, /executors/threadDump/json, /static, /, /api, /jobs/job/kill, /stages/stage/kill.
2019-03-12 15:15:16 INFO YarnSchedulerBackend$YarnSchedulerEndpoint:54 - ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
2019-03-12 15:15:16 INFO JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /metrics/json.
2019-03-12 15:15:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@42aa1324{/metrics/json,null,AVAILABLE,@Spark}
2019-03-12 15:15:16 INFO Utils:54 - Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
2019-03-12 15:15:16 INFO YarnClientSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
2019-03-12 15:15:16 INFO SharedState:54 - loading hive config file: file:/usr/local/spark/conf/hive-site.xml
2019-03-12 15:15:16 INFO SharedState:54 - spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
2019-03-12 15:15:16 INFO SharedState:54 - Warehouse path is '/user/hive/warehouse'.
2019-03-12 15:15:16 INFO JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL.
2019-03-12 15:15:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@9f0fc36{/SQL,null,AVAILABLE,@Spark}
2019-03-12 15:15:16 INFO JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/json.
2019-03-12 15:15:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5a06904{/SQL/json,null,AVAILABLE,@Spark}
2019-03-12 15:15:16 INFO JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution.
2019-03-12 15:15:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@286866cb{/SQL/execution,null,AVAILABLE,@Spark}
2019-03-12 15:15:16 INFO JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution/json.
2019-03-12 15:15:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@56ec6960{/SQL/execution/json,null,AVAILABLE,@Spark}
2019-03-12 15:15:16 INFO JettyUtils:54 - Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql.
2019-03-12 15:15:16 INFO ContextHandler:781 - Started o.s.j.s.ServletContextHandler@46ea78f0{/static/sql,null,AVAILABLE,@Spark}
2019-03-12 15:15:17 INFO HiveUtils:54 - Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
2019-03-12 15:15:17 INFO HiveClientImpl:54 - Warehouse location for Hive client (version 1.2.2) is /user/hive/warehouse
2019-03-12 15:15:17 INFO StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint
Spark master: yarn, Application Id: application_1552122052739_0004
2019-03-12 15:15:18 INFO SparkSQLCLIDriver:951 - Spark master: yarn, Application Id: application_1552122052739_0004
spark-sql>
进一步验证
spark-sql> show databases;
2019-03-12 15:18:15 INFO CodeGenerator:54 - Code generated in 198.952751 ms
default
kaidy
sparktest
tomasdata
Time taken: 1.31 seconds, Fetched 4 row(s)
2019-03-12 15:18:15 INFO SparkSQLCLIDriver:951 - Time taken: 1.31 seconds, Fetched 4 row(s)
spark-sql>