Spark under Windows reports an error when calling Hive: java.io.IOException: Access is denied.

The error message is as follows:

18/04/27 10:36:33 INFO SessionState: Created local directory: C:/Users/GUOXIA~1.HDS/AppData/Local/Temp/c5c4c419-58fb-4b84-b84e-c6c972255f12_resources
18/04/27 10:36:33 INFO SessionState: Created HDFS directory: /tmp/hive/hdfs/c5c4c419-58fb-4b84-b84e-c6c972255f12
18/04/27 10:36:33 INFO SessionState: Created local directory: C:/Users/GUOXIA~1.HDS/AppData/Local/Temp/guoxiang/c5c4c419-58fb-4b84-b84e-c6c972255f12
18/04/27 10:36:33 INFO SessionState: Created HDFS directory: /tmp/hive/hdfs/c5c4c419-58fb-4b84-b84e-c6c972255f12/_tmp_space.db
Exception in thread "main" org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: 拒绝访问。;
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
    at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:194)
    at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
    at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:39)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:54)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:52)
    ......
    Caused by: java.io.IOException: 拒绝访问。
    at java.io.WinNTFileSystem.createFileExclusively(Native Method)
    at java.io.File.createTempFile(File.java:2024)
    at org.apache.hadoop.hive.ql.session.SessionState.createTempFile(SessionState.java:818)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:513)
    ... 34 more

Analysis:
It seems to be a permission problem of the /tmp folder, which does not allow you to create files under /tmp, so first try to modify the /tmp permissions:
use the winutils tool under hadoop to modify the tmp permissions:

winutils chmod -R 777 /tmp
winutils chmod -R 777 D:/tmp

If it still doesn't work, add to
the code:

System.setProperty("user.name", "hdfs")

That solved it, as to the cause. . . . . . not clear

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=325973659&siteId=291194637