Small note --bug solve: Idea running locally Spark job, missing winutils.exe hadoop.dll file

1. The problem scenario:

  • window environment, using the idea Spark job development, job and run the job, an error
{"time":"2020-01-19 11:24:41","logtype":"WARN","loginfo":"Unable to load native-hadoop library for your platform... using builtin-java classes where applicable"}
{"time":"2020-01-19 11:24:41","logtype":"ERROR","loginfo":"Failed to locate the winutils binary in the hadoop binary path"}
java.io.IOException: Could not locate executable D:\hadoop\hadoop-2.6.0-cdh5.15.1\bin\winutils.exe in the Hadoop binaries.
	at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:407)
	at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:422)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:415)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79)
	at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:168)
	at org.apache.hadoop.security.Groups.<init>(Groups.java:132)
	at org.apache.hadoop.security.Groups.<init>(Groups.java:100)
	at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:435)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:341)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:308)
	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:895)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:861)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:728)
	at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
	at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2422)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:293)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
	at main.scala.com.xiaolin.huawei.ads.Companys$.main(Companys.scala:22)
	at main.scala.com.xiaolin.huawei.ads.Companys.main(Companys.scala)

2. solve the problem:

  • The cause of the problem: window environment is not compatible with reason, file deletion winutils.exe hadoop.dll
  • Download Path: https://github.com/steveloughran/winutils
  • The downloaded file into hadoop / bin (Note: You have to configure the system environment variable) directory, and copy hadoop.dll to the window / system32 / directory
  • Restart the computer or idea
Published 33 original articles · won praise 1 · views 2643

Guess you like

Origin blog.csdn.net/weixin_44131414/article/details/104038595