解决Input path does not exist:file://

任务提交指令:

spark-submit  --class orderinversioninspark.RelativeFrequency ~/soft/temp/bigdataAnalysis.jar 2 /input/inputfile.txt /outputfi

异常部分信息如下:

Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file://input/inputfile.txt
    at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:285)

        at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)

        at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)

按理说,默认应该寻找HDFS目录下的文件,现在却找的是本地文件。具体原因想了很久也没想明白,待续。

附上自己的解决方案:

spark-submit --class orderinversioninspark.RelativeFrequency ~/soft/temp/bigdataAnalysis.jar 2 hdfs://master:8020/input/inputfile.txt /outputfi0

即添加完整的hdfs URL。

另外,若要使用本地文件的话那么URL改为:file:///home/centos/soft/files/inputfile.txt

猜你喜欢

转载自blog.csdn.net/perfer258/article/details/80714407