hadoop 初学者在eclipse中开发的问题

那最简单的IO说吧在本地文件系统和HDFS文件系统中的IO操作

下面这个例子是合并本地文件系统的目录下的文件到HDFS文件系统

public static void main(String[] args) throws IOException, URISyntaxException {
        //Hadoop resource configuration class
        Configuration conf=new Configuration();
         //get FileSystem's instance
        //call factory method FileSystem.get(Configuration conf)
        FileSystem hdfs=FileSystem.get(new URI("hdfs://127.0.0.1:9000"),conf);
        //local FileSystem's instance
        FileSystem local =FileSystem.getLocal(conf);
        //Path Object encode file and direcotory
        Path inputDir=new Path("/home/hadoop/Documents"); //(args[0]);
        Path hdfsFile=new Path("/mytest_example2.txt");
        //use the FileSystem's listStatus()method() to get a list of files in a directory
        FileStatus[] inputFiles=local.listStatus(inputDir);

        //create FSDataOutputStream
        FSDataOutputStream out=hdfs.create(hdfsFile);
        //loop local files and merge a new file
        for(int i=0;i<inputFiles.length;i++)
        {
            //user the Path to request an FSDataInputStream object for reading in the file
            System.err.println(inputFiles[i].getPath().getName()+"-----------------");
            FSDataInputStream in =local.open(inputFiles[i].getPath());
            byte [] buffer=new byte[256];
            int bytesRead=0;
            while ((bytesRead=in.read(buffer))>0)
            {
                  out.write(buffer, 0, bytesRead);
            }
            in.close();
        }
        out.close();
    }

以上的代码是可行的没有什么错误。但是如果是一下这样，就会报错前提呢在你的项目的classpath下也就是bin目录下没有hadoop的HDFS配置文件（core-site.xml和hdfs-site.xml）

FileSystem hdfs=FileSystem.get(/*new URI("hdfs://127.0.0.1:9000"),*/conf);

Path hdfsFile=new Path("hdfs://127.0.0.1:9000/mytest_example2.txt");

错误提示：

log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: hdfs://127.0.0.1:9000/, expected: file:///
    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:646)
    at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:82)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:513)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:499)
    at org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:594)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:448)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:776)
    at com.liweichao.PutMerge.main(PutMerge.java:40)

解决途径一个是在初始化Filesystem的时候

1 ： Configuration conf=new Configuration();
//get FileSystem's instance
//call factory method FileSystem.get(Configuration conf)

//就会自动加载到hadoop的资源配置文件
FileSystem hdfs=FileSystem.get(new URI("hdfs://127.0.0.1:9000"),conf);

在有Path对象定位的具体目录

Path hdfsFile=new Path("/mytest_example2.txt");

2：在你的项目的classpath下也就是bin目录下没有hadoop的HDFS配置文件（core-site.xml和hdfs-site.xml）

hadoop 初学者在eclipse中开发的问题

猜你喜欢