hadoop集群Hadoop client 不在同一网段;本地eclipse连接云服务器hadoop集群,无法上传下载文件等操作;

问题:

    @Before
    public void init() throws Exception{
        conf = new Configuration();
        conf.set("fs.defaultFS", "hdfs://地址:9000/");
        //拿到一个文件系统操作的客户端实例对象
        /*fs = FileSystem.get(conf);*/
        //可以直接传入 uri和用户身份
        fs = FileSystem.get(new URI("hdfs://地址:9000/"),conf,"root"); //最后一个参数为用户名
    }
    @Test
    public void testUpload() throws Exception {
        Thread.sleep(2000);
        fs.copyFromLocalFile(new Path("D:/folder/test.gz"), new Path("/test.gz"));
        fs.close();
    }

运行显示打包完成,但无法连接datanode。namenode和datanode通信在局域网的。但为了学习之用,可以促使client在hadoop的内网中,可以在云服务器中配置一个vpn,用于客户端连接。

[main] DEBUG org.apache.hadoop.hdfs.DFSClient - Queued packet 80
[main] DEBUG org.apache.hadoop.hdfs.DFSClient - computePacketChunkSize: src=/test/test.gz, chunkSize=516, chunksPerPacket=127, packetSize=65532
[main] DEBUG org.apache.hadoop.hdfs.DFSClient - DFSClient writeChunk allocating new packet seqno=81, src=/test/test.gz, packetSize=65532, chunksPerPacket=127, bytesCurBlock=5266944
[main] DEBUG org.apache.hadoop.hdfs.DFSClient - DFSClient writeChunk packet full seqno=81, src=/test/test.gz, bytesCurBlock=5331968, blockSize=134217728, appendChunk=false
[IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client - IPC Client (22429093) connection to /117.48.231.43:9000 from root sending #5
[IPC Client (22429093) connection to /117.48.231.43:9000 from root] DEBUG org.apache.hadoop.ipc.Client - IPC Client (22429093) connection to /117.48.231.43:9000 from root got value #5
[Thread-3] DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine - Call: addBlock took 40ms
[Thread-3] DEBUG org.apache.hadoop.hdfs.DFSClient - pipeline = 192.168.0.3:50010
[Thread-3] DEBUG org.apache.hadoop.hdfs.DFSClient - Connecting to datanode 192.168.0.3:50010
[IPC Client (22429093) connection to /117.48.231.43:9000 from root] DEBUG org.apache.hadoop.ipc.Client - IPC Client (22429093) connection to /117.48.231.43:9000 from root: closed
[IPC Client (22429093) connection to /117.48.231.43:9000 from root] DEBUG org.apache.hadoop.ipc.Client - IPC Client (22429093) connection to /117.48.231.43:9000 from root: stopped, remaining connections 0

 连接vpn后的客户端地址:

猜你喜欢

转载自www.cnblogs.com/mryangbo/p/9341361.html