问题:
@Before public void init() throws Exception{ conf = new Configuration(); conf.set("fs.defaultFS", "hdfs://地址:9000/"); //拿到一个文件系统操作的客户端实例对象 /*fs = FileSystem.get(conf);*/ //可以直接传入 uri和用户身份 fs = FileSystem.get(new URI("hdfs://地址:9000/"),conf,"root"); //最后一个参数为用户名 } @Test public void testUpload() throws Exception { Thread.sleep(2000); fs.copyFromLocalFile(new Path("D:/folder/test.gz"), new Path("/test.gz")); fs.close(); }
运行显示打包完成,但无法连接datanode。namenode和datanode通信在局域网的。但为了学习之用,可以促使client在hadoop的内网中,可以在云服务器中配置一个vpn,用于客户端连接。
[main] DEBUG org.apache.hadoop.hdfs.DFSClient - Queued packet 80 [main] DEBUG org.apache.hadoop.hdfs.DFSClient - computePacketChunkSize: src=/test/test.gz, chunkSize=516, chunksPerPacket=127, packetSize=65532 [main] DEBUG org.apache.hadoop.hdfs.DFSClient - DFSClient writeChunk allocating new packet seqno=81, src=/test/test.gz, packetSize=65532, chunksPerPacket=127, bytesCurBlock=5266944 [main] DEBUG org.apache.hadoop.hdfs.DFSClient - DFSClient writeChunk packet full seqno=81, src=/test/test.gz, bytesCurBlock=5331968, blockSize=134217728, appendChunk=false [IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client - IPC Client (22429093) connection to /117.48.231.43:9000 from root sending #5 [IPC Client (22429093) connection to /117.48.231.43:9000 from root] DEBUG org.apache.hadoop.ipc.Client - IPC Client (22429093) connection to /117.48.231.43:9000 from root got value #5 [Thread-3] DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine - Call: addBlock took 40ms [Thread-3] DEBUG org.apache.hadoop.hdfs.DFSClient - pipeline = 192.168.0.3:50010 [Thread-3] DEBUG org.apache.hadoop.hdfs.DFSClient - Connecting to datanode 192.168.0.3:50010 [IPC Client (22429093) connection to /117.48.231.43:9000 from root] DEBUG org.apache.hadoop.ipc.Client - IPC Client (22429093) connection to /117.48.231.43:9000 from root: closed [IPC Client (22429093) connection to /117.48.231.43:9000 from root] DEBUG org.apache.hadoop.ipc.Client - IPC Client (22429093) connection to /117.48.231.43:9000 from root: stopped, remaining connections 0
连接vpn后的客户端地址: