Java operational problems arise hdfs

Foreword

Demand is golang save data to hdfs inside, but because of verification problems kerberos, golang open source wheels did not verify something, then give up golang, use Java to encapsulate related interfaces. Specific operation code not say, talk about several issues encountered.

springboot, hadoop log conflict

The need to remove the relevant dependent and associated slf4j of the servlet tomcat hadoop inside

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <exclusions>
    <exclusion>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
    </exclusion>
        <exclusion>
            <groupId>javax.servlet</groupId>
            <artifactId>servlet-api</artifactId>
        </exclusion>
    </exclusions>
    <version>2.7.4</version>

</dependency>

Failed to APPEND_FILE问题

Or there is no real-time closed FSDataOutputStream, either condition is not set to allow additional configuration

conf.set("dfs.support.append", "true");
conf.set("dfs.client.block.write.replace-datanode-on-failure.policy", "NEVER");
conf.set("dfs.client.block.write.replace-datanode-on-failure.enable", "true");

java concurrent write hdfs file problem

hdfs not allow concurrent writes to the same file, then you can add additional written a synchronized keyword brutal methods to prevent "fs handle" concurrent use