Do you understand cluster monitoring it? (B) - HDFS some indicators

This article then continued on article content, address: IDC cluster related indicators acquisition
after acquisition of its own index corresponding IDC machine, you also need to HDFS and Hadoop cluster YARN indicators for the collection, you can have two kinds on the general idea:

The first course, you can continue to use the CM API to acquire, as the CM tssql provides a very rich variety of indicators to monitor
the second namely to acquire data through jmxJ, in fact, by accessing the relevant URL above, and will be analyzing the json, so we need to obtain the data, the data will eventually merge together, the timing to perform the acquisition operation
using jmx such a manner in the actual practice of the process to be acquired, url requests related to the following:
HTTP : // localhost: 50070 / = JMX qry Hadoop:? Service = the NameNode, NameNodeInfo name =
HTTP: // localhost: 50070 / = JMX qry Hadoop:? Service = the NameNode, name = FSNamesystemState

Specific code implementation ideas are as follows:

You first need to have a httpclient, destination server initiates a request to obtain the corresponding json data here have written StatefulHttpClient
secondly JsonUtil the utility class for converting between data types and objects Json
Of course, we also need to be needed Gets monitoring indicators to sort it out, write our entity, here for HDFS, for example, mainly HdfsSummary and DataNodeInfo
this case the code on github, address:
here mainly to show the core code:

MonitorMetrics.java：

public class MonitorMetrics {
    // beans为通过jmx所返回的json串中最起始的key
    // 结构为{"beans":[{"":"","":"",...}]}
    List<Map<String, Object>> beans = new ArrayList<>();

    public List<Map<String, Object>> getBeans() {
        return beans;
    }

    public void setBeans(List<Map<String, Object>> beans) {
        this.beans = beans;
    }

    public Object getMetricsValue(String name) {
        if (beans.isEmpty()) {
            return null;
        }
        return beans.get(0).getOrDefault(name, null);
    }
}

HadoopUtil.java：

public class HadoopUtil {
    public static long gbLength = 1073741824L;
    public static final String hadoopJmxServerUrl = "http://localhost:50070";
    public static final String jmxServerUrlFormat = "%s/jmx?qry=%s";
    public static final String nameNodeInfo = "Hadoop:service=NameNode,name=NameNodeInfo";
    public static final String fsNameSystemState = "Hadoop:service=NameNode,name=FSNamesystemState";

    public static HdfsSummary getHdfsSummary(StatefulHttpClient client) throws IOException {
        HdfsSummary hdfsSummary = new HdfsSummary();
        String namenodeUrl = String.format(jmxServerUrlFormat, hadoopJmxServerUrl, nameNodeInfo);
        MonitorMetrics monitorMetrics = client.get(MonitorMetrics.class, namenodeUrl, null, null);
        hdfsSummary.setTotal(doubleFormat(monitorMetrics.getMetricsValue("Total"), gbLength));
        hdfsSummary.setDfsFree(doubleFormat(monitorMetrics.getMetricsValue("Free"), gbLength));
        hdfsSummary.setDfsUsed(doubleFormat(monitorMetrics.getMetricsValue("Used"), gbLength));
        hdfsSummary.setPercentUsed(doubleFormat(monitorMetrics.getMetricsValue("PercentUsed")));
        hdfsSummary.setSafeMode(monitorMetrics.getMetricsValue("Safemode").toString());
        hdfsSummary.setNonDfsUsed(doubleFormat(monitorMetrics.getMetricsValue("NonDfsUsedSpace"), gbLength));
        hdfsSummary.setBlockPoolUsedSpace(doubleFormat(monitorMetrics.getMetricsValue("BlockPoolUsedSpace"), gbLength));
        hdfsSummary.setPercentBlockPoolUsed(doubleFormat(monitorMetrics.getMetricsValue("PercentBlockPoolUsed")));
        hdfsSummary.setPercentRemaining(doubleFormat(monitorMetrics.getMetricsValue("PercentRemaining")));
        hdfsSummary.setTotalBlocks((int) monitorMetrics.getMetricsValue("TotalBlocks"));
        hdfsSummary.setTotalFiles((int) monitorMetrics.getMetricsValue("TotalFiles"));
        hdfsSummary.setMissingBlocks((int) monitorMetrics.getMetricsValue("NumberOfMissingBlocks"));

        String liveNodesJson = monitorMetrics.getMetricsValue("LiveNodes").toString();
        String deadNodesJson = monitorMetrics.getMetricsValue("DeadNodes").toString();
        List<DataNodeInfo> liveNodes = dataNodeInfoReader(liveNodesJson);
        List<DataNodeInfo> deadNodes = dataNodeInfoReader(deadNodesJson);
        hdfsSummary.setLiveDataNodeInfos(liveNodes);
        hdfsSummary.setDeadDataNodeInfos(deadNodes);

        String fsNameSystemStateUrl = String.format(jmxServerUrlFormat, hadoopJmxServerUrl, fsNameSystemState);
        MonitorMetrics hadoopMetrics = client.get(MonitorMetrics.class, fsNameSystemStateUrl, null, null);
        hdfsSummary.setNumLiveDataNodes((int) hadoopMetrics.getMetricsValue("NumLiveDataNodes"));
        hdfsSummary.setNumDeadDataNodes((int) hadoopMetrics.getMetricsValue("NumDeadDataNodes"));
        hdfsSummary.setVolumeFailuresTotal((int) hadoopMetrics.getMetricsValue("VolumeFailuresTotal"));

        return hdfsSummary;
    }

    public static List<DataNodeInfo> dataNodeInfoReader(String jsonData) throws IOException {
        List<DataNodeInfo> dataNodeInfos = new ArrayList<DataNodeInfo>();
        Map<String, Object> nodes = JsonUtil.fromJsonMap(String.class, Object.class, jsonData);
        for (Map.Entry<String, Object> node : nodes.entrySet()) {
            Map<String, Object> info = (HashMap<String, Object>) node.getValue();
            String nodeName = node.getKey().split(":")[0];
            DataNodeInfo dataNodeInfo = new DataNodeInfo();
            dataNodeInfo.setNodeName(nodeName);
            dataNodeInfo.setNodeAddr(info.get("infoAddr").toString().split(":")[0]);
            dataNodeInfo.setLastContact((int) info.get("lastContact"));
            dataNodeInfo.setUsedSpace(doubleFormat(info.get("usedSpace"), gbLength));
            dataNodeInfo.setAdminState(info.get("adminState").toString());
            dataNodeInfo.setNonDfsUsedSpace(doubleFormat(info.get("nonDfsUsedSpace"), gbLength));
            dataNodeInfo.setCapacity(doubleFormat(info.get("capacity"), gbLength));
            dataNodeInfo.setNumBlocks((int) info.get("numBlocks"));
            dataNodeInfo.setRemaining(doubleFormat(info.get("remaining"), gbLength));
            dataNodeInfo.setBlockPoolUsed(doubleFormat(info.get("blockPoolUsed"), gbLength));
            dataNodeInfo.setBlockPoolUsedPerent(doubleFormat(info.get("blockPoolUsedPercent")));

            dataNodeInfos.add(dataNodeInfo);
        }

        return dataNodeInfos;
    }

    public static DecimalFormat df = new DecimalFormat("#.##");

    public static double doubleFormat(Object num, long unit) {
        double result = Double.parseDouble(String.valueOf(num)) / unit;
        return Double.parseDouble(df.format(result));
    }

    public static double doubleFormat(Object num) {
        double result = Double.parseDouble(String.valueOf(num));
        return Double.parseDouble(df.format(result));
    }

    public static void main(String[] args) {
        String res = String.format(jmxServerUrlFormat, hadoopJmxServerUrl, nameNodeInfo);
        System.out.println(res);
    }

}

MonitorApp.java：

public class MonitorApp {

    public static void main(String[] args) throws IOException {
        StatefulHttpClient client = new StatefulHttpClient(null);
        HadoopUtil.getHdfsSummary(client).printInfo();

    }
}

The final results show the following:

YARN indicators on access to, similar ideas, there is no longer shows