Big Data Resource Monitoring (a) - IDC room clustered index acquisition

Background:
Our own IDC room, building large data clusters based on IDC room; the need to monitor cluster resources, CDH cluster uses a cluster, collecting two main points were:

YARN HDFS and related indicators collected
IDC machine itself metrics collection
Note : Some people may have doubts, CM interface has provided charts to monitor, and why do we need to show themselves. The reason is that the information within the data need to be integrated into the platform to above, made to correspond to data reporting, visual way to display data on their platform

Realization of ideas can be roughly divided into two types:

       使用CM所提供的Java API去获取
       使用CM提供的REST API去获取

In fact, the two are essentially the same, Java API CM is also provided in accordance with the set of REST API to achieve, both of which are consistent

The core code is as follows:

public class IdcHostResource {
    private static final Logger LOGGER = LoggerFactory.getLogger(IdcHostResource.class);

static RootResourceV18 apiRoot;

// TODO... 写死了，需要改进
static {
    apiRoot = new ClouderaManagerClientBuilder()
            .withHost("cm ip")
            .withPort(7180)
            .withUsernamePassword("user", "passwd")
            .build()
            .getRootV18();
}

/**
 * 固定获取Host的基本资源信息
 */
public static List<IdcHostBasicInfo> getAllHostResource() {
    List<IdcHostBasicInfo> hosts = new ArrayList<IdcHostBasicInfo>();
    HostsResourceV10 hostsResourceV10 = apiRoot.getHostsResource();
    List<ApiHost> hostLists = hostsResourceV10.readHosts(DataView.SUMMARY).getHosts();
    LOGGER.info("Total" + hostLists.size() + "Host");
    for (ApiHost hostList : hostLists) {
        IdcHostBasicInfo host = formatHost(hostsResourceV10.readHost(hostList.getHostId()));
        LOGGER.info("Host Name：" + host.getHostName());
        LOGGER.info("Host Health Summary：" + host.gethostHealthSummary());
        LOGGER.info("Host Physical Memory：" + host.getTotalPhysMemBytes());
        hosts.add(host);
    }
    return hosts;
}

public static IdcHostBasicInfo formatHost(ApiHost apiHost) {
    IdcHostBasicInfo idcHostBasicInfo = new IdcHostBasicInfo();
    idcHostBasicInfo.sethostHealthSummary(apiHost.getHealthSummary().toString());
    idcHostBasicInfo.setHostName(apiHost.getHostname());
    idcHostBasicInfo.setTotalPhysMemBytes(apiHost.getTotalPhysMemBytes());
    return idcHostBasicInfo;
}

/**
 * 通过tsquery来动态获取对应的metrics info
 *
 * @param query
 * @param startTime
 * @param endTime
 * @return
 */
public static List<IdcMetricInfo> getHostMetrics(String query, String startTime, String endTime) throws ParseException {
    TimeSeriesResourceV11 timeSeriesResourceV11 = apiRoot.getTimeSeriesResource();
    ApiTimeSeriesResponseList responseList = timeSeriesResourceV11.queryTimeSeries(query, startTime, endTime);
    List<ApiTimeSeriesResponse> apiTimeSeriesResponseList = responseList.getResponses();
    List<IdcMetricInfo> metrics = formatApiTimeSeriesResponseList(apiTimeSeriesResponseList);
    return metrics;
}

public static List<IdcMetricInfo> formatApiTimeSeriesResponseList(List<ApiTimeSeriesResponse> apiTimeSeriesResponseList) throws ParseException {
    List<IdcMetricInfo> metrics = new ArrayList<IdcMetricInfo>();
    DateUtils dateUtils = new DateUtils();
    for (ApiTimeSeriesResponse apiTimeSeriesResponse : apiTimeSeriesResponseList) {
        List<MetricData> dataList = new ArrayList<MetricData>();
        List<ApiTimeSeries> apiTimeSeriesResponseLists = apiTimeSeriesResponse.getTimeSeries();
        for (ApiTimeSeries apiTimeSeries : apiTimeSeriesResponseLists) {
            LOGGER.info("query sql is: " + apiTimeSeries.getMetadata().getExpression());
            IdcMetricInfo metric = new IdcMetricInfo();
            metric.setMetricName(apiTimeSeries.getMetadata().getMetricName());
            metric.setEntityName(apiTimeSeries.getMetadata().getEntityName());
            metric.setStartTime(apiTimeSeries.getMetadata().getStartTime().toString());
            metric.setEndTime(apiTimeSeries.getMetadata().getEndTime().toString());
            for (ApiTimeSeriesData apiTimeSeriesData : apiTimeSeries.getData()) {
                MetricData data = new MetricData();
                // 在Data中插入EntityName，避免重复数据的产生
                data.seHostname(apiTimeSeries.getMetadata().getEntityName());
                // CM默认得到的时间格式为 EEE MMM dd HH:mm:ss 'CST' yyyy，转换时间格式为 yyyy-MM-dd HH:mm:ss
                data.setTimestamp(dateUtils.parse(apiTimeSeriesData.getTimestamp().toString()));
                data.setType(apiTimeSeriesData.getType());
                data.setValue(apiTimeSeriesData.getValue());
                dataList.add(data);
            }
            metric.setData(dataList);
            metrics.add(metric);
        }
    }
    return metrics;
}

note:

DateUtils code related to the realization of the need to carry out their own
can go to get information idc metric corresponding cluster by passing through this part of the code tsquery way; we just need the following code to achieve the monitoring of indicators corresponding to get the code by ServiceImpl you can
if you want to pass cm api integration with spring boot, of which we will encounter two problems:
dependence conflicts, mainly in conflict with cxf jackson's; row can be solved by way of the jar package

Regular parsing error, the problem is a pit cm during use, which is still under investigation, specific form:

And there are spaces, so in the process of compiling the error will be reported directly to the regular resolved; but we can find in api cm 6.x version of this problem has not:

So you can upgrade directly api later to resolve the problem, but the problem is that the attendant cm version is inconsistent with the online operation (online version is 5.13.2), and therefore still need to think about how to solve; but after testing We found that the use cm 6.x version of the api, for a set of relevant indicators currently online version does not affect