Because Hadoop cluster recently upgraded line from mrv1 to mrv2, and monitoring changes in the template followed. Line is about 200 clusters, the link modules way to add that a large number of modules next link template, and then add the host to the template.
Such a machine doing almost 215 item.
In order to increase the monitoring of NM, also used link way to connect template, has been found to return a blank page when the link on the page.
For fast on-line, the changed method of using host.update api, host link directly to the template NM.
Looking back to this question:
In adopting the template page link, in fact, it is called zabbix template related api (specifically called template.update method)
Directly to the test by calling api script:
Test script:
#!/usr/bin/env python
import urllib2
import sys
import json
def requestJason(url,values): data = json.dumps(values)
print data
req = urllib2.Request(url, data, {'Content-Type': 'application/json-rpc'})
response = urllib2.urlopen(req, data)
data_get = response.read()
output = json.loads(data_get)
print output
try:
message = output['result']
except:
message = output['error']['data']
quit()
print json.dumps(message)
return output
def authenticate(url, username, password):
values = {'jsonrpc': '2.0',
'method': 'user.login',
'params': {
'user': username,
'password': password
},
'id': '0'
}
idvalue = requestJason(url,values)
return idvalue['result']
def getTemplate(hostname,url,auth):
values = {'jsonrpc': '2.0',
'method': 'template.get',
'params': {
'output': "extend",
'filter': {
'host': hostname
}
},
'auth': auth,
'id': '2'
}
output = requestJason(url,values)
print output['result'][0]['hostid']
return output['result'][0]['hostid']
def changeTemplate(idx,id_list,url,auth):
values = {'jsonrpc': '2.0',
'method': 'template.update',
'params': {
"templateid":idx,
"templates":id_list
},
'auth': auth,
'id': '2'
}
output = requestJason(url,values)
print output
def main():
id_list = []
hostname = "Vipshop_Template_OS_Linux_Hadoop_Datanode_Pro"
url = 'xxxx'
username = 'admin'
password = 'xxxx'
auth = authenticate(url, username, password)
idx = getTemplate(hostname,url,auth)
temlist = ['Vipshop_Template_LB_Tengine_8090','Vipshop_Template_Redis_6379','Vipshop_Template_Redis_6380','Vipshop_Template_Redis_6381','Vipshop_Template_Redis_6382','Vipshop_Template_Redis_6383']
for tem in temlist:
idtemp = getTemplate(tem,url,auth)
id_list.append({"templateid":idtemp})
print id_list
#id_list = [{"templateid":'10843'},{"templateid":"10554"},{"templateid":"10467"},{"templateid":"10560"},{"templateid":"10566"},{"templateid":"10105"}]
changeTemplate(idx,id_list,url,auth)
if __name__ == '__main__':
main()
Script results:
urllib2.HTTPError: HTTP Error 500: Internal Server Error
Because api actually sent a request to post jason format manually using curl to verify:
curl -vvv -i -X POST -H 'Content-Type:application/json' -d
'{"params": {"templates": [{"templateid": "10117"}, {"templateid": "10132"}, {"templateid": "10133"}, {"templateid": "10134"},
{"templateid": "10135"}, {"templateid": "10136"}], "templateid": "10464"}, "jsonrpc": "2.0", "method": "template.update", "auth": "421a04b400e859834357b5681a586a5f", "id": "2"}'
http://zabbix.idc.vipshop.com/api_jsonrpc.php
Return 500 error (ie errors encountered when php backend processing), adjust the php configuration, log into the debug format:
php-fpm.conf:
log_level = debug
We found the following errors in the error log in:
[04-May-2014 14:04:32.115189] WARNING: pid 6270, fpm_request_check_timed_out(), line 271: [pool www] child 6294, script '/apps/svr/zabbix/wwwroot/api_jsonrpc.php' (request: "POST /api_jsonrpc.php") executing too slow (1.269946 sec), logging
[04-May-2014 14:04:32.115327] DEBUG: pid 6270, fpm_got_signal(), line 72: received SIGCHLD
[04-May-2014 14:04:32.115371] NOTICE: pid 6270, fpm_children_bury(), line 227: child 6294 stopped for tracing
[04-May-2014 14:04:32.115385] NOTICE: pid 6270, fpm_php_trace(), line 142: about to trace 6294
[04-May-2014 14:04:32.115835] NOTICE: pid 6270, fpm_php_trace(), line 170: finished trace of 6294
[04-May-2014 14:04:32.115874] DEBUG: pid 6270, fpm_event_loop(), line 409: event module triggered 1 events
[04May-2014 14:04:35.318614] WARNING: pid 6270, fpm_stdio_child_said(), line 166: [pool www] child 6294 said into stderr: "NOTICE: sapi_cgi_log_message(), line 663: PHP message: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 512 bytes) in /apps/svr/zabbix/wwwroot/api/classes/CItem.php on line 1088"
[04-May-2014 14:04:35.318665] DEBUG: pid 6270, fpm_event_loop(), line 409: event module triggered 1 events
When that is done link template, we need to put php related data in memory, and the default setting is 128M, and more if the item's host when easily exceed this limit.
change to:
memory_limit = 1280M
Retest returned 502 Bad Gateway error, which led to the back-end execution timeout.
error log:
[04-May-2014 14:50:21.318071] WARNING: pid 4131, fpm_request_check_timed_out(), line 281: [pool www] child 4147, script '/apps/svr/zabbix/wwwroot/api_jsonrpc.php' (request: "POST /api_jsonrpc.php") execution timed out (10.030883 sec), terminating
Execution time exceeds request_terminate_timeout settings. 502 leads generated.
Change request_terminate_timeout = 1800 (default is 10s), max_execution_time = 0 (default 30s), re-test. ok.
summary
Unlike zabbix online application, when you call api to do updates, is a batch of behavior, there are certain requirements for memory and execution time.
Therefore, to set reasonable parameters of php, when debug log level down and turn slow log to easily locate the problem.