Python script to implement open-falcon custom monitoring server time
Some time ago, the time of the two servers of the company was suddenly inaccurate. Because the server time was not monitored, it was found that the server time was not synchronized when the feedback was implemented. The server is sure to synchronize the time, but I don't know why it is suddenly out of sync. To prevent similar things from happening again, I plan to add the system time to the open-falcon monitoring system.
openfalcon will not automatically monitor the server time and report it, so you need to write your own script and push it up.
Because the main monitoring is the accuracy of the server time, when the alarm is made, the alarm is determined by checking whether the difference between the server and network time exceeds the setting, so you can change the uploaded metric to the current server time and The difference in network time, not the current server time.
1. Scripting
The specific script is as follows:
#! /usr/bin/python3
import time
import datetime
import os
import http.client
import requests
import socket
import json
#获取网络服务器的时间
def get_werbservertime(host):
conn = http.client.HTTPConnection(host)
conn.request("GET","/")
r = conn.getresponse()
ts = r.getheader('date')#获取http头date部分
#将GMT时间转换成北京时间
ltime = time.strptime(ts[5:25], "%d %b %Y %H:%M:%S")
ttime = time.localtime(time.mktime(ltime)+8*60*60)
dat = "%u-%02u-%02u"%(ttime.tm_year,ttime.tm_mon,ttime.tm_mday)
tm = "%02u:%02u:%02u"%(ttime.tm_hour,ttime.tm_min,ttime.tm_sec)
print(dat,tm)
timeAll = dat + ' ' + tm
#将时间转换为时间戳
timeArray = time.strptime(timeAll,"%Y-%m-%d %H:%M:%S")
timestamp = time.mktime(timeArray)
return (timestamp)
#将数据上传至agent
def pushtime(hostname,timed):
ts = int(time.time())
payload = [
{
"endpoint": hostname,
"metric": "timed",
"timestamp": ts,
"step": 60,
"value": timed,
"counterType": "GAUGE",
"tags": "",
}
]
r = requests.post("http://127.0.0.1:1988/v1/push",data = json.dumps(payload))
print(r.text)
#设定为每60s上传一次
while True:
webts = int(get_werbservertime('10.200.9.111'))
print (webts)
#获取服务器时间戳
localts = int(time.time())
#打印服务器当前时间
print(time.strftime("%Y-%m-%d %H:%M:%S" ,time.localtime(time.time())))
print (localts)
#将服务器时间和时间服务器时间相减，得到的差值是秒数
timed = abs(localts - webts)
print (timed)
hostname = socket.gethostname()
pushtime(hostname,timed)
time.sleep(60)
2. Execute script test
python3 open-falcon-timed.py
After executing the command, check the printed information, and see success indicating that the execution was successful (because the virtual machine I tested here has not been time synchronized, so the time is quite different from the time server).
After a while, you can see the data reported on the openfalcon monitoring page.
Click in and you can see the line chart.
Because the uploaded timestamp is the server timestamp, once the server time is inaccurate, the time in the chart will also be inaccurate.
3. Generate executable files
Use pyinstaller to package the script to generate an executable file.
pyinstaller -F open-falcon-timed.py
It is recommended to install pyinstaller to generate executable files using a lower version system, which can be applied to any system.
The generated executable file is in the dist path.
Use the command ./open-falcon-timed to execute the file
4. Configure alarm
Configure the policy on Templates, the metric is timed, and the alarm will be triggered when the value of timed is greater than or equal to 5. And configure the alarm receiving group.
Create a hostgroup in the HostGroups menu, add the IP/hostname of the server to be monitored in the hosts, and bind the previously configured strategy in the templates.
You can see the alarm in Alarm-Dashboard after a while
5 Use plugin to execute plugins regularly
If there are many servers that need to be monitored, each server must be deployed with executable files, which is time-consuming to deploy one by one. Open-falcon provides the plugin function to automatically issue plugins and execute them regularly. Specific steps are as follows:
- Modify script
Plugin provides the function of executing plugins regularly, so there is no need to loop in the script to execute the method in the script every 60s.
You can change the following part of the script
to
execute once and then end
- Regenerate the executable file
pyinstaller -F open-falcon-timed.py
-
Rename the plugin and upload it to the designated code repository
. The plugin function of open-falcon will automatically execute the plugin under the specified path starting with a number and an underscore, where the number represents the interval in seconds, so I changed the file name to 60_open-falcon- timed, which means that the plugin will be executed every 60 seconds. timed.ini is the configuration file, here I have done configuration separation.
-
Configure the address of the code repository in the configuration file of the plugin module
in the agent of the server. Modify the following content in the configuration file cfg.json of the agent to the address and account secret of the code repository: the
format is: http://account:password@address
Modify the storage location of the plug-in pulled from the warehouse in the agent configuration file cfg.json.
Restart the agent after the modification is completed
./open-falcon restart agent
- Use the command line to test whether the files in the code repository can be obtained.
Note: Ensure that the git command has been installed. If it is not installed, use the command yum install -y git to download. After the download is complete, use the following command to download the plugin on the code repository:
curl http://127.0.0.1:1988/plugin/update
Displaying success proves that the pull is successful, and the plug-in can be seen in the path configured in the configuration file.
- Add executable permissions
chmod +x 60_open-falcon-timed
time.ini is the configuration file after configuration separation. If you need the open-falcon plugin to automatically execute the plug-in, you need to put the configuration file in the same directory as the executable file open-falcon that starts open-falcon.
7. Bind the plugin to
open the open-falcon dashboard, create a new HostGroup in the HostGroups menu bar, add an endpoint in the hosts, and add the path of the executable plugin in the plugtins. This path is a relative path, relative to the dir in the agent configuration file route of. The plugin will automatically execute the plugins starting with a number and an underscore under the specified path.
Check the agent log, you can see that the time plug-in is executed every 60s:
