A simple custom monitoring system implementation

Index of this article:

Requirement analysis of alarm system
Alarm system main script
Alarm system configuration file
Alarm system monitoring project

Requirement analysis of alarm system

Requirements: Use shell to customize various personalized alarm tools, but need unified management and standardized management.

Idea: Specify a script package, including main program, subprogram, configuration file, mail engine, output log, etc.

main program

The main program is the entrance of the entire script and the lifeblood of the entire system.
configuration file

It is equivalent to a control center, which controls the execution of each subprogram, specifies each associated log file, and realizes different functions.
subroutine

The real monitoring script that implements various functions is used to monitor various indicators, and is selectively switched by the configuration file.
mail engine

Here is a Python program that can define the server, sender, and sender password for sending emails, and realize the function of sending email alerts to users.
output log

Both execution and monitoring processes need to be recorded, and log output is required to facilitate subsequent troubleshooting.

A monitoring system has the same framework on different machines, the difference is that the configuration files need to be customized according to different roles.

A simple monitoring system architecture:

                    主目录mon
                        |
        ---------------- ----------------
        |       |       |       |       |
       bin     conf   shares   mail    log
        |       |       |       |       |
     main.sh mon.conf load.sh mail.py mon.log
                      502.sh  mail.sh err.log

说明：
bin目录下是主程序
conf目录下是配置文件
shares目录下是各个监控脚本
mail目录下是邮件引擎
log目录下是日志

Alarm system main script

It is best to place custom scripts in /usr/local/sbin to facilitate subsequent search

Create a frame

[root@test2 ~]# cd /usr/local/sbin/
[root@test2 sbin]# mkdir mon
[root@test2 sbin]# cd mon
[root@test2 mon]# mkdir bin conf shares log mail
[root@test2 mon]# ls -l
总用量 0
drwxr-xr-x 2 root root 6 2月   9 18:56 bin
drwxr-xr-x 2 root root 6 2月   9 18:56 conf
drwxr-xr-x 2 root root 6 2月   9 18:56 log
drwxr-xr-x 2 root root 6 2月   9 18:56 mail
drwxr-xr-x 2 root root 6 2月   9 18:56 shares

main scripting

[root@test2 mon]# cd bin/
[root@test2 bin]# vi main.sh
#!/bin/bash
# 是否发送邮件的开关
# export使变量在子shell内有效
export send=1

# 获取ens33的ip地址
export addr=`/sbin/ifconfig |grep -A1 "ens33: "|awk '/inet/ {print $2}'`

dir=`pwd`
# 获取最后一层目录名称
last_dir=`echo $dir | awk -F '/' '{print $NF}'`

# 确保在bin目录中，防止无法执行配置文件
if [ $last_dir == "bin" ] || [ $last_dir == "bin/" ]
then
    conf_file="../conf/mon.conf"
else
    echo "you shoud cd bin dir"
    exit
fi
exec 1>>../log/mon.log 2>>../log/err.log    #日志输出

echo "`date +"%F %T"` load average"
/bin/bash ../shares/load.sh         # 执行监控脚本load.sh

#判断配置文件中是否需要监控502
if grep -q 'to_mon_502=1' $conf_file
then
    export log=`grep 'logfile=' $conf_file |awk -F '=' '{print $2}' | sed 's/ //g'`
    /bin/bash ../shares/502.sh
fi

Alarm system configuration file

Configuration file: define monitoring related parameters

# 配置文件其实就是一个写了关于系统内各个应用监控内容的参数
# 在主脚本内通过提取相关参数来进行判断，实现对不同主机的自定义项目监控
# 例如本例中定义了关于数据库，httpd，php等不同进程的监控，并根据配置来判断是否进行监控（这里只有出现502状态码时需要进行监控）

[root@test2 bin]# vim ../conf/mon.conf
## 定义监控选项
## 定义mysql的服务器地址、端口以及user、password
## 0 or 1, default 0,0 not monitor, 1 monitor
to_mon_cdb=0    
db_ip=10.20.3.13    //数据库ip
db_port=3315        //数据库端口
db_user=username    //数据库用户名
db_pass=passwd      //数据库密码

## httpd    如果是1则监控，为0不监控
to_mon_httpd=0

## php 如果是1则监控，为0不监控
to_mon_php_socket=0

## http_code_502    需要定义访问日志的路径
to_mon_502=1
logfile=/data/log/xxx.xxx.com/access.log

## request_count    定义日志路径以及域名
to_mon_request_count=0
req_log=/data/log/www.discuz.net/access.log
domainname=www.discuz.net

Alarm system monitoring project

The so-called alarm system monitoring project is a detailed monitoring sub-script, which can be customized to achieve different functions.

load script

#! /bin/bash

# 切换load负载信息
load=`uptime |awk -F 'average:' '{print $2}'|cut -d',' -f1|sed 's/ //g' |cut -d. -f1`

# 判断在负载大于10，并且开启发送邮件开关时，才执行发邮件功能
if [ $load -gt 10 ] && [ $send -eq "1" ]
then
    echo "$addr `date +%T` load is $load" >../log/load.tmp
    /bin/bash ../mail/mail.sh [email protected] "$addr\_load:$load" `cat ../log/load.tmp`
fi
echo "`date +%T` load is $load"

502 Status Code Script

#! /bin/bash
d=`date -d "-1 min" +%H:%M`
# 查看1分钟前的日志，对状态码进行过滤
c_502=`grep :$d:  $log  | grep ' 502 '| wc -l`
if [ $c_502 -gt 10 ] && [ $send == 1 ]; then
    echo "$addr $d 502 count is $c_502">../log/502.tmp
    /bin/bash ../mail/mail.sh $addr\_502 $c_502 ../log/502.tmp
fi
echo "`date +%T` 502 $c_502"

memory monitoring script

#! /bin/bash
rm -f ../log/disk.tmp   //删除之前的暂存文件

# [ %]+：1或多个空格或百分号作为分隔符（同时指定多个字符作为分隔符）
for r in `df -h |awk -F '[ %]+' '{print $5}'|grep -v Use`
do
    // 进行判断：使用率大于90，发送开关开启
    if [ $r -gt 90 ] && [ $send -eq "1" ]
    then
        echo "$addr `date +%T` disk useage is $r" >>../log/disk.tmp
fi

if [ -f ../log/disk.tmp ]   // 判断暂存文件是否存在
then
    df -h >> ../log/disk.tmp
    /bin/bash ../mail/mail.sh $addr\_disk $r ../log/disk.tmp
    echo "`date +%T` disk useage is nook"
else
    echo "`date +%T` disk useage is ok"
fi