1. The relationship between ELK and Zabbix?
ELK is a set of combined log collection system, composed of Elasticsearch, Logstash and Kibana three software, through ELK can collect system logs, website logs, application system logs and other log data, and can also filter and filter logs. Then it is stored centrally and can be used for real-time retrieval and analysis. This is the basic function of ELK.
But if we hope that when collecting logs, we can extract abnormal log information (errors, failures), etc., and send them to the operation and maintenance personnel in time, then the operation and maintenance can troubleshoot and handle the failures in the first time, avoiding many failures happened.
Then you can use zaibbix to actually read the contents of the log in real time (logstash/filebeat), and it can also filter the log information. Through ELK reading, filtering and other functions, you can filter out some abnormal keywords (error, failed, OutOff, Warning) in the log, and then send the error log to the zabbix server through the zabbix plugin of logstash, then After zabbix receives this data, it can perform alarm processing according to the alarm mode defined by itself.
Second, the use of
Logstash and zabbix plug-ins logstash supports multiple output media, such as TCP, HTTP, elasticsearch, kafka, Redis, Nagios, zabbix, syslog, etc., then we want to output some error messages in the collected logs and give them an alarm , The logstash-output-zabbix plug-in is used. The plug-in can integrate Logstash and zabbix. In fact, it filters the data collected by logstash, and outputs the log with the error flag to zabbix, and finally the alarm through zabbix The mechanism is triggered and alarmed.
logstash-output-zabbix is a community-maintained plug-in. It is not installed in Logstash by default, but it is easy to install. Just run the following command in logstash:
/usr/local/logstash/bin/logstash-plugin install logstash-output-zabbix
2.1. List currently installed plugins
将列出所有已安装的插件
/usr/local/logstash/bin/logstash-plugin list
#将列出已安装的插件及版本信息
/usr/local/logstash/bin/logstash-plugin list --verbose
#将列出包含namefragment的所有已安装插件
/usr/local/logstash/bin/logstash-plugin list "zabbix"
#将列出特定组的所有已安装插件（ input，filter，codec，output）
/usr/local/logstash/bin/logstash-plugin list --group input
2.2. Install plug-in
To install a plug-in, for example, install the kafka plug-in, you can execute the following command:
/usr/local/logstash/bin/logstash-plugin install logstash-output-kafka
PS: The premise is that this plug-in installation method can be accessed online, and the plug-in hosted on the public repository (RubyGems.org) will be retrieved, then downloaded to the local machine and automatically installed on the Logstash installation.
2.3. Update plug-ins
Each plug-in has its own release cycle and version updates, and these updates are usually independent of the release cycle of Logstash. Therefore, sometimes you need to update the plugin separately, you can use the update subcommand to get the latest version of the plugin.
将更新所有已安装的插件
/usr/local/logstash/bin/logstash-plugin update
将仅更新指定的插件
/usr/local/logstash/bin/logstash-plugin update logstash-output-kafka
2.4. Delete the plugin
If you need to delete the plugin from the Logstash plugin, you can execute the following command:
/usr/local/logstash/bin/logstash-plugin remove logstash-output-kafka
This deletes the logstash-output-kafka plugin.
3. Use of
logstash-output-zabbix plug-in After logstash-output-zabbix is installed, it can be used in the logstash configuration file.
The following is an example of the use of logstash-output-zabbix:
zabbix {
zabbix_host => "[@metadata][zabbix_host]"
zabbix_key => "[@metadata][zabbix_key]"
zabbix_server_host => "x.x.x.x"
zabbix_server_port => "xxxx"
zabbix_value => "xxxx"
}
Among them:
zabbix_host: indicates the name of the Zabbix host name field. It can be a separate field or a subfield of the @metadata field. It is a required setting and has no default value.
zabbix_key: Represents the value of the Zabbix item key, that is, the item in zabbix. This field can be a separate field or a subfield of the @metadata field. There is no default value.
zabbix_server_host: indicates the IP or resolvable host name of the Zabbix server. The default value is "localhost", which needs to be set to the address where the zabbix server is located.
zabbix_server_port: indicates the listening port opened by the Zabbix server, the default value is 10051.
zabbix_value: indicates the field name corresponding to the value of the monitoring item to be sent to zabbix item, the default value is "message", that is, the content of the "message" field is sent to the zabbix item monitoring item defined by zabbix_key above, of course, you can also specify a specific The content of the field is sent to the zabbix item monitoring item.
4. Integrate logstash with zabbix
Here we collect logs with logstash, then read the logs, and finally select keywords to filter and call the zabbix alarm process to see how to configure logstash to implement zabbix alarms.
Requirement:
by monitoring the system log file, and then filtering some keywords in the log information, such as ERR, error, ERROR, Failed, WARNING, etc., filtering out the information in the log, then sending it to zabbix, and finally using zabbix The alarm function realizes the alarm for the above keywords in the system log.
For filtering keywords and alarming, different business systems may have different keywords. For example, for http systems, you may need to filter error codes such as 500, 403, and 503. For java-related systems, you may need to filter OutOfMemoryError and PermGen. , Java heap and other keywords. In the log output of some business systems, there may be some custom error messages, so these also need to be used as filtering keywords.
4.1. To configure the logstash event configuration file,
you can view the /usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns file to determine which matches file.
The next step is to create a logstash event configuration file. The configuration file is divided into three parts to introduce. The first is the input part, which is as follows:
vim /usr/local/logstash/config/zabbix_output.conf
input {
file {
path => ["/var/log/secure"]
type => "system"
start_position => "beginning"
}
}
#input部分是从/var/log/secure文件中读取数据，start_position 表示从secure文件开头读取内容。
Followed by the filter part, the content is as follows:
filter {
grok {
match => {
"message" => "%{SYSLOGTIMESTAMP:message_timestamp} %{SYSLOGHOST:hostname} %{DATA:message_program}(?:\[%{POSINT:message_pid}\])?: %{GREEDYDATA:message_content}" } #这里通过grok对message字段的数据进行字段划分，这里将message字段划分了5个子字段。其中，message_content字段会在output中用到。
}
mutate {
add_field => [ "[zabbix_key]", "oslogs" ] #新增的字段，字段名(key)是zabbix_key，值为oslogs。
add_field => [ "[zabbix_host]", "%{host}" ] #新增的字段，字段名(key)是zabbix_host，值可以在这里直接定义，也可以引用字段变量来获取。这里的%{host}获取的就是日志数据的主机名，这个主机名与zabbix web中“主机名称”需要保持一致。
}
mutate {
#这里是删除不需要的字段
remove_field => "@version"
remove_field => "message"
}
date {
#这里是对日志输出中的日期字段进行转换，其中message_timestamp字段是默认输出的时间日期字段，将这个字段的值传给 @timestamp字段。
match => [ "message_timestamp","MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"]
}
}
The filter part is a key point. In this part, the focus is on the message_timestamp field and the message_content field.
Finally, there is the output part, which is as follows:
output {
if [message_content] =~ /(ERR|error|ERROR|Failed)/ {
#定义在message_content字段中，需要过滤的关键字信息，也就是在message_content字段中出现给出的这些关键字，那么就将这些信息发送给zabbix。
zabbix {
zabbix_host => "[zabbix_host]" #这个zabbix_host将获取上面filter部分定义的字段变量%{host}的值
zabbix_key => "[zabbix_key]" #这个zabbix_key将获取上面filter部分中给出的值
zabbix_server_host => "192.168.2.3" #这是指定zabbix server的IP地址
zabbix_server_port => "10051" #这是指定zabbix server的监听端口
zabbix_value => "message_content" #这个很重要，指定要传给zabbix监控项item（oslogs）的值， zabbix_value默认的值是"message"字段，因为上面我们已经删除了"message"字段，因此，这里需要重新指定，根据上面filter部分对"message"字段的内容划分，这里指定为"message_content"字段，其实，"message_content"字段输出的就是服务器上具体的日志内容。
}
}
stdout {
codec => rubydebug } #这里是开启调试模式，当第一次配置的时候，建议开启，这样过滤后的日志信息直接输出的屏幕，方便进行调试，调试成功后，即可关闭。
}
Combine the above three parts into a file zabbix_output.conf, and then start the logstash service:
nohup /usr/local/logstash/bin/logstash -f /usr/local/logstash/config/zabbix_output.conf &
4.2, zabbix platform configuration log alarm
Log in to the zabbix web platform, select configuration —> template —> create template, set the name as logstash-output-zabbix, as shown in the figure below:
Create application set
click application set —> create application set
create monitoring Item (itme)
click monitoring item ---> create monitoring item
So far, zabbix monitoring logstash log data configuration is complete.
Let's simulate a failure, log in to the 192.168.2.15 host through SSH on any host, and then enter an incorrect password to make the system's /var/log/secure file generate an error log, and then see if logstash can be filtered and sent to zabbix.
ssh [email protected]
cat /var/log/secure
#内容
Apr 23 11:24:12 wzb sshd[49569]: Failed password for root from 192.168.2.3 port 42978 ssh2
There is the keyword Failed that we want to filter, so logstash will filter this content out and send it to zabbix.
Then, log zabbix web platform, click Monitoring -> the latest data, if zabbix can receive log, you can see the latest data in the following figure:
Click the History, you can view the details, as shown below:
see , The content in the red box is the content of the message_content field defined in logstash.
Zaibbix's itme type and application scenarios: zabbix
agent
Zabbix's own client program (passive mode), zabbix server actively collects monitoring data from it. The agent provides a wealth of keys, including but not limited to cpu, memory, network, disk, web, etc. If you don't mind or the system supports installing this program, then he is the first choice. It should be noted that the server has a timeout limit for retrieving data. The maximum timeout period is 30 seconds. If the retrieval data often exceeds 30 seconds, then it is not recommended that you use the agent in active mode. You can use the following types of agent active
zabbix agent (active)
also needs to install agent (active mode), which is the same as the previous one. But the data is actively submitted to zabbix server by zabbix agent
Simple check
basic detection, can detect network, port, fping, etc., with few functions and no need to install client.
The functions of snmp check
snmp v1 check, snmp v2 check and snmp v3 check are the same. The following scenarios are recommended:
Based on security considerations, the customer does not agree to install agent
routers, printers and other equipment that cannot be installed, but supports the snmp protocol.
Does not like frequent agent upgrades
zabbix internal
zabbix system internal use, such as the number of trend data records, the number of historical records, etc., daily business Monitoring is not useful for him.
Zabbix trapper
also needs to install agent (active mode), you need to use bin/zabbix_sender to submit data to zabbix server. Suitable for use in the following situations:
Retrieving data takes a long
time. There are a lot of data to be submitted at the same time, such as redis info information, which contains fifty or sixty items of data. Submitting through zabbix_sender at one time is obviously more convenient than fetching dozens of times by agent.
Zabbix aggretage
aggregate checks is an aggregate check. For example, I want to know the average host load of a group, the total amount of remaining hard disks, or the data of a few machines. Simply put, this method is used to understand an overall level , Without the need for us to watch the past one by one. All the data of this method comes from the database, so it does not require an agent.
The external check
zabbix server runs scripts or binary files to perform external inspections. External inspections do not need to run any agentd on the monitored side.
Note: Please do not overuse external detection, which will seriously reduce the performance of zabbix system
database monitor
zabbix obtains database data and database status by calling ODBC
IPMI agent is
used to monitor hardware devices, such as Dell or HP server motherboard temperature, cpu voltage, whether the lid is opened, etc.
SSH agent
zabbix uses the provided ssh information (server user password or certificate) to log in to the server, and execute the specified script to retrieve data. Suitable for the following people
Don't install agent, don't want to collide, agent
don't worry about account password/certificate in zabbix
Telnet agent is the
same as above, Windows does not support ssh, you can use Telnet agent.
The jmx agent
monitors java jvm, such as tomcat, through jmx. There is a big shortcoming at present. If there are multiple instances of tomcat, the jmx agent can only monitor one. For multiple instances, it is recommended to use agent + cmdline-jmxclient-0.10.3.jar
The zabbix Calculated
calculation type is calculated between several key values. For example, the info command that comes with redis can monitor the two values of keyspace_hits and keyspace_misses, but the hit rate of redis cannot be obtained directly. You can monitor redis through the calculated of zabbix Hit rate
Debug grok: http://grokdebug.herokuapp.com/
Help document: http://www.ttlsa.com/zabbix/how-to-chose-zabbix-item-type/