Use cases elk gather network equipment

Brief introduction

With the increase in server and network equipment in the room, log management, and query the system administrator has become a headache.

Common problems encountered by the system administrator as follows:

Routine maintenance is not possible to log in to each server and equipment up to view the log;
Limited storage space on the network equipment, the date can not be stored too long log, and system problems there may be some operations due to take place a long time ago;
In some trespass case, the intruder usually clear the local log, clear signs of intrusion;
zabbix and other monitoring systems can not replace log management, such as project monitoring system can not log on, the scheduled task execution.

For these reasons, Rsyslog build a log server for centralized management of logs in the current network environment, it is very necessary.

Advantage Rsyslog services are as follows:

Rsyslog server can support most network devices, device options in the system network device configuration options are mostly remote log service. Simply fill in the IP address and port (most devices have a default is 514), and then determine on it;
Linux servers only need to add in the local Rsyslog service configuration in a single line can send logs to the log server, configure and deploy it very simple;
deployment architecture

Deployment Architecture

Use cases elk gather network equipment

Rsyslog Configuration

系统环境及软件版本：
CentOS Linux release 7.5.1804 (Core)
Elasticserch-6.8.4
Kibana-6.8.4
Logstash-6.8.4
Filebeat-6.8.4
Rsyslog-8.24.0

The SELINUX set to disabled

# setenforce 0
# sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

Firewall Configuration

firewall-cmd  --add-service=syslog --permanent
firewall-cmd  --reload

Check whether the software is installed rsyslog

# CentOS7 installed by default rsyslog

[root@ZABBIX-Server ~]# rpm -qa |grep rsyslog
rsyslog-8.24.0-16.el7.x86_64

Rsyslog configuration file editing

vim /etc/rsyslog.conf # changes were as follows

[root@ZABBIX-Server mnt]# egrep -v "*#|^$" /etc/rsyslog.conf
$ModLoad imudp
$UDPServerRun 514
$ModLoad imtcp
$InputTCPServerRun 514
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
*.info;mail.none;authpriv.none;cron.none;local6.none;local5.none;local4.none                /var/log/messages
$template h3c,"/mnt/h3c/%FROMHOST-IP%.log"
local6.* ?h3c
$template huawei,"/mnt/huawei/%FROMHOST-IP%.log"
local5.* ?huawei
$template cisco,"/mnt/cisco/%FROMHOST-IP%.log"
local4.* ?cisco

Use cases elk gather network equipment

$ModLoad imudp            # immark是模块名，支持tcp协议  
$ModLoad imudp             # imupd是模块名，支持udp协议    
$InputTCPServerRun 514
$UDPServerRun 514          #允许514端口接收使用UDP和TCP协议转发过来的日志

note:

*.info;mail.none;authpriv.none;cron.none;local6.none;local5.none;local4.none         /var/log/messages
默认没有添加local6.none;local5.none;local4.none 命令，网络日志在写入对应的文件的同时会写入/var/log/messages 中

Check the rsyslog service

Use cases elk gather network equipment

Restart rsyslog service

systemctl restart rsyslog.service

Log store directory

Use cases elk gather network equipment

Point to the network device log syslog server, attention to the different devices from different manufacturers corresponding local, correspondence is as follows:

/mnt/huawei   --- local6
/mnt/h3c      --- local5
/mnt/cisco     --- local4

Network Device Configuration

Huawei：
info-center loghost source Vlanif99
info-center loghost 192.168.99.50 facility local5

H3C:
info-center loghost source Vlan-interface99
info-center loghost 192.168.99.50 facility local6

CISCO:
(config)#logging on
(config)#logging 192.168.99.50 
(config)#logging facility local4
(config)#logging source-interface e0

Ruijie：
logging buffered warnings
logging source interface VLAN 99
logging facility local6
logging server 192.168.99.50

Note: 192.168.99.50 to rsyslog server IP

Edit the configuration file filebeat

Log files collected under rsyslog to logstash

[root@ZABBIX-Server mnt]# egrep -v "^#|^$" /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /mnt/huawei/*
  tags: ["huawei"]
  include_lines: ['Failed','failed','error','ERROR','\bDOWN\b','\bdown\b','\bUP\b','\bup\b']
  drop_fields:
    fields: ["beat","input_type","source","offset","prospector"]
- type: log
  paths:
    - /mnt/h3c/*
  tags: ["h3c"]
  include_lines: ['Failed','failed','error','ERROR','\bDOWN\b','\bdown\b','\bUP\b','\bup\b']
  drop_fields:
    fields: ["beat","input_type","source","offset","prospector"]
setup.template.settings:
  index.number_of_shards: 3
output.logstash:
  hosts: ["192.168.99.185:5044"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

Edit the configuration file logstash

Filebeat log of transmitted according to different tags each treatment, the log data storage process is completed on the transmitted ES, and further on the visual display kibana

[root@elk-node1 ~]# egrep -v "^#|^$" /etc/logstash/conf.d/networklog.conf
input {
  beats {
    port => 5044
  }

}
filter {
  if "huawei" in [tags] {
    grok{
      match => {"message" => "%{SYSLOGTIMESTAMP:time} %{DATA:hostname} %{GREEDYDATA:info}"}
        }
  }
   else if "h3c" in [tags] {
    grok{
      match => {"message" => "%{SYSLOGTIMESTAMP:time} %{YEAR:year} %{DATA:hostname} %{GREEDYDATA:info}"}
        }
  }
mutate {
      remove_field => ["message","time","year","offset","tags","path","host","@version","[log]","[prospector]","[beat]","[input][type]","[source]"]
    }
}
output{
stdout {codec => rubydebug}
elasticsearch {
    index => "networklogs-%{+YYYY.MM.dd}"
    hosts => ["192.168.99.185:9200"]
    sniffing => false
    }
}