elk (log monitoring system construction), elastic search, kibana, logstash, filebeat construction

Architecture diagram

Demonstration effect

log input

kibana

35. elk installation

Ready to work

wget -c https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.3/elasticsearch-2.3.3.rpm
wget -c https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.3.2-1.noarch.rpm
wget https://download.elastic.co/kibana/kibana/kibana-4.5.1-1.x86_64.rpm
wget -c https://download.elastic.co/beats/filebeat/filebeat-1.2.3-x86_64.rpm

35.0 java installation

yum install java-1.8.0-openjdk -y

35.1 elasticsearch installation

yum localinstall elasticsearch-2.3.3.rpm -y
systemctl daemon-reload
systemctl enable elasticsearch
systemctl start elasticsearch
systemctl status elasticsearch
systemctl status elasticsearch -l
check es service
rpm -qc elasticsearch
	/etc/elasticsearch/elasticsearch.yml
	/etc/elasticsearch/logging.yml
	/etc/init.d/elasticsearch
	/etc/sysconfig/elasticsearch
	/usr/lib/sysctl.d/elasticsearch.conf
	/usr/lib/systemd/system/elasticsearch.service
	/usr/lib/tmpfiles.d/elasticsearch.conf
修改防火墙对外
firewall-cmd --permanent --add-port={9200/tcp,9300/tcp}
firewall-cmd --reload
firewall-cmd  --list-all

35.2 安装 kibana

yum localinstall kibana-4.5.1-1.x86_64.rpm –y
systemctl enable kibana
systemctl start kibana
systemctl status kibana
systemctl status kibana -l

检查kibana服务运行
netstat -nltp

firewall-cmd --permanent --add-port=5601/tcp
firewall-cmd --reload
firewall-cmd  --list-all
访问地址 http://192.168.206.130:5601/

35.3 安装 logstash

yum localinstall logstash-2.3.2-1.noarch.rpm –y
cd /etc/pki/tls/ && ls
创建证书
openssl req -subj '/CN=baoyou.com/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
cat /etc/logstash/conf.d/01-logstash-initial.conf

input {
  beats {
    port => 5000
    type => "logs"
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

filter {
  if [type] == "syslog-beat" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    geoip {
      source => "clientip"
    }
    syslog_pri {}
    date {
      match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch { }
  stdout { codec => rubydebug }
}

启动logstash
systemctl start logstash
/sbin/chkconfig logstash on
检查服务
 netstat -ntlp

添加防火墙对外
firewall-cmd --permanent --add-port=5000/tcp
firewall-cmd --reload
firewall-cmd --list-all



配置 es
cd /etc/elasticsearch/
mkdir es-01
mv *.yml es-01
vim elasticsearch.yml

http:
  port: 9200
network:
  host: baoyou.com
node:
  name: baoyou.com
path:
  data: /etc/elasticsearch/data/es-01


systemctl restart elasticsearch
systemctl restart logstash

3.4 filebeat 安装

yum localinstall filebeat-1.2.3-x86_64.rpm -y

cp logstash-forwarder.crt /etc/pki/tls/certs/.

cd /etc/filebeat/ && tree

vim filebeat.yml
filebeat:
  spool_size: 1024
  idle_timeout: 5s
  registry_file: .filebeat
  config_dir: /etc/filebeat/conf.d
output:
  logstash:
    hosts:
    - elk.test.com:5000
    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
    enabled: true
shipper: {}
logging: {}
runoptions: {}

mkdir conf.d && cd conf.d

vim authlogs.yml
filebeat:
  prospectors:
    - paths:
      - /var/log/secure
      encoding: plain
      fields_under_root: false
      input_type: log
      ignore_older: 24h
      document_type: syslog-beat
      scan_frequency: 10s
      harvester_buffer_size: 16384
      tail_files: false
      force_close_files: false
      backoff: 1s
      max_backoff: 1s
      backoff_factor: 2
      partial_line_waiting: 5s
      max_bytes: 10485760

vim syslogs.yml
filebeat:
  prospectors:
    - paths:
      - /var/log/messages
      encoding: plain
      fields_under_root: false
      input_type: log
      ignore_older: 24h
      document_type: syslog-beat
      scan_frequency: 10s
      harvester_buffer_size: 16384
      tail_files: false
      force_close_files: false
      backoff: 1s
      max_backoff: 1s
      backoff_factor: 2
      partial_line_waiting: 5s
      max_bytes: 10485760

service filebeat start
chkconfig filebeat on
netstat -aulpt

访问地址 http://192.168.206.130:5601/

备注：参看文章 elk 日志监控系统

http://467754239.blog.51cto.com/4878013/1700828/

https://my.oschina.net/itblog/blog/547250

https://www.ibm.com/developerworks/cn/opensource/os-cn-elk/

http://www.cnblogs.com/hanyifeng/p/5509985.html （我用该文章搭建成功了）

http://blog.csdn.net/dabokele/article/details/51765136

https://cloud.tencent.com/community/article/562397

捐助开发者

在兴趣的驱动下,写一个免费的东西，有欣喜，也还有汗水，希望你喜欢我的作品，同时也能支持一下。当然，有钱捧个钱场（支持支付宝和微信以及扣扣群），没钱捧个人场，谢谢各位。

个人主页：http://knight-black-bob.iteye.com/

谢谢您的赞助，我会做的更好！

elk (log monitoring system construction), elastic search, kibana, logstash, filebeat construction

捐助开发者

Guess you like