ELK

环境:
Vagrant 1.8.1
CentOS 7.2 192.168.0.228
Elasticsearch 2.3.2
logstash 2.2.4
Kibana 4.4.2
filebeat 1.2.2
topbeat 1.2.2

ELK安装
1. CentOS7系统配置

在开始安装elk之前，我们需要对CentOS7做一系列配置。CentOS7安装后默认没有安装ifconfig、iptables等命令。

1. 1. 安装iptables

$ systemctl stop firewalld

$ systemctl mask firewalld

$ yum install iptables-services

$ systemctl enable iptables

$ systemctl [stop|start|restart] iptables

$ service iptables save

1. 1. 安装ifconfig

$ ip addr

$ ip link

$ ip -s link

$ yum provides ifconfig

$ yum whatprovides ifconfig

$ yum install net-tools

$ ifconfig -a

1. 1. 禁用IPV6

方法一：

$ vi /etc/sysctl.conf

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.eth1.disable_ipv6 = 1

$ sysctl -p

方法二：

$ vi /etc/sysctl.d/disableipv6.conf

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.eth1.disable_ipv6 = 1

$ reboot

1. 安装Java并配置环境变量

$ cd ~

$ wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u73-b02/jdk-8u73-linux-x64.rpm"

$ sudo yum -y localinstall jdk-8u73-linux-x64.rpm

$ sudo vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_73

export CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$JAVA_HOME/bin:$PATH

$ source /etc/profile

1. 安装Elasticsearch

https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

1. 1. 导入elasticsearch公钥

$ sudo rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch

1. 1. 创建elasticsearch.repo

$ echo '[elasticsearch-2.x]

name=Elasticsearch repository for 2.x packages

baseurl=http://packages.elastic.co/elasticsearch/2.x/centos

gpgcheck=1

gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch

enabled=1

' | sudo tee /etc/yum.repos.d/elasticsearch.repo

1. 1. 使用yum install安装

$ sudo yum -y install elasticsearch

1. 1. 修改elasticsearch配置（修改主机ip）

$ sudo vim /etc/elasticsearch/elasticsearch.yml

network.host: 192.168.0.228

1. 1. 启动elasticsearch

$ sudo systemctl start elasticsearch

1. 1. 将elasticsearch添加到开机自启动

$ sudo systemctl enable elasticsearch

1. 1. 访问elasticsearch rest服务

使用http://192.168.0.228:9200/出现如下内容表示elasticsearch安装成功。

注：

Elasticsearch默认http端口为9200，节点端口为9300
Elasticsearch rest服务访问不到则记得查看防火墙配置。
Elasticsearch默认安装到/usr/share/elasticsearch目录下
Elasticsearch配置文件默认在/etc/elasticsearch/目录下。可以使用rpm -qc命令查看。如下所示：

$ rpm -qc elasticsearch

/etc/elasticsearch/elasticsearch.yml

/etc/elasticsearch/logging.yml

/etc/init.d/elasticsearch

/etc/sysconfig/elasticsearch

/usr/lib/sysctl.d/elasticsearch.conf

/usr/lib/systemd/system/elasticsearch.service

/usr/lib/tmpfiles.d/elasticsearch.conf

1. 安装Kibana

https://www.elastic.co/guide/en/kibana/current/index.html

1. 1. 创建kibana.repo

$ sudo vim /etc/yum.repos.d/kibana.repo

[kibana-4.4]

name=Kibana repository for 4.4.x packages

baseurl=http://packages.elastic.co/kibana/4.4/centos

gpgcheck=1

gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch

enabled=1

1. 1. 使用yum install安装kibana

$ sudo yum -y install kibana

注：

Kibana默认端口为5601
kibana默认安装在/opt/kibana目录下
Kibana配置文件路径为/opt/kibana/config/kibana.yml

$ rpm -qc kibana

/opt/kibana/config/kibana.yml

1. 1. 修改kibana配置

$ sudo vim /opt/kibana/config/kibana.yml

server.host: "192.168.0.228"

elasticsearch.url: "http://192.168.0.228:9200"

1. 1. 启动kibana并添加为开机自启动服务

$ sudo systemctl start kibana

$ sudo chkconfig kibana on

1. 安装Nginx（此部分内容不是必须）

由于elasticsearch、kibana自身均没有提供访问权限安全问题，这里使用nginx代理来验证用户身份。

1. 1. 安装nginx

$ sudo yum -y install epel-release

$ sudo yum -y install nginx httpd-tools

1. 1. 创建用户并设定密码

$ sudo htpasswd -c /etc/nginx/htpasswd.users kibanaadmin #创建kibanaadmin用户

注：

这里创建的用户为kibanaadmin/kibanaadmin（用户密码均为kibanaadmin）

1. 1. 修改/etc/nginx/nginx.conf

$ sudo vim /etc/nginx/nginx.conf

user nginx;

worker_processes auto;

error_log /var/log/nginx/error.log;

pid /run/nginx.pid;

events {

worker_connections 1024;

}

http {

log_format main '$remote_addr - $remote_user [$time_local] "$request" '

'$status $body_bytes_sent "$http_referer" '

'"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log main;

sendfile on;

tcp_nopush on;

tcp_nodelay on;

keepalive_timeout 65;

types_hash_max_size 2048;

include /etc/nginx/mime.types;

default_type application/octet-stream;

include /etc/nginx/conf.d/*.conf;

}

1. 1. 创建/etc/nginx/conf.d/kibana.conf文件

sudo vim /etc/nginx/conf.d/kibana.conf

server {

listen 80;

server_name 192.168.0.228;

auth_basic "Restricted Access";

auth_basic_user_file /etc/nginx/htpasswd.users;

location / {

proxy_pass http://192.168.0.228:5601;

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection 'upgrade';

proxy_set_header Host $host;

proxy_cache_bypass $http_upgrade;

}

注：

以上配置使用http basic认证用户身份。
使用nginx反向代理到kibana所在服务器（http://192.168.0.228:5601）

为了使上述配置生效并能成功代理，需做如下操作

$ sudo setsebool -P httpd_can_network_connect 1

至此，访问nginx时则会要求输入用户名密码（kibanaadmin/kibanaadmin）。输入正确后请求会请求代理到kibana

1. 1. 启动nginx并添加到开启自启动服务

$ sudo systemctl start nginx

$ sudo systemctl enable nginx

1. 安装Logstash

https://www.elastic.co/guide/en/logstash/current/index.html

1. 1. 创建logstash.repo

$ sudo vim /etc/yum.repos.d/logstash.repo

[logstash-2.2]

name=logstash repository for 2.2 packages

baseurl=http://packages.elasticsearch.org/logstash/2.2/centos

gpgcheck=1

gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch

enabled=1

1. 1. 使用yum install安装logstash

$ sudo yum -y install logstash

注：

logstash默认安装在/opt/logstash目录
Logstash默认配置文件目录rpm -qc logstash

/etc/init.d/logstash

/etc/logrotate.d/logstash

/etc/sysconfig/logstash

1. 1. 生成ssl证书
    1. 根据ip生成

修改/etc/pki/tls/openssl.cnf文件，找到[ v3_ca ]节点。修改subjectAltName为IP:ELK安装机器IP。

sudo

内容如下：

[ v3_ca ]

subjectAltName = IP: 192.168.0.228

切换到/etc/pki/tls目录，生成证书

$ cd /etc/pki/tls

$ sudo openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

1. 1. 1. 根据域名生成

$ cd /etc/pki/tls

$ sudo openssl req -subj '/CN=www.elk.com/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

1. 1. Logstash配置

这里所有的配置均在/etc/logstash/conf.d目录下。

1. 1. 1. Input

创建一个beats input

$ sudo vim /etc/logstash/conf.d/02-beats-input.conf

input {

beats {

port => 5044

ssl => true

ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"

ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"

}

这里使用beats input，监听在5044端口上，使用之前生成的证书文件。

1. 1. 1. Filter

为syslog创建一个filter

$ sudo vim /etc/logstash/conf.d/10-syslog-filter.conf

filter {

if [type] == "syslog" {

grok {

match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

add_field => [ "received_at", "%{@timestamp}" ]

add_field => [ "received_from", "%{host}" ]

}

syslog_pri { }

date {

match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]

}

1. 1. 1. Output

将beat输入输出到elasticsearch

$ sudo vim /etc/logstash/conf.d/30-elasticsearch-output.conf

output {

elasticsearch {

hosts => ["192.168.0.228:9200"]

sniffing => true

manage_template => false

index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

document_type => "%{[@metadata][type]}"

}

1. 1. 测试配置是否正确

$ sudo service logstash configtest

如果显示Configuration OK则表示没有任何语法错误。

1. 1. 启动logstash并添加为开机自启动服务

$ sudo systemctl restart logstash

$ sudo chkconfig logstash on

1. 1. 安装 Kibana Dashboards

$ curl -L -O http://download.elastic.co/beats/dashboards/beats-dashboards-1.2.2.zip

$ unzip beats-dashboards-1.2.2.zip

$ cd beats-dashboards-1.2.2/

$ vim ./load.sh

ELASTICSEARCH=http://192.168.0.228:9200

$ ./load.sh

执行完后会创建如下index pattern

[packetbeat-]YYYY.MM.DD

[topbeat-]YYYY.MM.DD

[filebeat-]YYYY.MM.DD

[winlogbeat-]YYYY.MM.DD

使用kibana时，选择filebeat模式

1. 安装Filebeat

https://www.elastic.co/guide/en/beats/filebeat/1.2/index.html

1. 1. 导入elasticsearch公钥

$ sudo rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch

1. 1. 创建elastic-beats.repo

$ sudo vim /etc/yum.repos.d/elastic-beats.repo

[beats]

name=Elastic Beats Repository

baseurl=https://packages.elastic.co/beats/yum/el/$basearch

enabled=1

gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch

gpgcheck=1

1. 1. 安装filebeat

$ sudo yum -y install filebeat

或者

$ curl -L -O https://download.elastic.co/beats/filebeat/filebeat-1.2.2-x86_64.rpm

$ sudo rpm -vi filebeat-1.2.2-x86_64.rpm

注：rpm -qc filebeat查找filebeat核心配置文件为/etc/filebeat/filebeat.yml

1. 1. 配置Filebeat

Filebeat默认安装后其配置文件为/etc/filebeat/filebeat.yml。该配置文件遵从yaml语法格式。有较强的缩进等语法。可使用下列网站进行校验

http://yaml-online-parser.appspot.com/
http://www.yamllint.com/

1. 1. 1. 一个简单的配置
      1. 使用elasticsearch作为输出

filebeat:

prospectors:

paths:

- "/var/log/*.log"

output:

elasticsearch:

hosts: ["192.168.0.228:9200"]

以上配置表示filebeat收集/var/log/目录下所有以.log结尾的日志文件，输出到elasticsearch

1. 1. 1. 1. 使用logstash作为输出

filebeat:

prospectors:

paths:

- "/var/log/*.log"

document_type: syslog

output:

logstash:

bulk_max_size: 1024

hosts:

- "192.168.0.228:5044"

tls:

certificate_authorities:

- /etc/pki/tls/certs/logstash-forwarder.crt

以上配置表示filebeat收集/var/log/目录下所有以.log结尾的日志文件，输出到logstash。其中document_type为之前在logstash中/etc/logstash/conf.d/10-syslog-filter.conf中定义的type类型。5044端口为之前在/etc/logstash/conf.d/02-beats-input.conf中为beats定义的port。certificate_authorities同理，不再赘述。

1. 1. load filebeat template

$ curl -XPUT 'http://192.168.0.228:9200/_template/filebeat' -d@/etc/filebeat/filebeat.template.json

返回{"acknowledged":true}则表示成功。

删除filebeat template

$ curl -XDELETE 'http://192.168.0.228:9200/filebeat-*'

其中192.168.0.228:9200为elasticsearch服务。

1. 1. 启动filebeat并添加为系统开机自启动服务

$ sudo systemctl start filebeat

$ sudo systemctl enable filebeat

1. 1. 测试filebeat

$ curl -XGET 'http://192.168.0.228:9200/filebeat-*/_search?pretty'

{

"took" : 2,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"failed" : 0

"hits" : {

"total" : 1159,

"max_score" : 1.0,

"hits" : [ {

"_index" : "filebeat-2016.05.17",

"_type" : "syslog",

"_id" : "AVS8XSsvMXchSyg0mTVB",

"_score" : 1.0,

"_source" : {

"message" : "May 16 21:35:11 c1 journal: Journal started",

"@version" : "1",

"@timestamp" : "2016-05-17T01:35:11.000Z",

"source" : "/var/log/messages",

"input_type" : "log",

"type" : "syslog",

"count" : 1,

"fields" : null,

"beat" : {

"hostname" : "c1",

"name" : "c1"

"offset" : 527932,

"host" : "c1",

"tags" : [ "beats_input_codec_plain_applied" ],

"syslog_timestamp" : "May 16 21:35:11",

"syslog_hostname" : "c1",

"syslog_program" : "journal",

"syslog_message" : "Journal started",

"received_at" : "2016-05-17T01:36:06.259Z",

"received_from" : "c1",

"syslog_severity_code" : 5,

"syslog_facility_code" : 1,

"syslog_facility" : "user-level",

"syslog_severity" : "notice"

}

}]

}

观察控制台输出，若有结果输出则表示配置成功，否则仔细检查配置。

1. 1. Connect to Kibana

http://192.168.0.228/会要求输入用户名密码，输入之前设置的kibanaadmin/kibanaadmin后，会反向代理到http://192.168.0.228/app/kibana

第一次请求系统要求设置一个默认的index pattern。这里默认设置filebeat-*为默认。

依次点Settings->filebeat- ->★ 即可。

Discover

1. 安装topbeat

https://www.elastic.co/guide/en/beats/topbeat/current/index.html

1. 1. 导入elasticsearch公钥

$ sudo rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch

1. 1. 创建elastic-beats.repo

$ sudo vim /etc/yum.repos.d/elastic-beats.repo

[beats]

name=Elastic Beats Repository

baseurl=https://packages.elastic.co/beats/yum/el/$basearch

enabled=1

gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch

gpgcheck=1

1. 1. 安装topbeat

$ sudo yum -y install topbeat

或

$ curl -L -O https://download.elastic.co/beats/topbeat/topbeat-1.2.2-x86_64.rpm

$ sudo rpm -vi topbeat-1.2.2-x86_64.rpm

注：rpm -qc topbeat 查找topbeat核心配置文件为/etc/topbeat/topbeat.yml

1. 1. 配置Topbeat

$ sudo vim /etc/topbeat/topbeat.yml

output:

logstash:

hosts: ["192.168.0.228:5044"]

tls:

certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

这里配置同filebeat不再赘述。

1. 1. load topbeat template

$ curl -XPUT 'http://192.168.0.228:9200/_template/topbeat' -d@/etc/topbeat/topbeat.template.json

返回{"acknowledged":true}则表示成功。

删除topbeat template

$ curl -XDELETE 'http://192.168.0.228:9200/topbeat-*'

其中192.168.0.228:9200为elasticsearch服务。

1. 1. 启动topbeat并添加为系统开机自启动服务

$ sudo systemctl restart topbeat

$ sudo systemctl enable topbeat

1. 1. 测试topbeat

$ curl -XGET 'http://192.168.0.228:9200/topbeat-*/_search?pretty'

{

"took" : 8,

"timed_out" : false,

"_shards" : {

"total" : 5,

"successful" : 5,

"failed" : 0

"hits" : {

"total" : 277442,

"max_score" : 1.0,

"hits" : [ {

"_index" : "topbeat-2016.05.17",

"_type" : "system",

"_id" : "AVS8XHQPMXchSyg0mTFD",

"_score" : 1.0,

"_source" : {

"@timestamp" : "2016-05-17T01:37:26.228Z",

"type" : "system",

"load" : {

"load1" : 4.07,

"load5" : 1.8,

"load15" : 0.68

"cpu" : {

"user" : 3126,

"user_p" : 0.0293,

"nice" : 3190,

"system" : 2627,

"system_p" : 0.0984,

"idle" : 156,

"iowait" : 2322,

"irq" : 0,

"softirq" : 485,

"steal" : 0

"mem" : {

"total" : 3009445888,

"used" : 948916224,

"free" : 2060529664,

"used_p" : 0.32,

"actual_used" : 664776704,

"actual_free" : 2344669184,

"actual_used_p" : 0.22

"swap" : {

"total" : 1610608640,

"used" : 0,

"free" : 1610608640,

"used_p" : 0

"count" : 1,

"beat" : {

"hostname" : "c1",

"name" : "c1"

"@version" : "1",

"host" : "c1",

"tags" : [ "beats_input_raw_event" ]

}

}]

}

返回类似如上信息则表示配置成功。

1. 1. Connect to Kibana

使用Topbeat Dashboard

1. logstash扩展配置

首先得明确以下几点

logstash安装在/opt/logstash
logstash配置目录为/etc/logstash/conf.d
确定存在名为02-beats-input.conf配置文件，该文件在上文之前创建配置过
确定存在名为30-elasticsearch-output.conf配置文件，该文件在上文之前创建配置过

创建patterns

$ sudo mkdir -p /opt/logstash/patterns

$ sudo chown logstash: /opt/logstash/patterns

修改/etc/filebeat/filebeat.yml文件

filebeat:

prospectors:

document_type: syslog

paths:

- /var/log/secure

- /var/log/messages

document_type: sys-log

input_type: log

paths:

- /var/log/*.log

registry_file: /var/lib/filebeat/registry

logging:

files:

rotateeverybytes: 10485760

output:

logstash:

bulk_max_size: 1024

hosts:

- "192.168.0.228:5044"

tls:

certificate_authorities:

- /etc/pki/tls/certs/logstash-forwarder.crt

shipper: ~

1. 1. Nginx日志配置
    1. Logstash Patterns: Nginx

$ sudo mkdir -p /opt/logstash/patterns

sudo vim /opt/logstash/patterns/nginx

NGUSERNAME [a-zA-Z\.\@\-\+_%]+

NGUSER %{NGUSERNAME}

NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}

$ sudo chown logstash: /opt/logstash/patterns/nginx

1. 1. 1. Logstash Filter: Nginx

$ sudo vim /etc/logstash/conf.d/11-nginx-filter.conf

filter {

if [type] == "nginx-access" {

grok {

match => { "message" => "%{NGINXACCESS}" }

}

1. 1. 1. 重启logstash

$ sudo service logstash restart

1. 1. 1. Filebeat Prospector: Nginx

修改/etc/filebeat/filebeat.yml配置

$ sudo vim /etc/filebeat/filebeat.yml

filebeat:

prospectors:

document_type: nginx-access

paths:

- /var/log/nginx/access.log

registry_file: /var/lib/filebeat/registry

logging:

files:

rotateeverybytes: 10485760

output:

logstash:

bulk_max_size: 1024

hosts:

- "192.168.0.228:5044"

tls:

certificate_authorities:

- /etc/pki/tls/certs/logstash-forwarder.crt

shipper: ~

1. 1. 1. 重启filebeat

$ sudo service filebeat restart

1. 1. 1. kibana搜索效果图

1. 1. Apache HTTP Web Server日志配置
    1. Logstash Filter: Apache

$ sudo vi /etc/logstash/conf.d/12-apache.conf

filter {

if [type] == "apache-access" {

grok {

match => { "message" => "%{COMBINEDAPACHELOG}" }

}

1. 1. 1. 重启logstash

$ sudo service logstash restart

1. 1. 1. Filebeat Prospector: Apache

$ sudo vim /etc/filebeat/filebeat.yml

filebeat:

prospectors:

document_type: apache-access

input_type: log

paths:

- /var/log/apache2/access.log

registry_file: /var/lib/filebeat/registry

logging:

files:

rotateeverybytes: 10485760

output:

logstash:

bulk_max_size: 1024

hosts:

- "192.168.0.228:5044"

tls:

certificate_authorities:

- /etc/pki/tls/certs/logstash-forwarder.crt

shipper: ~

1. 1. 1. 重启filebeat

sudo service filebeat restart

1. 1. Tomcat日志配置

参考链接

http://blog.kazaff.me/2015/06/05/%E6%97%A5%E5%BF%97%E6%94%B6%E9%9B%86%E6%9E%B6%E6%9E%84--ELK/

https://aggarwalarpit.wordpress.com/2015/12/03/configuring-elk-stack-to-analyse-apache-tomcat-logs/

https://www.systemcodegeeks.com/web-servers/apache/configuring-elk-stack-analyse-apache-tomcat-logs/

http://stackoverflow.com/questions/25429377/how-can-i-integrate-tomcat6s-catalina-out-file-with-logstash-elasticsearch

https://blog.codecentric.de/en/2014/10/log-management-spring-boot-applications-logstash-elastichsearch-kibana/

https://github.com/sdd330/tomcat-elk

https://blog.lanyonm.org/articles/2014/01/12/logstash-multiline-tomcat-log-parsing.html

https://spredzy.wordpress.com/2013/03/02/monitor-your-cluster-of-tomcat-applications-with-logstash-and-kibana/

1. 1. 1. 定义Logstash Patterns: Tomcat

$ vim /opt/logstash/patterns/tomcat

JAVACLASS (?:[a-zA-Z0-9-]+\.)+[A-Za-z0-9$]+

JAVALOGMESSAGE (.*)

# MMM dd, yyyy HH:mm:ss eg: Jan 9, 2014 7:13:13 AM

CATALINA_DATESTAMP %{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM)

# yyyy-MM-dd HH:mm:ss,SSS ZZZ eg: 2014-01-09 17:32:25,527 -0800

TOMCAT_DATESTAMP 20%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{ISO8601_TIMEZONE}

CATALINALOG %{CATALINA_DATESTAMP:timestamp} %{JAVACLASS:class} %{JAVALOGMESSAGE:logmessage}

# 2014-01-09 20:03:28,269 -0800 | ERROR | com.example.service.ExampleService - something compeletely unexpected happened...

TOMCATLOG %{TOMCAT_DATESTAMP:timestamp} \| %{LOGLEVEL:level} \| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}

1. 1. 1. 定义Logstash Filter: Tomcat

$ vim /etc/logstash/conf.d/13-tomcat.conf

filter {

if [type] == "tomcat_access" {

grok {

match => [ "message", "%{TOMCATLOG}", "message", "%{CATALINALOG}" ]

}

date {

match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS Z", "MMM dd, yyyy HH:mm:ss a" ]

}

1. 1. 1. 重启logstash

$ sudo service logstash restart

1. 1. 1. Filebeat Prospector: Tomcat

修改/etc/filebeat/filebeat.yml配置

$ sudo vim /etc/filebeat/filebeat.yml

filebeat:

prospectors:

document_type: tomcat-access

input_type: log

paths:

- /home/vagrant/tomcat-7.0.69/logs/*.log

registry_file: /var/lib/filebeat/registry

logging:

files:

rotateeverybytes: 10485760

output:

logstash:

bulk_max_size: 1024

hosts:

- "192.168.0.228:5044"

tls:

certificate_authorities:

- /etc/pki/tls/certs/logstash-forwarder.crt

shipper: ~

1. 1. 1. 重启filebeat

$ sudo service filebeat restart

1. 1. 1. kibana搜索效果图

1. 1. 最终配置

/etc/filebeat/filebeat.yml集各配置于一体的一个最终配置如下：

---

filebeat:

prospectors:

document_type: syslog

paths:

- /var/log/secure

- /var/log/messages

document_type: sys-log

input_type: log

paths:

- /var/log/*.log

document_type: nginx-access

paths:

- /var/log/nginx/access.log

document_type: apache-access

input_type: log

paths:

- /var/log/apache2/access.log

document_type: tomcat-access

input_type: log

paths:

- /home/vagrant/tomcat-7.0.69/logs/*.log

registry_file: /var/lib/filebeat/registry

logging:

files:

rotateeverybytes: 10485760

output:

logstash:

bulk_max_size: 1024

hosts:

- "192.168.0.228:5044"

tls:

certificate_authorities:

- /etc/pki/tls/certs/logstash-forwarder.crt

shipper: ~

1. 使用Kibana查询分析日志

这里注意涉及Discover、Visualize、Dashboard、Settings面板的使用。具体用法请结合官方文档。这里不再赘述。

系统日志

Nginx日志

详细可参考git地址：https://github.com/ameizi/ELK

1. Elasticsearch插件安装

https://www.elastic.co/guide/en/elasticsearch/plugins/current/installation.html

1. 1. plugin命令介绍

Elasticsearch默认安装在/usr/share/elasticsearch路径下

进入到/usr/share/elasticsearch/bin目录。使用该目录下的plugin命令管理插件

使用./plugin -h命令会列出plugin命令选项的提示信息

./plugin install #安装插件

./plugin remove #移除插件

./plugin list #列出已安装插件列表

这里列出我常用的也是功能最为强大的三款插件

1. 1. 安装head插件

$ sudo /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head

访问http://192.168.0.228:9200/_plugin/head/

1. 1. 安装bigdesk插件

$ sudo /usr/share/elasticsearch/bin/plugin install lukas-vlcek/bigdesk/2.5.0

访问http://192.168.0.228:9200/_plugin/bigdesk/

1. 1. 安装kopf插件

$ sudo /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf/2.1.2

访问http://192.168.0.228:9200/_plugin/kopf/

1. 1. 插件列表

https://www.elastic.co/guide/en/elasticsearch/plugins/current/management.html

https://www.elastic.co/guide/en/elasticsearch/plugins/current/integrations.html

参考资料

https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-7

https://www.digitalocean.com/community/tutorials/how-to-gather-infrastructure-metrics-with-topbeat-and-elk-on-centos-7

https://www.digitalocean.com/community/tutorials/adding-logstash-filters-to-improve-centralized-logging

https://www.digitalocean.com/community/tutorials/how-to-use-kibana-dashboards-and-visualizations

https://www.digitalocean.com/community/tutorials/how-to-map-user-location-with-geoip-and-elk-elasticsearch-logstash-and-kibana

使用Elasticsearch做全文检索

当使用elasticsearch搜索中文时就需要安装中文分词器。

关于elasticsearch更多内容可参阅

https://github.com/sxyx2008/elasticsearch elasticsearch中文版，基于elasticsearch-1.7.1。集成常用的各种插件

https://github.com/sxyx2008/elasticsearch-jest-example ElasticSearch Java API编程接口

https://github.com/sxyx2008/elasticsearch/issues/2 elasticsearch analysis ansj分词器的安装及使用

https://github.com/sxyx2008/elasticsearch/issues/3 elasticsearch-jdbc插件的使用

https://github.com/sxyx2008/elasticsearch/issues/5 elasticsearch rest api快速上手

centos7搭建elk日志分析系统

ELK

猜你喜欢