grafana 安装和 Nginx 、EL 联调

1、grafana 安装
wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.6.2-1.x86_64.rpm
为了使用yum方便，直接下载阿里的yum源
[root@hadoop001 yum.repos.d]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo
[root@hadoop001 elk]# yum install initscripts fontconfig

[root@hadoop001 elk]# rpm -Uvh grafana-4.6.2-1.x86_64.rpm
warning: grafana-4.6.2-1.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 24098cb6: NOKEY
error: Failed dependencies:
urw-fonts is needed by grafana-4.6.2-1.x86_64
[root@hadoop001 elk]# yum install -y urw-fonts
[root@hadoop001 elk]# rpm -Uvh grafana-4.6.2-1.x86_64.rpm
warning: grafana-4.6.2-1.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 24098cb6: NOKEY
Preparing... ########################################### [100%]
1:grafana ########################################### [100%]
### NOT starting grafana-server by default on bootup, please execute
sudo /sbin/chkconfig --add grafana-server
### In order to start grafana-server, execute
sudo service grafana-server start
POSTTRANS: Running script

启动
[root@hadoop001 elk]# service grafana-server start
Starting Grafana Server: ... [ OK ]

添加自启动
[root@hadoop001 elk]# /sbin/chkconfig --add grafana-server

[root@hadoop001 elk]# ps -ef | grep grafana
grafana 25478 1 0 16:32 ? 00:00:00 /usr/sbin/grafana-server --pidfile=/var/run/grafana-server.pid --config=/etc/grafana/grafana.ini cfg:default.paths.data=/var/lib/grafana cfg:default.paths.logs=/var/log/grafana cfg:default.paths.plugins=/var/lib/grafana/plugins
root 25502 24192 0 16:33 pts/3 00:00:00 grep grafana
[root@hadoop001 elk]#
[root@hadoop001 elk]#
[root@hadoop001 elk]#
[root@hadoop001 elk]# netstat -nlp | grep 25478
tcp 0 0 :::3000 :::* LISTEN 25478/grafana-serve

打开网页：
admin / admin

2、简单使用
这里我们是和 nginx es logstash 联用。
打开这个网页 https://grafana.com/dashboards/2292
这是一个做好的dashboard ，按照上面对 nginx access 配置

2.1配置 nginx access log format
找到配置log_format 的地方
log_format main '{"@timestamp":"$time_iso8601",'
'"@source":"$server_addr",'
'"hostname":"$hostname",'
'"ip":"$http_x_forwarded_for",'
'"client":"$remote_addr",'
'"request_method":"$request_method",'
'"scheme":"$scheme",'
'"domain":"$server_name",'
'"referer":"$http_referer",'
'"request":"$request_uri",'
'"args":"$args",'
'"size":$body_bytes_sent,'
'"status": $status,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamaddr":"$upstream_addr",'
'"http_user_agent":"$http_user_agent",'
'"https":"$https"'
'}';
重启 nginx
[root@hadoop001 nginx]# sbin/nginx -s reload

查看日志
[root@hadoop001 nginx]# tail -f logs/access.log
刷新nginx 网页，发现生成新的记录。
{"@timestamp":"2017-12-03T16:40:36+08:00","@source":"192.168.137.11","hostname":"hadoop001","ip":"-","client":"192.168.137.1","request_method":"GET","scheme":"http","domain":"192.168.137.11","referer":"-","request":"/","args":"-","size":0,"status": 304,"responsetime":0.000,"upstreamtime":"-","upstreamaddr":"-","http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0","https":""}

nginx 配置完成

2.2 配置 Logstash nginx_access.conf
[root@hadoop001 nginx]# cd /opt/software/elk/logstash-6.0.0/
[root@hadoop001 logstash-6.0.0]# vi logstash-nginx-access-log.conf
input {
file {
#这里根据自己日志命名使用正则匹配所有域名访问日志
path => [ "/usr/local/nginx/logs/access.log" ]
ignore_older => 0
codec => json
}
}

filter {
mutate {
convert => [ "status","integer" ]
convert => [ "size","integer" ]
convert => [ "upstreatime","float" ]
remove_field => "message"
}
geoip {
source => "ip"
}
}
output {
elasticsearch {
hosts => "192.168.137.11:9200"
index => "logstash-nginx-access-log"
}
# stdout {codec => rubydebug}
}

重启 logstash
kill -9 $(pgrep -f logstash)
[root@hadoop001 logstash-6.0.0]# kill -9 $(pgrep -f logstash)
[root@hadoop001 logstash-6.0.0]# nohup bin/logstash -f logstash-nginx-access-log.conf &

刷新网页，查看 nohup 和 es 中数据
注意：如果es 中原来有数据，最好清除掉，否则可能会像这样报错
[2017-12-03T16:50:57,757][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-nginx-access-log", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x7664ee1b>], :response=>{"index"=>{"_index"=>"logstash-nginx-access-log", "_type"=>"doc", "_id"=>"_rqSG2ABfyb-pkaaQGEt", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [logstash-nginx-access-log] as the final mapping would have more than 1 type: [doc, nginx_access]"}}}}

3、add data source