ELK部署

版本

elasticsearch-6.0.0.tar.gz
logstash-6.0.0.tar.gz
kibana-6.0.0-linux-x86_64.tar.gz

总体流程

Nginx —生成—> access.log —> logstash进行收集 —> 收集到elasticsearch中进行存储 —> kibana进行展示

es部署

[hadoop@hadoop001 ~]$ cd /opt/app/elasticsearch-6.0.0/
[hadoop@hadoop001 elasticsearch-6.0.0]$ mkdir data
[hadoop@hadoop001 elasticsearch-6.0.0]$ mkdir logs

修改elasticsearch.yml：

[/opt/app/elasticsearch-6.0.0/config/elasticsearch.yml]
cluster.name: ruozedata_cluster
node.name: node131                                  ## 131为机器ip的尾号(192.168.26.131)
path.data: /opt/app/elasticsearch-6.0.0/data
path.logs: /opt/app/elasticsearch-6.0.0/logs
network.host: 192.168.26.131

启动es：

[hadoop@hadoop001 elasticsearch-6.0.0]$ bin/elasticsearch

启动失败(通过查看进程发现)：

[hadoop@hadoop001 elasticsearch-6.0.0]$ ps -ef | grep elastic
hadoop     3680   3438  0 06:36 pts/0    00:00:00 grep elastic

查看logs/ruozedata_cluster.log：

[2018-04-30T06:35:54,704][ERROR][o.e.b.Bootstrap          ] [node131] node validation exception
[4] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: max number of threads [1024] for user [hadoop] is too low, increase to at least [4096]
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[4]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

针对4个报错的解决方案

问题1：
nofile 表示进程允许打开的最大文件数。elasticsearch 进程要求可以打开的最大文件数不低于 65536

[root@hadoop001 ~]# echo "* soft nofile 65536" >> /etc/security/limits.conf
[root@hadoop001 ~]# echo "* hard nofile 131072" >> /etc/security/limits.conf

问题2：
nproc 表示最大线程数。elasticsearch 要求最大线程数不低于 4096
hadoop为用户名

[root@hadoop001 ~]# echo "hadoop soft nproc 4096" >> /etc/security/limits.conf
[root@hadoop001 ~]# echo "hadoop hard nproc 4096" >> /etc/security/limits.conf

问题3：
vm.max_map_count 表示虚拟内存大小，它是一个内核参数。elasticsearch 默认要求 vm.max_map_count 不低于 262144。
临时生效

[root@hadoop001 ~]# sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144

永久生效

[root@hadoop001 ~]# echo "vm.max_map_count=262144" >> /etc/sysctl.conf
[root@hadoop001 ~]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
vm.max_map_count = 262144

问题4：
2种方案：

升级centos6 –> 7
选择禁用
进行禁用，修改配置文件elasticsearch.yml

[/opt/app/elasticsearch-6.0.0/config/elasticsearch.yml]
bootstrap.memory_lock: false
bootstrap.system_call_filter: false

查看是否修改成功

[root@hadoop001 ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31407
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31407
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

发现并没有全部生效，需要进行重启之后，才会进行生效

重新启动es：

-d 表示不直接打印日志在控制台上

[hadoop@hadoop001 elasticsearch-6.0.0]$ bin/elasticsearch -d

查看进程，看es是否起来了

[hadoop@hadoop001 elasticsearch-6.0.0]$ ps -ef | grep elastic

看日志，看es是否起来了

[2018-04-30T14:04:34,820][INFO ][o.e.n.Node               ] [node131] started
[2018-04-30T14:04:34,829][INFO ][o.e.g.GatewayService     ] [node131] recovered [0] indices into cluster_state

标志es启动成功：

[hadoop@hadoop001 logs]$ curl -XGET '192.168.26.131:9200/?pretty'
{
  "name" : "node131",
  "cluster_name" : "ruozedata_cluster",
  "cluster_uuid" : "BjfFLxYUR9S9WYdDRC60sQ",
  "version" : {
    "number" : "6.0.0",
    "build_hash" : "8f0685b",
    "build_date" : "2017-11-10T18:41:22.859Z",
    "build_snapshot" : false,
    "lucene_version" : "7.0.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

[hadoop@hadoop001 logs]$ jps
3233 Jps
3182 Elasticsearch

Nginx部署

安装PCRE库

[hadoop@hadoop001 ~]$ su - root
[root@hadoop001 ~]$ cd /opt/software/
[root@hadoop001 software]$ tar -zxvf pcre-8.36.tar.gz -C /opt/app/
[root@hadoop001 app]$ cd pcre-8.36/
[root@hadoop001 pcre-8.36]$ ./configure
[root@hadoop001 pcre-8.36]$ make
[root@hadoop001 pcre-8.36]$ make install

安装zlib库

[root@hadoop001 pcre-8.36]# cd /opt/software/
[root@hadoop001 software]# tar -zxvf zlib-1.2.8.tar.gz -C /opt/app/
[root@hadoop001 software]# cd /opt/app/zlib-1.2.8/
[root@hadoop001 zlib-1.2.8]# ./configure
[root@hadoop001 zlib-1.2.8]# make
[root@hadoop001 zlib-1.2.8]# make install

安装ssl

[root@hadoop001 zlib-1.2.8]# cd /opt/software/
[root@hadoop001 software]# tar -zxvf openssl-1.0.1j.tar.gz -C /opt/app/
[root@hadoop001 software]# cd /opt/app/openssl-1.0.1j/
[root@hadoop001 openssl-1.0.1j]# ./config
[root@hadoop001 openssl-1.0.1j]# make
[root@hadoop001 openssl-1.0.1j]# make install

安装Nginx

[root@hadoop001 local]# cd /usr/local
[root@hadoop001 local]# wget http://nginx.org/download/nginx-1.8.0.tar.gz
[root@hadoop001 local]# cd nginx-1.8.0/
[root@hadoop001 nginx-1.8.0]# ./configure --prefix=/usr/local/nginx --with-pcre=/opt/app/pcre-8.36 --with-zlib=/opt/app/zlib-1.2.8
[root@hadoop001 nginx-1.8.0]# make
[root@hadoop001 nginx-1.8.0]# make install

启动Nginx

[root@hadoop001 nginx]# /usr/local/nginx/sbin/nginx

查看状态和配置文件
-t 查看当前配置文件是否OK，当前状态是否OK

[root@hadoop001 nginx]# /usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

查看进程

[root@hadoop001 nginx]# ps -ef | grep nginx

查看端口号

[root@hadoop001 nginx]# netstat -nlp | grep nginx

默认端口是80

解决端口冲突的2种方法：

httpd的服务，默认端口是80，因此需要将其停止
修改nginx.conf
[root@hadoop001 ~]# cd /usr/local/nginx/conf/
[root@hadoop001 conf]# vi nginx.conf
修改下列内容：

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

access_log  logs/access.log  main;

listen       888;
server_name  hadoop001;

通过 http://192.168.26.131:888/ 去访问Nginx的web ui

[root@hadoop001 logs]# cd /usr/local/nginx/logs
[root@hadoop001 logs]# tail -F access.log

logstash部署

[root@hadoop001 logs]# cd /opt/app/logstash-6.0.0/
[root@hadoop001 logstash-6.0.0]# vi logstash-nginx-access-log.conf
input {
    file {
        path => ["/usr/local/nginx/logs/access.log"]
        type => "nginx_access"
        start_position => "beginning"
    }
}

filter {
  grok {
    match => {
      "message" => '%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] "%{WORD:request_action} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:response} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:agent}"'
    }
  }

  date {
    match => [ "time", "dd/MMM/YYYY:HH:mm:ss Z" ]
    locale => en
  }}

output {
  elasticsearch {
        hosts => ["192.168.26.131:9200"]
        index => "logstash-nginx-access-log"
    }
  # 输出到当前的控制台
  stdout { codec => rubydebug }
}

启动logstash

[hadoop@hadoop001 ~]$ cd /opt/app/logstash-6.0.0
[hadoop@hadoop001 logstash-6.0.0]$ nohup bin/logstash -f logstash-nginx-access-log.conf  &
[hadoop@hadoop001 logstash-6.0.0]$ tail -F nohup.out 
刷新一次http://192.168.26.131:888/，就会有日志收集过来，并在控制台上打印出来(从Nginx的日志中收集过来，因此Nginx也需要启动)
{
           "request" => "/",
    "request_action" => "GET",
             "agent" => "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36",
         "user_name" => "-",
      "http_version" => "1.1",
           "message" => "192.168.26.1 - - [01/May/2018:10:12:26 +0800] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36\" \"-\"",
              "type" => "nginx_access",
              "path" => "/usr/local/nginx/logs/access.log",
          "referrer" => "-",
        "@timestamp" => 2018-05-01T02:12:26.000Z,
         "remote_ip" => "192.168.26.1",
          "response" => "304",
             "bytes" => "0",
          "@version" => "1",
              "host" => "hadoop001",
              "time" => "01/May/2018:10:12:26 +0800"
}

kibana部署

[hadoop@hadoop001 ~]$ cd /opt/software/
[hadoop@hadoop001 software]$ tar -zxvf kibana-6.0.0-linux-x86_64.tar.gz -C /opt/app/
[hadoop@hadoop001 software]$ cd /opt/app/kibana-6.0.0-linux-x86_64/
[hadoop@hadoop001 kibana-6.0.0-linux-x86_64]$ cd config/

修改如下配置信息：

server.host: "192.168.26.131"
elasticsearch.url: "http://192.168.26.131:9200"

启动kibana

[hadoop@hadoop001 kibana-6.0.0-linux-x86_64]$ bin/kibana

通过 http://192.168.26.131:5601 可以进行访问
在Management中配置

Index pattern               logstash-nginx-access-log  (即logstash-nginx-access-log.conf中配置的index的值)
Time Filter field name      @timestamp

配置完之后，有图片展示：
这里写图片描述

版本