ELK+redis+filebeat收集Apache日志

准备环境：两台centos7
规划如下：
ip地址：
服务端：192.168.232.135：
jdk
kibana
logstash
elasticsearch
redis
192.168.232.136：
jdk
elasticsearch
filebeat
Apache

1.关闭防火墙：

systemctl stop firewalld
setenforce 0

2.上传ELK压缩包并解压：

3.安装jdk（两台都安装）

[root@localhost ELK]# rpm -ivh jdk-8u131-linux-x64_.rpm

4.安装elasticsearch（两台都装）

rpm -ivh elasticsearch-6.6.2.rpm

5.配置192.168.232.135的elasticsearch配置文件

[root@localhost ELK]# vim /etc/elasticsearch/elasticsearch.yml

开启elasticsearch

[root@localhost ELK]# systemctl start elasticsearch

验证：

5.1配置192.168.232.136的elasticsearch配置文件


开启elasticsearch

[root@localhost ELK]# systemctl start elasticsearch

验证：

6.服务端安装redis

[root@localhost ~]# tar zxf redis-5.0.0.tar.gz
[root@localhost ~]# cp -r redis-5.0.0 /usr/local/redis
[root@localhost ~]# cd /usr/local/redis/
[root@localhost redis]# yum -y install gcc-c++
[root@localhost redis]# make
[root@localhost redis]# make MALLOC=libc
1.给redis做软链接
	[root@localhost redis]# ln -sv /usr/local/redis/src/redis-server /usr/bin/redis-	server
	"/usr/bin/redis-server" -> "/usr/local/redis/src/redis-server"
	[root@localhost redis]# ln -sv /usr/local/redis/src/redis-cli /usr/bin/redis-cli
	"/usr/bin/redis-cli" -> "/usr/local/redis/src/redis-cli"
2.修改配置文件
	[root@localhost redis]# vim /usr/local/redis/redis.conf
	69行修改：
 	bind 192.168.232.135
 	508行添加：
 	requirepass 123321
 3.启动redis：
 	[root@localhost redis]# redis-server ./redis.conf
 4.把511追加到/proc/sys/net/core/somaxconn：
 	[root@localhost redis]# echo 511 >> /proc/sys/net/core/somaxconn
 5.去/etc/sysctl.conf最后一行添加下面命令：
 	[root@localhost redis]# vim /etc/sysctl.conf
 		vm.overcommit_memory = 1
 6.在/etc/rc.local里面添加：
 	echo never > /sys/kernel/mm/transparent_hugepage/enabled
 7.进入redis配置文件136行将daemonize no修改为yes
 	[root@localhost redis]# vim /usr/local/redis/redis.conf
 		136行将daemonize no修改为daemonize yes
 8.重启redis：
 	[root@localhost redis]# redis-server ./redis.conf
 9.测试：
 192.168.232.135:6379> auth 123321
 OK
 192.168.232.135:6379> get *
 (nil)
 

7.客户端安装httpd并启动：

[root@localhost ELK]# yum -y install httpd
[root@localhost ELK]# systemctl start httpd

8.客户端安装filebeat：

[root@localhost ~]# rpm -ivh filebeat-6.8.1-x86_64.rpm

9.配置filebeat.yml文件收集httpd的日志

[root@localhost ~]# cd /etc/filebeat/
备份一份原配置文件，以防改错：
[root@localhost filebeat]# cp filebeat.yml filebeat.yml.bak
[root@localhost filebeat]# vim filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/httpd/access_log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  setup.ilm.enabled: false
  setup.template.name: "filebeat-httpd"
  setup.template.pattern: "filebeat-httpd-*"
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.redis:
  hosts: ["192.168.232.135:6379"]
  key: "filebeat-httpd"
  db: 1
  timeout: 5
  password: 123321

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

10.服务端安装logstash

[root@localhost ELK]# rpm -ivh logstash-6.6.0.rpm

11.配置logstash文件添加httpd的索引

[root@localhost ELK]# cd /etc/logstash/conf.d/
[root@localhost conf.d]# vim httpd.conf
input {
    
    
        redis {
    
    
                data_type => "list"
                host => "192.168.232.135"
                password => "123321"
                port => "6379"
                db => "1"
                key => "filebeat-httpd"
        }
}

output {
    
    

         elasticsearch {
    
    
                hosts => ["192.168.232.135:9200"]
                index => "redis-httpdlog-%{+YYYY.MM.dd}"




         }
}

12.开启logstash并验证：

1.[root@localhost conf.d]# systemctl start logstash
2.[root@localhost conf.d]# netstat -nltp |grep 9600

13.安装kibana：

1.[root@localhost ELK]# rpm -ivh kibana-6.6.2-x86_64.rpm

13.1.修改kibana配置文件：

[root@localhost ELK]# vim /etc/kibana/kibana.yml

13.2.开启kibana服务并验证：

[root@localhost ELK]# systemctl start kibana
[root@localhost ELK]# netstat -nltp |grep 5601

14.http://192.168.232.135:5601/浏览器访问kibana页面：




附加：如果进入页面之后索引出不来就进入httpd网页刷新页面httpd索引就会出来了