ELK+redis+filebeat collect Apache logs

Prepare the environment : two centos7
planned as follows:
ip address:
server: 192.168.232.135:
the JDK
kibana
logstash
elasticsearch
Redis
192.168.232.136:
the JDK
elasticsearch
filebeat
the Apache

1. Turn off the firewall:

systemctl stop firewalld
setenforce 0

2. Upload the ELK compressed package and decompress it:

3. Install jdk (both installed)

[root@localhost ELK]# rpm -ivh jdk-8u131-linux-x64_.rpm

4. Install elasticsearch (both installed)

rpm -ivh elasticsearch-6.6.2.rpm

5. Configure the elasticsearch configuration file of 192.168.232.135

[root@localhost ELK]# vim /etc/elasticsearch/elasticsearch.yml

Open elasticsearch

[root@localhost ELK]# systemctl start elasticsearch

Verification:

5.1 Configure the elasticsearch configuration file of 192.168.232.136,


open elasticsearch

[root@localhost ELK]# systemctl start elasticsearch

Verification:

6. Install redis on the server

[root@localhost ~]# tar zxf redis-5.0.0.tar.gz
[root@localhost ~]# cp -r redis-5.0.0 /usr/local/redis
[root@localhost ~]# cd /usr/local/redis/
[root@localhost redis]# yum -y install gcc-c++
[root@localhost redis]# make
[root@localhost redis]# make MALLOC=libc
1.给redis做软链接
	[root@localhost redis]# ln -sv /usr/local/redis/src/redis-server /usr/bin/redis-	server
	"/usr/bin/redis-server" -> "/usr/local/redis/src/redis-server"
	[root@localhost redis]# ln -sv /usr/local/redis/src/redis-cli /usr/bin/redis-cli
	"/usr/bin/redis-cli" -> "/usr/local/redis/src/redis-cli"
2.修改配置文件
	[root@localhost redis]# vim /usr/local/redis/redis.conf
	69行修改：
 	bind 192.168.232.135
 	508行添加：
 	requirepass 123321
 3.启动redis：
 	[root@localhost redis]# redis-server ./redis.conf
 4.把511追加到/proc/sys/net/core/somaxconn：
 	[root@localhost redis]# echo 511 >> /proc/sys/net/core/somaxconn
 5.去/etc/sysctl.conf最后一行添加下面命令：
 	[root@localhost redis]# vim /etc/sysctl.conf
 		vm.overcommit_memory = 1
 6.在/etc/rc.local里面添加：
 	echo never > /sys/kernel/mm/transparent_hugepage/enabled
 7.进入redis配置文件136行将daemonize no修改为yes
 	[root@localhost redis]# vim /usr/local/redis/redis.conf
 		136行将daemonize no修改为daemonize yes
 8.重启redis：
 	[root@localhost redis]# redis-server ./redis.conf
 9.测试：
 192.168.232.135:6379> auth 123321
 OK
 192.168.232.135:6379> get *
 (nil)
 

7. The client installs httpd and starts:

[root@localhost ELK]# yum -y install httpd
[root@localhost ELK]# systemctl start httpd

8. Client install filebeat:

[root@localhost ~]# rpm -ivh filebeat-6.8.1-x86_64.rpm

9. Configure the filebeat.yml file to collect httpd logs

[root@localhost ~]# cd /etc/filebeat/
备份一份原配置文件，以防改错：
[root@localhost filebeat]# cp filebeat.yml filebeat.yml.bak
[root@localhost filebeat]# vim filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/httpd/access_log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  setup.ilm.enabled: false
  setup.template.name: "filebeat-httpd"
  setup.template.pattern: "filebeat-httpd-*"
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.redis:
  hosts: ["192.168.232.135:6379"]
  key: "filebeat-httpd"
  db: 1
  timeout: 5
  password: 123321

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

10. Install logstash on the server

[root@localhost ELK]# rpm -ivh logstash-6.6.0.rpm

11. Configure logstash file to add httpd index

[root@localhost ELK]# cd /etc/logstash/conf.d/
[root@localhost conf.d]# vim httpd.conf
input {
    
    
        redis {
    
    
                data_type => "list"
                host => "192.168.232.135"
                password => "123321"
                port => "6379"
                db => "1"
                key => "filebeat-httpd"
        }
}

output {
    
    

         elasticsearch {
    
    
                hosts => ["192.168.232.135:9200"]
                index => "redis-httpdlog-%{+YYYY.MM.dd}"




         }
}

12. Open logstash and verify:

1.[root@localhost conf.d]# systemctl start logstash
2.[root@localhost conf.d]# netstat -nltp |grep 9600

13. Kibana:

1.[root@localhost ELK]# rpm -ivh kibana-6.6.2-x86_64.rpm

13.1. Modify the kibana configuration file:

[root@localhost ELK]# vim /etc/kibana/kibana.yml

13.2. Open the kibana service and verify:

[root@localhost ELK]# systemctl start kibana
[root@localhost ELK]# netstat -nltp |grep 5601

14.http://192.168.232.135:5601/Browser to visit the kibana page:




Additional: If the index cannot be found after entering the page, enter the httpd page and refresh the page. The httpd index will come out