logstash collects tomcat-json logs

Architecture diagram

This time, I didn't use redis, and I used logstash directly (in the production environment, it is recommended to build and use filebeat) to send the collected data to es. This disadvantage, if the equivalent is large, will lead to es pressure.

1. Installation

Omitted jdk, es, kibana. .. ..

#install tomcat
[root@localhost ~]# wget https://archive.apache.org/dist/tomcat/tomcat-8/v8.0.38/bin/apache-tomcat-8.0.38.tar.gz
[root@localhost ~]# tar xvf apache-tomcat-8.0.38.tar.gz
[root@localhost ~]# mv apache-tomcat-8.0.38 /data/application/tomcat

2. Start tomcat

# Come to a simple interface
[root@localhost ~]#mkdir /data/application/tomcat/webapps/webdir
[root@localhost ~]#echo "Hello,Tomcat" > /data/application/tomcat/webapps/webdir/index.html

#Start tomcat
[root@localhost webapps]# /data/application/tomcat/bin/startup.sh 
Using CATALINA_BASE:   /data/application/tomcat
Using CATALINA_HOME:   /data/application/tomcat
Using CATALINA_TMPDIR: /data/application/tomcat/temp
Using JRE_HOME:        /data/application/jdk1.8.0_91/
Using CLASSPATH:       /data/application/tomcat/bin/bootstrap.jar:/data/application/tomcat/bin/tomcat-juli.jar

to visit

3. Modify the tomcat log to json format

# Modify according to your needs

[root@localhost tomcat]# vim /data/application/tomcat/conf/server.xml

<Valve className="org.apache.catalina.valves.AccessLogValve"

directory="logs"

prefix="tomcat_access_log" suffix=".log"

pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","method":"%r","status":"","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>

#stop tomcat
[root@localhost tomcat]# ./bin/shutdown.sh
#Start tomcat
[root@localhost tomcat]# ./bin/startup.sh

4. View the tomcat log format

#Verify json log

https://www.json.cn

4. Test the logstash configuration file

[root@localhost server_config]# /data/application/logstash-5.2.0/bin/logstash -f tomcat.conf -t

#Start logstash service
[root@localhost server_config]# cat tomcat.conf 
input {
  file {
    path => "/data/application/tomcat/logs/tomcat_access_log*.log"
    start_position => "end"                    
    type => "tomcat-log"                
    stat_interval => "3"
  }
}
output {
  if [type] == "tomcat-log" {      
    elasticsearch {
      hosts => ["192.168.201.135:9200"]    
      index => "tomcat-access-%{+YYYY.MM.dd}" 
      codec => "json"            
    }}
}
[root@localhost server_config]# /data/application/logstash-5.2.0/bin/logstash -f tomcat.conf

5. Check es status

6, kibana effect

At this point, the tomcat collection log is over

Notice:

In the output of logstash, the effect on different machines entering the same cluster is the same, because they are a cluster. Of course, you can also specify the output to a different cluster.

2. It can be seen from the kibana Tomcat page that the Tomcat log has become json format, which is the result of specifying the parameter codec="json" in logstash, otherwise kibana is a mess here

3. This pit is about time synchronization. If kibana is not synchronized with your logstash time, the logs on kibana may not be displayed. This requires you to develop the habit of standardization. Remember

logstash collects tomcat-json logs

Guess you like