Log from common-log, self4j, log4j to logback, logstash

1. The history of log

log4j -- other logs later log4j--->logback (same author as log4j, improved performance, more flexible configuration)
   |
   V
common-log unified interface, automatic scanning implementation class, has shortcomings, in independent process The class in cannot be scanned
   |
   V
sef4j interface

2. How to print the log:
log.info("Running error{}",e.getMessage());
log.info("Running error",e);
Mainly use the above method When printing the log, if you use %c%L to print the class name and line number, the name of the class displays the name of the superclass, and L displays the line number of the current class.

3. The pit
core-MD of the project actually uses three getLoger methods, including common-logs, sellf4j, and log4j, so some problems were encountered during the investigation.
When multiple log implementations coexist, such as log4j and logback, log implementations are randomly obtained.

Due to historical reasons, the system has two sets of logs, logback and log4j. I tried to close log4j, but found that spring's log is gone. Spring and hibernate use log4j. The exercise automatically finds the configuration of logback. In addition, if logback.xml is not configured in spring, the src/test/resoureces/log_back.xml file is used when maven starts, and the src/main/resource directory is used when publishing.

On production machine A, since multiple projects use public lib, there is always a problem with log printing, which cannot be uploaded to logstash, and it seems that files cannot be generated.
It works fine when using a separate lib on another production machine.

addActivity attribute: whether or not other matching print logs.

      For example, the first logger sets additivity=true, so in addition to writing to file_access, it also writes to stdout.
   

<logger name="com.mindao.app.ea.interceptor.AccessLogInterceptor" level="info" additivity="true">        
        <appender-ref ref="file_access" />
    </logger>

    <logger name="com.mindao" level="info" additivity="false">        
        <appender-ref ref="stdout" />
    </logger>

4. How does log4j connect to logstash

log4j.rootLogger = INFO, console, socket

# for package com.demo.elk, log would be sent to socket appender.
#log4j.logger.com.mindao.elk=info, socket
log4j.logger.com.mindao.elk=info, socket

# appender socket
log4j.appender.socket=org.apache.log4j.net.SocketAppender
log4j.appender.socket.Port=4567
log4j.appender.socket.RemoteHost=120.76.44.235
log4j.appender.socket.layout=org.apache.log4j.PatternLayout
log4j.appender.socket.layout.ConversionPattern=%d [%-5p] [%l] %m%n
log4j.appender.socket.ReconnectionDelay=10000

# appender console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d [%-5p] [%l] %m%n

Configuration of logstash
mkdir config
vi config/log4j_to_es.conf
Enter the following:
# For detail structure of this file# Set: https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html
input {
  # For detail config for log4j as input, # See: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
  log4j {
    mode => "server"
    host => "centos2"
    port => 4567
  }
}
filter {
}
output {
  # For detail config for elasticsearch as output, # See: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
  elasticsearch {
    action => "index"#The operation on ES
    hosts  => "centos2:9200"#ElasticSearch host, can be array.
    index  => "applog"#The index to write data to.
  }
}

5 How logback connects to logstash
https://github.com/logstash/logstash-logback-encoder
Including it in your project

Maven style:
<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>4.7</version>
</dependency>
UDP Appender

To output JSON for LoggingEvents to a syslog/UDP channel, use the LogstashSocketAppender in your logback.xml like this:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <appendername="stash"class="net.logstash.logback.appender.LogstashSocketAppender">
    <host>MyAwesomeSyslogServer</host>
    <!-- port is optional (default value shown) -->
    <port>514</port>
  </appender>
  <rootlevel="all">
    <appender-refref="stash" />
  </root>
</configuration>

Configuration of logstash

input {
  syslog {
    codec => "json"
  }
}