Dynamically specify the log file location of logback through environment variables

background

I won’t talk about the basic use of logback. Let’s talk about the background of writing this article. I have built a distributed offline scheduling platform before. To put it simply, it runs scheduled tasks, but it supports many types of tasks, not just Java. For scheduled tasks, each task will generate some logs when running. These logs should be displayed on the page for users to view some information when the task is running and facilitate troubleshooting. Generally, we may think of collecting the logs and saving them. es, and then the page directly queries es, but this will increase dependencies, so I thought of storing the log on the machine, and directly reading this file when the page is accessed.

question

1. Because there are many supported task types, including python, shell, hive, java, etc., and everyone’s development habits are different, there is no way to unify one logging framework.

2. The logs when each task is running should be stored in different directories and different files, not all in the same directory, let alone the same log file.

Solutions

Regarding question 1, we have agreed with the task developer that the logs inside the tasks will be uniformly output to the standard output stream. In this way, when our offline scheduling platform calls up these tasks, we only need to obtain the logs inside the tasks through the standard input stream.

For question 2, after obtaining the internal log of task running, record the log by dynamically specifying the log path of logback, for example: /app/logs/${task}/${date}/info.log, logkback also supports it. Dynamically read the parameter value after -D in the java startup program, so we use the following logback configuration to record the log file:

<?xml version="1.0" encoding="UTF-8" ?>

<configuration>
    　
    <property name="logPath" value="${LOG_HOME}/${task}/logs"/>

    <!--  日志打印的格式 -->
    <appender name="consoleLog" class="ch.qos.logback.core.ConsoleAppender">
        <layout class="ch.qos.logback.classic.PatternLayout">
            <pattern>
                %d{yyyy-MM-dd HH:mm:ss.SSS} %10logger [%thread] : %msg%n
            </pattern>
        </layout>
    </appender>

    <!-- info 状态下的日志   -->
    <appender name="fileInfoLog" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <filter class="ch.qos.logback.classic.filter.LevelFilter">
            <level>ERROR</level>
            <onMatch>DENY</onMatch>
            <onMismatch>ACCEPT</onMismatch>
        </filter>
        <encoder>
            <pattern>
                %d{yyyy-MM-dd HH:mm:ss.SSS} %10logger [%thread] : %msg%n
            </pattern>
        </encoder>
        <!--滚动策略-->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${logPath}/info.%d.log</fileNamePattern>
            <maxHistory>7</maxHistory>
        </rollingPolicy>
    </appender>

    <!--  错误级别的日志文件  -->
    <appender name="fileErrorLog" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>ERROR</level>
        </filter>
        <encoder>
            <pattern>
                %d{yyyy-MM-dd HH:mm:ss.SSS} %10logger [%thread] : %msg%n
            </pattern>
        </encoder>
        <!--滚动策略-->
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${logPath}/error.%d.log</fileNamePattern>
            <maxHistory>7</maxHistory>
        </rollingPolicy>
    </appender>

    <root level="info">
        <appender-ref ref="consoleLog"/>
        <appender-ref ref="fileInfoLog"/>
        <appender-ref ref="fileErrorLog"/>
    </root>

</configuration>

In this way, we can dynamically specify the log storage location of each task through -D at startup. Here is the rendering: