logger: A Python artifact for managing logs

I want to open a new project recently, and I need a logger to manage the log. I will share it today. If you like it, remember to like, follow, and favorite.

[Note] An exchange and mutual aid group is provided at the end of the article

import logging
ori_logger = logging.getLogger('custom_logger')
ori_logger.setLevel(logging.INFO)
ori_logger.addHandler(logging.StreamHandler())
ori_logger.info('learn log')
# learn log

Emmm... The log feels ugly, NOT GOOD. Why don't you try mmcv's get_logger, it is fully functional and can be done in one step. Na~ take the link:

https://github.com/open-mmlab/mmcv/blob/master/mmcv/utils/logging.py

terminal log

mmcv_logger = get_logger('mmcv_logger', log_file='a.log')
mmcv_logger.info('learn log')
# 2022-03-24 10:38:35,998 - mmcv_logger - INFO - learn log

log file

2022-03-24 10:37:25,832 - mmcv_logger - INFO - learn log

So convenient! Timestamp, log name, log level, and even save log files for you, easily done with one line of code.

How does mmcv's get_logger configure such a convenient logger? You only need to configure the three treasures of logger: log level, formatter and log handler, so that logger will obey you.

logging log level ** ✦ **

The log levels of the logging module are as follows:

CRITICAL = 50        
FATAL = CRITICAL
ERROR = 40           
WARNING = 30        
WARN = WARNING
INFO = 20            
DEBUG = 10           
NOTSET = 0          

Logging.Logger (the logger in this article is an instance) and the level attribute of logging.Handler represent the log level of the corresponding instance. When we call logger.info(msg) and logger.warning(msg), the output message also has a log level. Only when the log level of the message is greater than or equal to logger.level, the log may be output. Take the logging.root we mentioned earlier as an example:

import logging

logger = logging.root
print(logger.level)  # 30， warning。root 日志等级为 WARNING (30)
logger.info("info msg")  # 日志等级为 INFO (20)，小于 WARNING，无消息
logger.warning("warn msg")  # 日志等级为 WARNING (30)，输出消息 warn msg
logger.error("erro msg")  # 日志等级为 ERROR (40)，输出消息 erro msg
logger.setLevel(logging.ERROR) # 设置日志等级为 ERROR (40)
print(logger.level)  # 40， ERROR
logger.warning("warn msg")  # 无消息
logger.error("erro msg")  # erro msg

The log level of logging.root is WARNING, so INFO level messages cannot be output. When we adjust the log level to ERROR, it will not be able to output WARNING level messages. But the strange thing is that we mentioned in the first issue that logging.root is an embryo by default, and it does not have the ability to output logs. **Why can you still output warning and error level logs here? **Leave a question here, and the answer will be given in the Handler section.

logging 之 Handler ** ✦ **

Since the logging module outputs logs through Handler, this section will first introduce the two major Handlers of the logging module: streamHandler and fileHandler.

StreamHandler

output information to the terminal

The logger configured with StreamHandler can output log information to the terminal in the same way as print. The example is as follows:

logger = logging.root
# 创建 streamHandler
stream_handler = logging.StreamHandler()
# 设置日志等级
logger.setLevel(logging.INFO)
logger.info("learn logging")  # 没有配置 Handler，终端不会输出日志
# 为 root logger配置 Handler
logger.addHandler(stream_handler)
logger.info("learn logging")  #  配置 Handler，输出日志

Output log information to file

When instantiating a StreamHandler, the log can be stored to a file if the stream is configured to write to the target file.

f = open('output.txt', 'w')
logger = logging.root
logger.setLevel(logging.INFO)
stream_handler = logging.StreamHandler(stream=f)
logger.addHandler(stream_handler)
logger.warning("learn logging")  # 此时 learn logging 会被写入到 output.txt 中
f.close()

FileHandler

FileHandler inherits from StreamHandler and can specify the encoding format for writing files. Compared with StreamHandler, it is more flexible and easy to use:

logger = logging.root
logger.setLevel(logging.INFO)
# 设置输出文件和编码方式
file_handler = logging.FileHandler('output.txt', encoding='utf-8')
logger.addHandler(file_handler)
logger.info("WARNING")  # 此时 learn logging 会被写入到 output.txt 中

Handler's log level

Handlers also have their own log levels. For a message to be output through the Handler, the log level of the message needs to be greater than the log level of the logger and the log level of the handler.

logger = logging.root
logger.setLevel(logging.DEBUG)  # logger 的日志等级为 DEBUG
handler = logging.StreamHandler()
handler.setLevel(logging.INFO)  # handler 的日志等级为 INFO
logger.addHandler(handler)
logger.debug('learn logging')  # 不满足 handler，无法被输出
logger.info('learn logging')  # 同时满足 logger 含 handler，正常输出

The behind-the-scenes operation of logging

Can log output without handler?

First of all, the conclusion is that the logger itself does not have the ability to output messages without a handler. The first example of streamHandler has already explained this problem. But why logger.warning(msg) and logger.error(msg) can output logs without configuring handler? This is actually the protection mechanism of the logging module . For messages of warning and error levels, if the log level of the message is greater than the log level of the logger, and the logger is not configured with any handler, the built-in streamHandler of the logging module will be called to output the information.

In order to prove this logic, we configure a fileHandler for the logger. At this time, logger.warning(msg) will not output logs in the terminal.

logger = logging.root
logger.warning("learn logging")  # 输出日志到终端
file_handler = logging.FileHandler('output.txt', encoding='utf-8')
logger.addHandler(file_handler)
logger.warning("learn logging")  # 不会输出日志到终端

Crazy logging.xxx

When you call logging.info and logging.warning happily, you think you just output a log, but in fact, you may have secretly configured streamHandler for logging.root.

logging.info('learn logging')
print(logging.root.handlers)
# [<StreamHandler <stderr> (NOTSET)>]

You may be thinking, what's the point of adding a streamHandler. Remember the horror of being dominated by multiple logs? Why am I getting multiple logs all of a sudden using pytorch 1.10? Tracing the source is because the DistributedDataParallel module of pytorch 1.10 calls logging.info (https://github.com/pytorch/pytorch/blob/71f889c7d265b9636b93ede9d651c0a9c4bee191/torch/nn/parallel/distributed.py#L874) during the forward process, and then There is one more streamHandler in logging.root (this process happens after mmcv.get_logger, the log level of the handler is not set to ERROR), which eventually leads to the occurrence of multiple logs.

logging 之 Formatter

If configuring a handler is equivalent to teaching a logger to speak, then configuring a formatter for a handler is equivalent to teaching a logger to speak "elegantly".

Add a "subject" to the log

We can configure the formatter so that the log output by the logger has its own logger name.

logger = logging.root
handler = logging.StreamHandler()
handler.setFormatter(Formatter('%(name)s - %(message)s'))
logger.addHandler(handler)
logger.setLevel(logging.INFO)
logger.info('learn logging')
# root - learn logging

In this way, the output log will have its own "root" log name.

add time to log

Primary school Chinese teachers must have taught that time, place, characters are the top three of the seven elements, and time is at the top of the list. Therefore, we can configure the formatter to allow the log to carry time information.

logger = logging.root
handler = logging.StreamHandler()
handler.setFormatter(Formatter('%(asctime)s - %(name)s - %(message)s'))
logger.addHandler(handler)
logger.setLevel(logging.INFO)
logger.info('learn logging')
# 2022-03-22 01:36:42,900 - root - learn logging

add a rating to the log

The log level can highlight the importance of the log. For example, we will pay special attention to the log with the ERROR field. Therefore, we can let the log carry the level information through the formatter.

logger = logging.root
handler = logging.StreamHandler()
handler.setFormatter(Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s'))
logger.addHandler(handler)
logger.setLevel(logging.INFO)
logger.info('learn logging')
# 2022-03-22 01:46:26,667 - root - INFO - learn logging

Hey, it smells like that. Is it the same as the logs of the OpenMMLab series? In fact, the process of mmcv configuring logger is similar, and there is a more comprehensive handler configuration logic: mmcv/logging.py at master · open-mmlab/mmcv (github.com), you can refer to it for reference.

summary

After understanding the concepts of loglevel, handler and formatter, we can also write a simplified version of the configuration logger function ourselves.

def custom_get_logger(name, out_file, log_level):
    # 设置日志名
    logger = logging.getLogger(name)
    # 设置日志登记
    logger.setLevel(log_level)
    # 设置格式
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s'
                                  ' - %(message)s')
    # 配置输出日志到终端的 Handler
    stream_handler = logging.StreamHandler()
    stream_handler.setFormatter(formatter)
    # 配置保存日志到文件的 Handler
    file_handler = logging.FileHandler(out_file)
    file_handler.setFormatter(formatter)
    # 添加 Handler
    logger.addHandler(file_handler)
    logger.addHandler(stream_handler)
    return logger

custom_logger = custom_get_logger('custom_logger', 'a.log', 'INFO')
custom_logger.info('learn log')
# 2022-03-24 11:18:52,120 - custom_logger - INFO - learn log

The logger obtained through the custom_get_logger interface has a good-looking log format and can also be stored in a local backup, which is basically aligned with mmcv. However, the logger configured in this way still has some hidden dangers, such as the multiple logs mentioned in the previous issue. To solve these problems, we might as well go back and look at the code of mmcv.get_logger. I believe that after this period of study, a lot of logic will become easier to understand.

So far, the basic functions of the logging module have been introduced, but there are still some implicit dark-box operations in logging. If you want to fully understand the logging module, please look forward to the content of the next issue~

recommended article

• Li Hongyi's "Machine Learning" Mandarin Course (2022) is here

• Someone made a Chinese version of Mr. Wu Enda's machine learning and deep learning

• I'm addicted, and recently I gave the company a big visual screen (with source code)

• So elegant, 4 Python automatic data analysis artifacts are really fragrant

• After combing for more than half a month, I carefully prepared 17 knowledge mind maps. This time I want to explain statistics clearly

• Year-end summary: 20 visual large-screen templates, directly apply Zhenxiang (source code at the end of the article)

Technology Exchange

Welcome to reprint, collect, like and support!

At present, a technical exchange group has been opened, and the group has more than 2,000 members . The best way to remark when adding is: source + interest direction, which is convenient to find like-minded friends

• Method 1. Send the following picture to WeChat, long press to identify, and reply in the background: add group;
• Method ②, add micro-signal: dkl88191 , note: from CSDN
• Method ③, WeChat search public account: Python learning and data mining , background reply: add group