[Python module] logging log module

When getting started with a language, the easiest and most intuitive way to print log information is to use the print() function, and this is only done after practice and testing. When participating in a project, you will definitely use the log module to print and record log information, and Python provides a built-in log module logging, so it is necessary to learn more about it.

1. Basic settings of log options

There are five levels of logging logs, and there are output priorities: critical > error > warning > info > debug. By default, info and debug level logs are not printed. The test is as follows:

After the logging module is introduced, if no option is set, only the warning level and higher logs will be output, and the output format is relatively simple. The default format is [log level: root: log output content].

At this point, we can use the basicConfig() method to make more settings. The parameter options of this method are:

• filename: specifies the name of the output log file;
• filemode: the mode in which the log information is written to the file, the default is a;
• format: format, the options are:
  • %(levelno)s: Print the value of the log level
  • %(asctime)s: the time to print the log
  • %(levelname)s: print the name of the log level
  • %(pathname)s: Print the path of the currently executing program, which is actually sys.argv[0]
  • %(filename)s: Print the name of the currently executing program
  • %(funcName)s: Print the current function of the log
  • %(lineno)d: Print the current line number of the log
  • %(process)d: print process ID
  • %(thread)d: print thread ID
  • %(threadName)s: print thread name
  • %(message)s: print log information
• datefmt: specify the formatted date and time;
• style: Specify the formatting symbol, the default is %;
• level: set the log level;
• stream: Specify the log output stream stream or filename, when specified together with filename, stream will be ignored;
• handlers: specify the logging processor;
• force: When specified as true, any program in the root logger will be ignored.

According to the parameter options of this API, you can customize the configuration log output style, for example:

import logging
from datetime import date, datetime

filename = f'D:\\XXX\\{str(date.today())}.log'
format_option = '%(asctime)s - %(filename)s[line:%(lineno)d] - %(threadName)s - %(levelname)s: %(message)s'
logging.basicConfig(filename=filename,  # 输出到指定log文件
                    filemode='a+',  # 以a+写入
                    format=format_option,
                    datefmt='%Y-%m-%d %H:%M:%S',  # 日期时间格式：2022-11-25 21:59:59
                    level=logging.INFO  # 高于该级别的日志都可以输出显示)
logging.info(f"current datetime {datetime.strftime(datetime.now(), '%Y-%m-%d %H:%M:%S')}")
logging.warning(f"current datetime {datetime.strftime(datetime.now(), '%Y/%m/%d %H:%M:%S')}")

The log output to the 2022-11-25.log file is as follows:

Of course, there are also limitations in the way of using basicConfig() to set the log style, such as: cutting the log and recording the log into multiple files.

2. Set log options based on the processor Handler

Python is also an object-oriented language. Log options can be set through the Handler object. There are two ways to introduce Handler.

<1>. The Handlers imported through the logging module are:

• Handler: is a base class that defines the interface that all handlers should have, and establishes some default behaviors that subclasses can use (or override);
• StreamHandler: output log records to a data stream, such as sys.stdout (standard output stream), sys.stderr (standard error stream) or any file-like object (that is, any object that supports write() and flush() methods);
• FileHandler: output log records to disk files, inherited from StreamHandler;
• NullHandler: It does not perform any formatting or output, it is actually a "no-op" handler for use by library developers.

Here, the most commonly used ones are StreamHandler and FileHandler , which will be described in detail later.

<2>. The Handlers introduced by the logging.handlers module are:

• HTTPHandler: use the GET or POST method to send messages to the HTTP server;
• QueueHandler: sends messages to queues, such as those implemented in the queue or multiprocessing modules;
• SMTPHandler: Send a message to a specified email address;
• MemoryHandler: Send the message to the buffer in memory, as long as certain conditions are met, the buffer will be refreshed;
• SocketHandler: send messages to TCP/IP sockets (since 3.4, Unix domain sockets are also supported);
• DatagramHandler: Send messages to UDP sockets (since 3.4, Unix domain sockets are also supported);
• RotatingFileHandler: Send messages to disk files, support maximum log file size and log file cutting;
• TimedRotatingFileHandler: Send messages to disk files, cut log files at specific time intervals;
• BaseRotatingHandler: It is the base class of the processor that rotates the log file. It should not be instantiated directly. Instead, use RotatingFileHandler or TimedRotatingFileHandler instead;
• NTEventLogHandler: Sends messages to the Windows NT/2000/XP event log;
• SysLogHandler: sends messages to a Unix syslog daemon, possibly on a remote computer;
• WatchedFileHandler: will watch the file they want to write to the log. If the file is changed, it is closed and reopened with the filename, this handler is only useful on Unix-like systems, Windows does not support the underlying mechanism it depends on.

The logging module provides such a rich log processor, we can choose the corresponding Handler according to different usage scenarios. For example, RotatingFileHandler is the most suitable choice for log cutting ; QueueHandler is more suitable to solve the problem of log record blocking caused by multiple processes ; ......, will be described in detail later.

2.1 StreamHandler

Stream log record processor, supports custom log style, will output the log content in the form of stream, and output to sys.stderr by default, or specify a specific steam instance in the constructor. The demonstration is as follows:

# 获取日志记录器
logger = logging.getLogger()
# 设置全局日志输出级别
logger.setLevel(logging.INFO)
# 创建stream日志记录处理器，默认使用sys.stderr
streamHandler = logging.StreamHandler()
# 定义日志输出风格（格式器）
format_option = '%(asctime)s - %(filename)s[line:%(lineno)d] - %(threadName)s - %(levelname)s: %(message)s'
streamHandler.setFormatter(logging.Formatter(format_option))
# 将日志记录处理器加入日志对象
logger.addHandler(streamHandler)
logger.info(f"current datetime {datetime.strftime(datetime.now(), '%Y-%m-%d %H:%M:%S')}")
logger.info(f"current datetime {datetime.strftime(datetime.now(), '%Y/%m/%d %H:%M:%S')}")

The effect is as follows:

2.2 FileHandler

The file logging processor also supports custom log styles, and outputs the contents of the log to disk files for storage. The constructor supports log output file name, writing mode, encoding method, whether to delay recording delay and other option settings. The demonstration is as follows:

logger = logging.getLogger() # 获取日志记录器
logger.setLevel(logging.INFO) # 设置全局日志输出级别
# 创建文件日志记录处理器，并指定一些设置选项
fileHandler = logging.FileHandler(filename=f'D:\\XXX\\{str(date.today())}_fileHandler.log',
                                  mode='a+', encoding='utf-8', delay=False)
# 定义日志输出风格（格式器）
format_option = '%(asctime)s - %(filename)s[line:%(lineno)d] - %(threadName)s - %(levelname)s: %(message)s'
fileHandler.setFormatter(logging.Formatter(format_option))
# 定义日志输出级别（这个不起作用，需要使用全局定义）
# fileHandler.setLevel(logging.DEBUG)
# 将日志记录处理器加入日志对象
logger.addHandler(fileHandler)
logger.info(f"current datetime {datetime.strftime(datetime.now(), '%Y-%m-%d %H:%M:%S')}")
logger.info(f"current datetime {datetime.strftime(datetime.now(), '%Y/%m/%d %H:%M:%S')}")

The effect is as follows:

Please note that if the delay parameter is True, the file opening will be postponed until the first call to the emit() method, and the emit() method needs to introduce logging.LogRecord(), which will not be studied further here.

2.3 RotatingFileHandler implements log cutting

The parent class of RotatingFileHandler is BaseRotatingHandler, and the parent class of BaseRotatingHandler is logging.FileHandler, so RotatingFileHandler essentially operates disk files through FileHandler. Realize log segmentation, the demonstration is as follows:

logger = logging.getLogger()  # 获取日志记录器
logger.setLevel(logging.DEBUG)  # 设置全局日志级别
# 基本设置：log文件路径、写入模式、切割后每个文件大小（5k）、文件数量(现分在4个)、编码方式、是否延迟记录日志
rotatingFileHandler = RotatingFileHandler('D:\\XXX\\test_rotatingFileHandler.log',
                                          mode='a', maxBytes=5 * 1024,
                                          backupCount=4, encoding='utf-8', delay=False)
# 定义日志输出风格（格式器）
format_option = '%(asctime)s - %(filename)s[line:%(lineno)d] - %(threadName)s - %(levelname)s: %(message)s'
rotatingFileHandler.setFormatter(logging.Formatter(format_option))
# 将日志记录处理器加入日志对象
logger.addHandler(rotatingFileHandler)
for i in range(1, 221):
    logger.info(f"你可曾记得，这已经是我第{i}次说了，哎!")

The cutting effect is as follows:

2.4 QueueHandler  solves the blocking problem of logging

In web applications, there is often the problem of multiple processes blocking log records, we can use the QueueHandler + QueueListener solution to solve it.

In practice, some key threads that require performance can only connect a QueueHandler to the log object. The log object only needs to be simply written to the queue. Here, the queue can be set with a large enough capacity, or the upper limit of the capacity can not be set during initialization.

The advantage of QueueListener is that you can use the same instance to serve multiple QueueHandlers. QueueListener requires a queue and one or more handlers to be passed in, and an internal thread is started to listen to the LogRecord queue sent by QueueHandlers (or other LogRecords sources), and LogRecords will be removed from the queue and passed to the handler for processing. The demonstration is as follows:

3. Set log options based on configuration files

We know that in Python 3.2 and later versions, the log configuration can be loaded from the dictionary, which means that the log configuration can be loaded through JSON or YAML files. Take the .yml file as an example.

The first step is to customize the zdy_log.yml file and configure it as follows:

version: 1
disable_existing_loggers: False
# 格式器配置
formatters:
  simple:
    format: '%(asctime)s - %(filename)s[line:%(lineno)d] - %(threadName)s - %(levelname)s: %(message)s'
# 处理器配置
handlers:
  console: # 控制台
    class: logging.StreamHandler
    level: DEBUG
    formatter: simple
    stream: ext://sys.stdout
  info_file_handler: # info级别的日志文件配置
    class: logging.handlers.RotatingFileHandler
    level: INFO
    formatter: simple
    filename: info.log
    maxBytes: 10485760
    backupCount: 20
    encoding: utf8
  error_file_handler: # errors级别的日志文件配置
    class: logging.handlers.RotatingFileHandler
    level: ERROR
    formatter: simple
    filename: errors.log
    maxBytes: 10485760
    backupCount: 20
    encoding: utf8
# 根日志
loggers:
  my_module:
    level: ERROR
    handlers: [ info_file_handler ]
    propagate: no
root:
  level: INFO
  handlers: [ console,info_file_handler,error_file_handler ]

The second step is to encode the code to realize the logging function, as follows:

import logging.config, yaml
from datetime import datetime

# 加载日志文件
with open('zdy_log.yml', 'r', encoding='utf-8') as f:
    logging.config.dictConfig(yaml.load(f))
# 输出日志
logging.info(f"current datetime {datetime.strftime(datetime.now(), '%Y-%m-%d %H:%M:%S')}")
logging.info(f"current datetime {datetime.strftime(datetime.now(), '%Y/%m/%d %H:%M:%S')}")

The third step is to run and view the results:

Of course, you can also choose to use the .conf file format to configure log options, but it is recommended to use dictionary configuration here.

4. Others

4.1 Summary of important components of the logging module

The logging library adopts a modular approach and provides several important components: loggers, processors, filters, and formatters. The functions are described as follows:

• Loggers: Expose the interface directly used by the application code, responsible for configuring and sending log messages to the corresponding processor.
• Processor Handlers: Responsible for sending log records (created by loggers) to specified targets.
• Formatter Formatter: Responsible for specifying the style of log records in the final output.
• Filters: Responsible for providing finer-grained functionality for determining which log records to output.

Log event information is passed between loggers, processors, formatters, and filters in a LogRecord instance.

Usually use the getLogger(name=None) method to create a logger object. When the name is None, the default is the root logger; when the name is set to a specific name, then search for the root logger by this name. The most common configuration methods for loggers are: setLevel(), addFilter() and removeFilter(), addHandler() and removeHandler(), and log methods of different levels (critical()/error()/… /debug()), it should be noted that this log level configuration is global.

After the logger object is created, add 0 or more handler objects to itself through addHandler(). According to the requirements of different processing scenarios, the processor will dispatch log messages to specific Handlers. In addition, the processor also provides some configuration methods: setLevel(), setFormatter(), addFilter() and removeFilter(), among which setLevel() will only take effect inside the processor.

Formatter Formatter, which can easily customize the output log content. Important parameters include log content formatting, date time formatting, and formatting style. By default, the correctness of the style is verified. The constructor is as follows:

logging.Formatter.__init__(self, fmt=None, datefmt=None, style='%', validate=True)

A filter is a finer-grained control of log output content, which can be used by loggers and processors to implement more complex filtering operations than provided by level. The basic filter class only allows lower than a specific level in the logger hierarchy event. Usually use the Filter(name='') method to create a filter object. If name is initialized with an empty string, all log events will pass; if the filter is initialized with 'ab', 'ab', 'abc', Log events for 'abcd' are passed, like 'acb', 'bc' etc are not allowed. There are three commonly used methods for filters: addFilter(filter), removeFilter(filter), filter(record), as follows:

class Filter(object):
    def __init__(self, name=''):
        .....
    def filter(self, record):
        .....

class Filterer(object):
    def addFilter(self, filter):
        .....
    def removeFilter(self, filter):
        .....

4.2 Exception logging

An important purpose of logging is to quickly locate and solve problems in the program, and the error level logs provided by the logging module can print exception information through the exc_info parameter option, as follows:

try:
    doSomeThings()
except Exception as e:
    logging.error('error！！！！', exc_info=True)

Of course, in order to more conveniently print or save the running exceptions in the program for exception analysis, it is recommended to use the module traceback to track exception information  . The common APIs are:

• print_exc(): directly print out the exception information on the terminal;
• format_exc(): returns a string of exception information, which can be used to record the information into a file;

Save the exception information to a file, as follows:

traceback.print_exc(file=open('traceback_ERROR.txt','w+'))

The traceback module is relatively flexible in tracking exception information and is recommended.

[Life is short, learn Python! The Python module series will be continuously updated and documented...]