logging module

Introduction to logging module

Many programs have the need to record logs, and the information contained in the log is the normal program access log, and may also output information such as errors and warnings.
Python's logging module provides a standard log interface, through which you can store various logs. Logs in different formats can be divided into
five levels: debug(), info(), warning(), error() and critical().

Basic usage

import logging

logging.warning("user [alex] attempted wrong password more than 3 times")
logging.critical("server is down")

WARNING:root:user [alex] attempted wrong password more than 3 times
CRITICAL:root:server is down

logging log level

• FATAL: fatal error
• CRITICAL: Very bad things like running out of memory, empty disk space, rarely used in general
• ERROR: When an error occurs, such as an IO operation failure or a connection problem
• WARNING: When a very important event occurs, but it is not an error, such as a wrong user login password
• INFO: Handle daily transactions such as requests or state changes
• DEBUG: Use the DEBUG level during debugging, such as the intermediate state of each loop in the algorithm

write log to file

import logging

logging.basicConfig(filename='example.log',level=logging.INFO)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')

level=loggin.INFOThe meaning of the following sentence is to set the log record level to INFO, that is, only logs higher INFOthan or higher INFOthan the log will be recorded in the file. In this example, the first log will not be It is recorded. If you want to record the debug log, you can change the log level DEBUG.

Custom log format

I feel that the above log format forgot to add the time, the log does not know how to do it, so let's add it below!

import logging
logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')
logging.warning('is when this event was logged.')

# 输出
12/12/2010 11:46:36 AM is when this event was logged.

• log output format parameters

  • %(levelno)s: print the value of the log level
  • %(levelname)s: print the name of the log level
  • %(pathname)s: Print the path of the currently executing program, which is actually sys.argv[0]
  • %(filename)s: print the name of the currently executing program
  • %(funcName)s: the current function that prints the log
  • %(lineno)d: print the current line number of the log
  • %(asctime)s: the time to print the log
  • %(thread)d: print thread ID
  • %(threadName)s: print thread name
  • %(process)d: print process ID
  • %(message)s: print log information

Parameters of logging.basicConfig function

• filename: Specifies the log file name
• filemode: the same as the file function, specifies the opening mode of the log file, 'w' or 'a'
• format: specify the format and content of the output, format can output a lot of useful information
• datefmt: specify the time format, same as time.strftime()
• level: set the log level, the default is logging.WARNNING
• stream: specify the output stream of the log, you can specify the output to sys.stderr, sys.stdout or a file, the default output is to sys.stderr, when stream and filename are specified at the same time, stream is ignored

Log output to both screen and file

In order to print the log to the screen and the file log at the same time, you need to understand a little complicated knowledge. Python logging using the logging module involves four main classes

• logger provides an interface that applications can use directly;
• The handler sends the log records (created by the logger) to the appropriate destination output;
• filter provides a granularity device to decide which log records to output;
• The formatter determines the final output format of the logging.

The main function of each component

logger

Every program gets a Logger before outputting information. Logger usually corresponds to the module name of the program. For example, the graphical interface module of a chat tool can get its Logger like this:

LOG=logging.getLogger(”chat.gui”)
而核心模块可以这样：

LOG=logging.getLogger(”chat.kernel”)
还可以绑定handler和filters

Logger.setLevel(lel)  # 指定最低的日志级别，低于level的级别将被忽略。debug是最低的内置级别，critical为最高
Logger.addFilter(filt)、Logger.removeFilter(filt)  # 添加或删除指定的filter
Logger.addHandler(hdlr)、Logger.removeHandler(hdlr)  # 增加或删除指定的handler
Logger.debug()、Logger.info()、Logger.warning()、Logger.error()、Logger.critical()  # 可以设置的日志级别

handler

The handler object is responsible for sending the relevant information to the specified destination. Python's logging system has a variety of Handlers that can be used. Some Handlers can output information to the console, some Handlers can output information to files, and some Handlers can send information to the network. If you think it is not enough, you can also write your own Handler. Multiple multiple handlers can be added through the addHandler() method

Handler.setLevel(lel)  # 指定被处理的信息级别，低于lel级别的信息将被忽略
Handler.setFormatter()  # 给这个handler选择一个格式
Handler.addFilter(filt)、Handler.removeFilter(filt)  # 新增或删除一个filter对象

Multiple Handlers can be attached to each Logger. Next, we will introduce some commonly used Handlers:

• logging.StreamHandler Use this Handler to output information to any file object like sys.stdout or sys.stderr.

• logging.FileHandler and StreamHandlersimilar, are used to output logging information to a file. But FileHandler will open the file for you

• logging.handlers.RotatingFileHandler

  • This Handler is similar to the above FileHandler, but it can manage file size. When the file reaches a certain size, it will automatically rename the current log file, and then create a new log file with the same name to continue outputting. For example, the log file is chat.log. When chat.log reaches the specified size, RotatingFileHandlerthe file is automatically renamed to chat.log.1. However, if chat.log.1 already exists, it will first rename chat.log.1 to chat.log.2. Finally, recreate chat.log and continue to output log information. Its function is:

RotatingFileHandler( filename[, mode[, maxBytes[, backupCount]]])

where filenameand the modetwo parameters are FileHandlerthe same as sum.
maxBytes is used to specify the maximum file size of the log file. If maxBytes is 0, which means that the log file can be infinitely large, the renaming process described above will not occur.
backupCount is used to specify the number of backup files to keep. For example, if 2 is specified, when the renaming process described above occurs, the original chat.log.2 will not be renamed, but will be deleted.

• logging.handlers.TimedRotatingFileHandler

  • This Handler RotatingFileHandleris similar, however, it does not decide when to recreate the log file by judging the file size, but automatically creates a new log file at a certain interval. The renaming process is RotatingFileHandlersimilar, but instead of appending a number, the new file is the current time. Its function is:

    TimedRotatingFileHandler( filename [,when [,interval [,backupCount]]])

where filenameparameter and backupCountparameter sum RotatingFileHandlerhave the same meaning.

intervalis the time interval.

whenThe parameter is a string. The unit representing the time interval, which is not case-sensitive. It has the following values:
S seconds
M minutes
H hours
D days
W every week (interval==0 represents Monday)
midnight every morning

formatter component

The formatter of the log is an independent component that can be combined with the handler

fh = logging.FileHandler("access.log")
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

fh.setFormatter(formatter) #把formmater绑定到fh上

filter component

If you want to filter the log content, you can customize a filter

class IgnoreBackupLogFilter(logging.Filter):
    """忽略带db backup 的日志"""
    def filter(self, record): #固定写法
        return   "db backup" not in record.getMessage()

# 注意filter函数会返加True or False，logger根据此值决定是否输出此日志

Then add this filter to the logger

logger.addFilter(IgnoreBackupLogFilter())
# 下面的日志就会把符合filter条件的过滤掉

logger.debug("test ....")
logger.info("test info ....")
logger.warning("start to run db backup job ....")
logger.error("test error ....")

A complete example of output to screen, file, with filter at the same time

import logging


class IgnoreBackupLogFilter(logging.Filter):
    """忽略带db backup 的日志"""
    def filter(self, record): # 固定写法
        return   "db backup" not in record.getMessage()


# console handler
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
# file handler
fh = logging.FileHandler('mysql.log')
# fh.setLevel(logging.WARNING)

# formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
#bind formatter to ch
ch.setFormatter(formatter)
fh.setFormatter(formatter)

logger = logging.getLogger("Mysql")
logger.setLevel(logging.DEBUG) #logger 优先级高于其它输出途径的

# add handler   to logger instance
logger.addHandler(ch)
logger.addHandler(fh)

# add filter
logger.addFilter(IgnoreBackupLogFilter())

logger.debug("test ....")
logger.info("test info ....")
logger.warning("start to run db backup job ....")
logger.error("test error ....")

Example of automatic file truncation

import logging

from logging import handlers

logger = logging.getLogger(__name__)

log_file = "timelog.log"

# fh = handlers.RotatingFileHandler(filename=log_file,maxBytes=10,backupCount=3)
fh = handlers.TimedRotatingFileHandler(filename=log_file,when="S",interval=5,backupCount=3)

formatter = logging.Formatter('%(asctime)s %(module)s:%(lineno)d %(message)s')

fh.setFormatter(formatter)

logger.addHandler(fh)

logger.warning("test1")
logger.warning("test12")
logger.warning("test13")
logger.warning("test14")