[Reserved] python_logging module

 

Original: https: //www.cnblogs.com/liujiacai/p/7804848.html

1 logging module Introduction

Python logging module is built standard modules, mainly for outputting the operation log, the log output level can be set, save the log path, the log file rollback like; Print compared, has the following advantages:

1. You can set different log levels, output in the release version only important information without having to show a lot of debugging information;
2. All print information is output to the standard output, seriously affecting other developers view data from the standard output; logging can be determined by the developers to output information to where and how the output;

2 logging module

2.1 Basic use

The logging basic settings, and then outputs the log in the console,

import logging
logging.basicConfig(level = logging.INFO,format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

logger.info("Start print log")
logger.debug("Do something")
logger.warning("Something maybe fail.")
logger.info("Finish")

Run, the console output,

2016-10-09 19:11:19,434 - __main__ - INFO - Start print log 2016-10-09 19:11:19,434 - __main__ - WARNING - Something maybe fail. 2016-10-09 19:11:19,434 - __main__ - INFO - Finish

Many message logging level may be selected, such as debug, info, warning, error, and critical. By assigning different levels logger or handler, the developer can only output error messages to a specific log file, or just record debugging information when debugging.

For example, we will change the logger level DEBUG, and then look at the output,

logging.basicConfig(level = logging.DEBUG,format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s')

Console output, can be found, the output of the debug information.

2016-10-09 19:12:08,289 - __main__ - INFO - Start print log 2016-10-09 19:12:08,289 - __main__ - DEBUG - Do something 2016-10-09 19:12:08,289 - __main__ - WARNING - Something maybe fail. 2016-10-09 19:12:08,289 - __main__ - INFO - Finish

logging.basicConfig function of the parameters:

filename: Specify the log file name;

The same meaning and function file, the log file specified open mode, 'w' or 'a';: filemode

format: Specifies the format and content of output, format can output a lot of useful information,

参数：作用

%(levelno)s：打印日志级别的数值
%(levelname)s：打印日志级别的名称 %(pathname)s：打印当前执行程序的路径，其实就是sys.argv[0] %(filename)s：打印当前执行程序名 %(funcName)s：打印日志的当前函数 %(lineno)d：打印日志的当前行号 %(asctime)s：打印日志的时间 %(thread)d：打印线程ID %(threadName)s：打印线程名称 %(process)d：打印进程ID %(message)s：打印日志信息

datefmt: Specifies the time format, with The time.strftime ();

level: set the log level, the default is logging.WARNNING;

stream: Specifies the output stream of the log, can specify the output to sys.stderr, sys.stdout or file, the default output to sys.stderr, and the filename specified when the stream at the same time, are ignored stream;

2.2 The logs are written to a file

2.2.1 written to the log file

Set logging, create a FileHandler, and set the output format of the message, add it to the logger, then writes the log to the specified file,

import logging
logger = logging.getLogger(__name__)
logger.setLevel(level = logging.INFO)
handler = logging.FileHandler("log.txt")
handler.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)

logger.info("Start print log")
logger.debug("Do something")
logger.warning("Something maybe fail.")
logger.info("Finish")

log.txt log data,

2016-10-09 19:01:13,263 - __main__ - INFO - Start print log 2016-10-09 19:01:13,263 - __main__ - WARNING - Something maybe fail. 2016-10-09 19:01:13,263 - __main__ - INFO - Finish

2.2.2 At the same time the log output to the screen and log files

The StreamHandler logger added, the log may be output to the screen,

import logging
logger = logging.getLogger(__name__)
logger.setLevel(level = logging.INFO)
handler = logging.FileHandler("log.txt")
handler.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)

console = logging.StreamHandler()
console.setLevel(logging.INFO)

logger.addHandler(handler)
logger.addHandler(console)

logger.info("Start print log")
logger.debug("Do something")
logger.warning("Something maybe fail.")
logger.info("Finish")

Log.txt file and can be seen in the console,

2016-10-09 19:20:46,553 - __main__ - INFO - Start print log 2016-10-09 19:20:46,553 - __main__ - WARNING - Something maybe fail. 2016-10-09 19:20:46,553 - __main__ - INFO - Finish

Can be found, the logging main object of the logging process, other ways are added to it by addHandler, handler logging included have the following main,

handler名称：位置；作用

StreamHandler：logging.StreamHandler；日志输出到流，可以是sys.stderr，sys.stdout或者文件 FileHandler：logging.FileHandler；日志输出到文件 BaseRotatingHandler：logging.handlers.BaseRotatingHandler；基本的日志回滚方式 RotatingHandler：logging.handlers.RotatingHandler；日志回滚方式，支持日志文件最大数量和日志文件回滚 TimeRotatingHandler：logging.handlers.TimeRotatingHandler；日志回滚方式，在一定时间区域内回滚日志文件 SocketHandler：logging.handlers.SocketHandler；远程输出日志到TCP/IP sockets DatagramHandler：logging.handlers.DatagramHandler；远程输出日志到UDP sockets SMTPHandler：logging.handlers.SMTPHandler；远程输出日志到邮件地址 SysLogHandler：logging.handlers.SysLogHandler；日志输出到syslog NTEventLogHandler：logging.handlers.NTEventLogHandler；远程输出日志到Windows NT/2000/XP的事件日志 MemoryHandler：logging.handlers.MemoryHandler；日志输出到内存中的指定buffer HTTPHandler：logging.handlers.HTTPHandler；通过"GET"或者"POST"远程输出到HTTP服务器

2.2.3 log roll

Use RotatingFileHandler, log roll can be achieved,

import logging
from logging.handlers import RotatingFileHandler
logger = logging.getLogger(__name__)
logger.setLevel(level = logging.INFO)
#定义一个RotatingFileHandler，最多备份3个日志文件，每个日志文件最大1K
rHandler = RotatingFileHandler("log.txt",maxBytes = 1*1024,backupCount = 3)
rHandler.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
rHandler.setFormatter(formatter)

console = logging.StreamHandler()
console.setLevel(logging.INFO)
console.setFormatter(formatter)

logger.addHandler(rHandler)
logger.addHandler(console)

logger.info("Start print log")
logger.debug("Do something")
logger.warning("Something maybe fail.")
logger.info("Finish")

Can be seen in the project directory, the backup log file,

2016/10/09 19:36 732 log.txt 2016/10/09 19:36 967 log.txt.1 2016/10/09 19:36 985 log.txt.2 2016/10/09 19:36 976 log.txt.3

2.3 Set level message

Can set different log level, for outputting a control log,

日志等级：使用范围

FATAL：致命错误
CRITICAL：特别糟糕的事情，如内存耗尽、磁盘空间为空，一般很少使用
ERROR：发生错误时，如IO操作失败或者连接问题
WARNING：发生很重要的事件，但是并不是错误时，如用户登录密码错误
INFO：处理请求或者状态变化等日常事务
DEBUG：调试过程中使用DEBUG等级，如算法中每个循环的中间状态

2.4 capture traceback

The Python traceback module is used to track return exception information, may be recorded in the traceback logging in,

Code,

import logging
logger = logging.getLogger(__name__)
logger.setLevel(level = logging.INFO)
handler = logging.FileHandler("log.txt")
handler.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)

console = logging.StreamHandler()
console.setLevel(logging.INFO)

logger.addHandler(handler)
logger.addHandler(console)

logger.info("Start print log")
logger.debug("Do something")
logger.warning("Something maybe fail.")
try:
    open("sklearn.txt","rb")
except (SystemExit,KeyboardInterrupt):
    raise
except Exception:
    logger.error("Faild to open sklearn.txt from logger.error",exc_info = True)

logger.info("Finish")

Console and log file log.txt output,

Start print log
Something maybe fail.
Faild to open sklearn.txt from logger.error Traceback (most recent call last): File "G:\zhb7627\Code\Eclipse WorkSpace\PythonTest\test.py", line 23, in <module> open("sklearn.txt","rb") IOError: [Errno 2] No such file or directory: 'sklearn.txt' Finish

It may also be used logger.exception (msg, _args), which is equivalent to logger.error (msg, exc_info = True, _args),

will

logger.error("Faild to open sklearn.txt from logger.error",exc_info = True)

Replace,

logger.exception("Failed to open sklearn.txt from logger.exception")

Console and log file log.txt output,

Start print log
Something maybe fail.
Failed to open sklearn.txt from logger.exception Traceback (most recent call last): File "G:\zhb7627\Code\Eclipse WorkSpace\PythonTest\test.py", line 23, in <module> open("sklearn.txt","rb") IOError: [Errno 2] No such file or directory: 'sklearn.txt' Finish

Over 2.5 logging module

The main module mainModule.py,

import logging
import subModule
logger = logging.getLogger("mainModule")
logger.setLevel(level = logging.INFO)
handler = logging.FileHandler("log.txt")
handler.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)

console = logging.StreamHandler()
console.setLevel(logging.INFO)
console.setFormatter(formatter)

logger.addHandler(handler)
logger.addHandler(console)


logger.info("creating an instance of subModule.subModuleClass")
a = subModule.SubModuleClass()
logger.info("calling subModule.subModuleClass.doSomething")
a.doSomething()
logger.info("done with  subModule.subModuleClass.doSomething")
logger.info("calling subModule.some_function")
subModule.som_function()
logger.info("done with subModule.some_function")

Sub-module subModule.py,

import logging

module_logger = logging.getLogger("mainModule.sub")
class SubModuleClass(object):
    def __init__(self):
        self.logger = logging.getLogger("mainModule.sub.module")
        self.logger.info("creating an instance in SubModuleClass")
    def doSomething(self):
        self.logger.info("do something in SubModule")
        a = []
        a.append(1)
        self.logger.debug("list a = " + str(a))
        self.logger.info("finish something in SubModuleClass")

def som_function():
    module_logger.info("call function some_function")

After execution, control, and output the log file log.txt,

2016-10-09 20:25:42,276 - mainModule - INFO - creating an instance of subModule.subModuleClass 2016-10-09 20:25:42,279 - mainModule.sub.module - INFO - creating an instance in SubModuleClass 2016-10-09 20:25:42,279 - mainModule - INFO - calling subModule.subModuleClass.doSomething 2016-10-09 20:25:42,279 - mainModule.sub.module - INFO - do something in SubModule 2016-10-09 20:25:42,279 - mainModule.sub.module - INFO - finish something in SubModuleClass 2016-10-09 20:25:42,279 - mainModule - INFO - done with subModule.subModuleClass.doSomething 2016-10-09 20:25:42,279 - mainModule - INFO - calling subModule.some_function 2016-10-09 20:25:42,279 - mainModule.sub - INFO - call function some_function 2016-10-09 20:25:42,279 - mainModule - INFO - done with subModule.some_function

First, the main module defines logger'mainModule ', and that it is configured, it can getLogger elsewhere inside the interpreter process (' mainModule ') to give an object are the same, without reconfiguring, directly use. The sub-logger logger defined, the parent can share definitions and logger configuration, the logger is called Sons identified by name, to any logger 'mainModule' is beginning its sub-logger, e.g. 'mainModule.sub'.

The actual development of an application, may first be good that the corresponding configuration application passes logging configuration file write, can generate a root Logger, such as 'PythonAPP', then loading the logging configuration by fileConfig in the main function, followed by the rest of application, different, the module may be used a sub-root logger logger, such as 'PythonAPP.Core', 'PythonAPP.Web' to log, without the need for repeated logger definition and configuration of each module.

3 via the logging module file JSON or YAML

Although you can configure the Python code logging, but this does not flexible enough, the best way is to use a configuration file to configure. In Python 2.7 and later, you can load the logging configuration from the dictionary, which means you can load the configuration log through JSON or YAML file.

3.1 by JSON configuration file

JSON configuration file,

{
    "version":1, "disable_existing_loggers":false, "formatters":{ "simple":{ "format":"%(asctime)s - %(name)s - %(levelname)s - %(message)s" } }, "handlers":{ "console":{ "class":"logging.StreamHandler", "level":"DEBUG", "formatter":"simple", "stream":"ext://sys.stdout" }, "info_file_handler":{ "class":"logging.handlers.RotatingFileHandler", "level":"INFO", "formatter":"simple", "filename":"info.log", "maxBytes":"10485760", "backupCount":20, "encoding":"utf8" }, "error_file_handler":{ "class":"logging.handlers.RotatingFileHandler", "level":"ERROR", "formatter":"simple", "filename":"errors.log", "maxBytes":10485760, "backupCount":20, "encoding":"utf8" } }, "loggers":{ "my_module":{ "level":"ERROR", "handlers":["info_file_handler"], "propagate":"no" } }, "root":{ "level":"INFO", "handlers":["console","info_file_handler","error_file_handler"] } }

Load profiles via JSON, and then configure logging by logging.dictConfig,

import json
import logging.config
import os

def setup_logging(default_path = "logging.json",default_level = logging.INFO,env_key = "LOG_CFG"):
    path = default_path
    value = os.getenv(env_key,None)
    if value:
        path = value
    if os.path.exists(path):
        with open(path,"r") as f:
            config = json.load(f)
            logging.config.dictConfig(config)
    else:
        logging.basicConfig(level = default_level)

def func():
    logging.info("start func")

    logging.info("exec func")

    logging.info("end func")

if __name__ == "__main__":
    setup_logging(default_path = "logging.json")
    func()

3.2 By YAML configuration file

Configured via YAML file, it seems more than JSON Introduction and clear,

version: 1
disable_existing_loggers: False formatters:  simple:  format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s" handlers:  console:  class: logging.StreamHandler  level: DEBUG  formatter: simple  stream: ext://sys.stdout  info_file_handler:  class: logging.handlers.RotatingFileHandler  level: INFO  formatter: simple  filename: info.log  maxBytes: 10485760  backupCount: 20  encoding: utf8  error_file_handler:  class: logging.handlers.RotatingFileHandler  level: ERROR  formatter: simple  filename: errors.log  maxBytes: 10485760  backupCount: 20  encoding: utf8 loggers:  my_module:  level: ERROR  handlers: [info_file_handler]  propagate: no root:  level: INFO  handlers: [console,info_file_handler,error_file_handler]

Loading a configuration file by YAML, and then configure logging by logging.dictConfig,

import yaml
import logging.config
import os

def setup_logging(default_path = "logging.yaml",default_level = logging.INFO,env_key = "LOG_CFG"):
    path = default_path
    value = os.getenv(env_key,None)
    if value:
        path = value
    if os.path.exists(path):
        with open(path,"r") as f:
            config = yaml.load(f)
            logging.config.dictConfig(config)
    else:
        logging.basicConfig(level = default_level)

def func():
    logging.info("start func")

    logging.info("exec func")

    logging.info("end func")

if __name__ == "__main__":
    setup_logging(default_path = "logging.yaml ")
    func()
    

4 Reference

Every Python programmer must know log practice

Python standard logging module

python logging module learning log