The Python Logging Module: Practical Applications and Best Practices

This article analyzes Python's logging module in detail, from basic introduction to practical applications and best practices. We explain how to use this module for logging efficiently and how to avoid common pitfalls through specific code examples, aiming to help readers better grasp this powerful tool.

1. Introduction to Python logging module

The concept of logging and its role in software development

During the development process, in order to record the running status of the application, we usually use the method of printing logs. This method can not only help us understand the running status of the software system, but also help us quickly locate the problem when an error occurs in the system.

For example, suppose you have the following piece of code, which simply outputs some information:

print("This is some information.")

output:

This is some information.

However, if we need to record more complex information, such as error messages, warnings, or other important runtime information, just using print is not enough. This is where we need the logging module.

Introduction to Python logging module

Python's built-in logging module provides us with a complete logging solution. In many cases, you may want your application to be able to output some form of status information at runtime, especially when the application needs to handle long-running tasks, or when you are faced with a problem that needs to be diagnosed. Is your right-hand man.

The logging module can help us capture, process and record log information, so that we can quickly record log information anywhere the program is running. Compared with the simple print function, it is more flexible and can output logs to multiple places at the same time, such as: console, file, HTTP GET/POST, SMTP, Socket, etc., and can independently set the log level of each output .

Here is a simple example to illustrate how to use the logging module:

import logging

# Create a logger and set the log level to INFO
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

# Add a StreamHandler to send log messages to console
console_handler = logging.StreamHandler()
logger.addHandler(console_handler)

# Log an informational message
logger.info("This is an informational message.")

This code will output the following information to the console:

This is an informational message.

The basic composition of the logging module

The logging module mainly consists of the following parts:

Logger: Used to provide an interface for direct use by applications.

Handler: Sends log records (produced by the logger) to the appropriate destination output.

Filter: Provides finer-grained tools for deciding which log records to output.

Formatter: Specifies the final output format of the log record.

2. Detailed explanation of logging log module

Basic use of logging

Using Python's logging module is fairly simple, here's a basic example of how to create a log and output it to the console.

import logging

# This will log the message to the console
logging.warning('This is a warning message')

This code will output the following warning message:

WARNING:root:This is a warning message

Understanding log levels

In the logging module, we have 5 levels to describe the importance of logging. These levels are:

DEBUG: Detailed information, usually only used when diagnosing problems.

INFO: Confirming that things are working as expected.

WARNING: Something unexpected happened, or a problem might occur in the near future (for example, "Insufficient disk space"). But the software is still working fine.

ERROR: The software cannot perform certain functions due to a more serious problem.

CRITICAL: Critical errors, indicating that the program itself may not continue to run.

By default, the logging module logs to the console, and only processes logs whose level is above WARNING.

Loggers、Handlers和Formatters

In this part, we will explain the three main components of Loggers, Handlers and Formatters in detail.

The role and use of Loggers

Logger is a log object whose main task is to record logs. Wherever logs are needed in the application code, a logger instance can be created and used to record the required information. Here is a simple example of using logger:

import logging

# Create a logger
logger = logging.getLogger(__name__)

# Log some messages
logger.debug("This is a debug message.")
logger.info("This is an informational message.")
logger.warning("Careful! Something does not look right.")
logger.error("You have encountered an error.")
logger.critical("The program cannot recover from this situation!")

NOTE: When we run this code, we don't see any output. This is because by default, the logger level is set to WARNING, so only logs with a level above WARNING will be processed.

Types and functions of Handlers

The Handler object is responsible for sending log records to the appropriate destination. Different handlers can send logs to the console, files, mail, or even HTTP POST parameters, etc. Here's a simple example of how to use a handler to log to a file and to the console:

import logging

# Create a logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

# Create a file handler
file_handler = logging.FileHandler('my_log.log')
logger.addHandler(file_handler)

# Create a console handler
console_handler = logging.StreamHandler()
logger.addHandler(console_handler)

# Log some messages
logger.debug("This is a debug message.")
logger.info("This is an informational message.")
logger.warning("Careful! Something does not look right.")
logger.error("You have encountered an error.")
logger.critical("The program cannot recover from this situation!")

Functionality of Formatters and custom log formats

The Formatter object specifies the final order, structure, and content of the log records. You can customize the format of the log information to make the log information more readable. Here is an example of how to use formatter:

import logging

# Create a logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

# Create a console handler
console_handler = logging.StreamHandler()

# Create a formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# Add the formatter to the console handler
console_handler.setFormatter(formatter)

# Add the console handler to the logger
logger.addHandler(console_handler)

# Log some messages
logger.debug("This is a debug message.")
logger.info("This is an informational message.")
logger.warning("Careful! Something does not look right.")
logger.error("You have encountered an error.")
logger.critical("The program cannot recover from this situation!")

3. Application of Python log module in practice

Use logging to record exception information

In Python programming, it is often necessary to catch and handle exceptions. At this time, it is very convenient to use the logging module to record exception information. In the logging module, we can use the exception() method to record exception stack information. As shown in the following example:

import logging

logger = logging.getLogger(__name__)

try:
    a = [1, 2, 3]
    value = a[3]
except IndexError as e:
    logger.error("Unhandled exception", exc_info=True)
		```
当运行此段代码，日志记录器将记录下出现的异常信息，如下：

```python
ERROR:__main__:Unhandled exception
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
IndexError: list index out of range

Log rotation using RotatingFileHandler

When our application runs for a long time and generates a large number of logs, writing all the logs to a file may cause the log file to be too large. At this time, we can use RotatingFileHandler to perform log scrolling. When reaching a certain size or a certain time, RotatingFileHandler will automatically back up the current log file, and create a new log file to continue writing. As shown in the following example:

import logging
from logging.handlers import RotatingFileHandler

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

# Create a file handler
handler = RotatingFileHandler('my_log.log', maxBytes=2000, backupCount=10)
logger.addHandler(handler)

# Log some messages
for _ in range(10000):
    logger.info("Hello, world!")
		```
这段代码将在日志文件大小达到2000字节时创建一个新的日志文件，并保留最新的10个日志文件。

## 配置日志级别
根据我们的需要，可以在运行时改变日志的级别。例如，当我们在调试应用程序时，我们可能需要输出所有级别的日志。但是在生产环境中，我们可能只关心错误及以上级别的日志。我们可以通过setLevel()函数来改变日志级别。如下例所示：
```python
import logging

# Create a logger
logger = logging.getLogger(__name__)

# Set log level to DEBUG
logger.setLevel(logging.DEBUG)

# Log some messages
logger.debug("This is a debug message.")
logger.info("This is an informational message.")
logger.warning("Careful! Something does not look right.")
logger.error("You have encountered an error.")
logger.critical("The program cannot recover from this situation!")

Fourth, the best practice of Python log module

Create a logger at the module level using __name__

In Python, the __name__ variable is a built-in variable that represents the name of the current module. When we create loggers on a per-module level and use __name__ as the name, we can easily track which module the logging happened to.

import logging

# Create a logger at the module level
logger = logging.getLogger(__name__)

Use an appropriate log level

Different log levels represent different severities. Correct use of log levels can help us find the information we care about in a large number of logs. Generally speaking, for very serious errors, we should use CRITICAL or ERROR; for warning information, we should use WARNING; for general running information, we should use INFO; for debugging information, we should use DEBUG.

Use structured log messages

When our application has a lot of logs, we may want to log the log messages in a parseable way. For example, we can use JSON format for logging. In this way, we can analyze the logs with various log analysis tools.

import logging
import json

# Create a logger
logger = logging.getLogger(__name__)

# Log a structured message
logger.info(json.dumps({
    
    
    'action': 'User login',
    'username': 'user123',
    'ip_address': '123.123.123.123',
    'status': 'success',
}))

Use exception logging

When an exception is caught, we should use logger.exception(), so that the complete exception stack information can be recorded in the log.

import logging

logger = logging.getLogger(__name__)

try:
    x = 1 / 0
except ZeroDivisionError:
    logger.exception("Zero Division Error Caught.")

Such logs will contain enough information to help us find and fix problems.

Do not log sensitive information

Logs may be used by attackers to find vulnerabilities in the system, so we must not record sensitive information in logs, such as passwords, keys, and users' private data.

V. Summary

In this article, we introduce Python's logging module in detail, including its basic introduction, detailed explanation, practical application, and some best practices. To summarize the above:

1. The logging module is a flexible and powerful logging tool built into Python, which can output information during the running of the program to various output sources, such as standard output, files, mail, network, etc.
2. The logging module provides various levels of logging, including DEBUG, INFO, WARNING, ERROR, and CRITICAL. We can set different log levels according to requirements to record and display information of different severities.
3. In practice, we can use the logging module to log exception information, use RotatingFileHandler for log rotation, and change the log level at runtime.
4. For the best practices for the logging module, we mentioned creating loggers with __name__ at the module level, using appropriate log levels, using structured log messages, using exception logging, and not logging sensitive information.

The logging module of Python is a very powerful tool, I hope you can have a deeper understanding and more flexible use after reading this article.

If it is helpful, please pay more attention to
the personal WeChat public account: [Python full perspective]
TeahLead_KrisChang, 10+ years of experience in the Internet and artificial intelligence industry, 10+ years of experience in technology and business team management, Tongji Software Engineering Bachelor, Fudan Engineering Management Master, Aliyun certified cloud service senior architect, head of AI product business with hundreds of millions of revenue.