MongoDB-based python logging function

MongoDB is a database designed for scalability, high performance, and high availability. It can be applied to enterprises of all sizes, industries, and applications, and its data schema can be flexibly updated with the development of applications.

The daily operation and maintenance of the server usually generates a large amount of log information (such as errors, warnings, and user behaviors), which are usually stored in the log files of the server in text format. Although the log in text format is very readable, it is difficult to use and analyze the log every time the server is opened for viewing. Combined with the trend of today's microservice architecture, the log storage method based on native log files is also It will bring a lot of additional and avoidable workload to development and operation and maintenance. Storing the log in the database will make the use and analysis of the log more efficient. MongoDB has high performance and is easy to expand, and schama freenessit is very suitable to store logs in MongoDB. Many developers and enterprises store logs in MongoDB.

So, when developing based on Python, how to use MongoDB to store logs?

1. log4mongo-python

log4mongo-python  provides a mongodb handler for the Python logging module, which depends on the pymongo driver and can be seamlessly applied to the Python logging module, so as long as you understand the content introduced in " Python Logging Function Details ", log4mongo-python can get started directly . Therefore, this article will not repeat the use of the Python logging module, but directly provide a reference example (you can also view the latest version in GitHub ):

#!/usr/local/bin/python
# -*- coding: utf-8 -*-

import logging
from log4mongo.handlers import MongoHandler

logger = logging.getLogger('mongo_example')

mon = MongoHandler(host='localhost', database_name='mongo_logs')
mon.setLevel(logging.WARNING)

ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)

logger.addHandler(mon)
logger.addHandler(ch)


logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')

In the above example, the core of using MongoDB to store logs is to create the corresponding handler, that is, the following line:

mon = MongoHandler(host='localhost', database_name='mongo_logs')

Creating a MongoHandler is very simple. Most parameters have default values. If you want to configure more parameters, just look at MongoHandlerthe __init__functions:

def __init__(self, level=logging.NOTSET, host='localhost', port=27017,
                 database_name='logs', collection='logs',
                 username=None, password=None, authentication_db='admin',
                 fail_silently=False, formatter=None, capped=False,
                 capped_max=1000, capped_size=1000000, reuse=True, **kwargs)

2. 通过Dict对象配置log4mongo-python

基于我个人的开发实践，中大型项目开发中，通过Dict配置logging模块用的最多，由于log4mongo-python只是在logging模块上增加了一个新的handler，所以Dict与《Python 日志功能详解》中的写法一致，并根据实际情况赋上MongoHandler 初始化的参数值即可。举例如下：

#!/usr/local/bin/python
# -*- coding: utf-8 -*-

import logging
import logging.config
from log4mongo.handlers import MongoHandler

config = {
    'version': 1,
    'formatters': {
        'simple': {
            'format': '%(asctime)s - %(name)s - %(levelname)s - %(message)s',
        },
    },
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'level': 'DEBUG',
            'formatter': 'simple'
        },
        'file': {
            'class': 'logging.FileHandler',
            'filename': 'logging.log',
            'level': 'DEBUG',
            'formatter': 'simple'
        },
        'mongo': {
            'class': 'log4mongo.handlers.MongoHandler',
            'host': 'localhost', 
            # 'port': 27017,
            'database_name': 'mongo_logs2',
            # 'collection': 'logs',
            'level': 'DEBUG',
        },
    },
    'loggers':{
        'root': {
            'handlers': ['console'],
            'level': 'DEBUG',
            # 'propagate': True,
        },
        'simple': {
            'handlers': ['console', 'file'],
            'level': 'WARN',
        },
        'mongo': {
            'handlers': ['console', 'mongo'],
            'level': 'DEBUG',
        }
    }
}

logging.config.dictConfig(config)


# print 'logger:'
# logger = logging.getLogger('root')

# logger.debug('debug message')
# logger.info('info message')
# logger.warn('warn message')
# logger.error('error message')
# logger.critical('critical message')


# print 'logger2:'
# logger2 = logging.getLogger('simple')

# logger2.debug('debug message')
# logger2.info('info message')
# logger2.warn('warn message')
# logger2.error('error message')
# logger2.critical('critical message')

print 'logger3:'
logger2 = logging.getLogger('mongo')

logger2.debug('debug message')
logger2.info('info message')
logger2.warn('warn message')
logger2.error('error message')
logger2.critical('critical message')

3. What's More？

基于log4mongo将日志信息写入MongoDB后，可以通过RoboMongo或MongoChef等MongoDB客户端查看日志信息，并利用各种MongoDB的数据库查询语言，对日志进行分析处理。