python code tuning, who calls the database connection?

Disclaimer: This article is a blogger original article, shall not be reproduced without the bloggers allowed. https://blog.csdn.net/u011510825/article/details/83929527

    The latest in a complex page optimization, what happens then to the complex, light request may have 30-50, in which case, if each request is about 3,5 database requests, we open this page, it may generate a database request 150, the page open for longer than 30 seconds. . . . Speaking of boss intolerable proportions, issued instructions, two seconds to open the page, we can not get out (of the club to complete the tender mode).

   Hard to force the code of agriculture, in order to feed their families (soft mode), you can only analyze this page. The main problem with this page, there is a chart consisting of N, the data for each chart is dynamic, also returns different data according to different visitors (do data permission). So these are generated query the database, to speed up the connection that the database is certainly not to be, and all must be transferred to the cache database (Redis) in.

    So, the question is, how in a request, it performs see what sql statement it? Program should be a lot of, I said that under our program.

    python connections mysql, the current mainstream should be pymysql and MysqlDB, no matter what, you have a place in the final implementation of SQL, we need something with a context in place to record the sql statement, the sample code as follows:    

    def _execute(self, sql, params=None):
        """
        check if connection is alive. if not, reconnect
        :param sql:
        :param params:
        :rtype Cursor:
        """
        if hasattr(g, "profiling"):
            g.sqls.append(
                {
                    "sql": sql,
                    "params": params,
                    "db": self.db,
                }
            )
        self.cur.execute(sql, params)
        return self.cur

    Each implementation of a sql statement (in the case opened debug mode, profiling have value), will be added to the array g.sqls a record, add the tip on this g.sqls return it. (Description under our g, is a context variable, whose life cycle is the beginning of the end of a request to the request. Reference Flask)

    The front end to get the data:

    We can clearly see that each request is executed in the end how many sql statement.

    Know how much the implementation of sql statement, then the question is, how to find a place to call these sql statements it? We need to use the python stack information provided to track where the original call. When it comes to stack logs you might have to think, yes, with log stack similar, there are articles talking about the log stack ( Python elegant log (stack trace) )

    I used here is the traceback module, python standard module, it actually calls sys.exc_info (information). Here are the main reason for using traceback do it with a layer of packaging, supports a number of parameters, more convenient to use, we recommend using. code show as below:

    def _execute(self, sql, params=None):
        """
        check if connection is alive. if not, reconnect
        :param sql:
        :param params:
        :rtype Cursor:
        """
        if hasattr(g, "profiling"):
            import traceback

            extracted_list = traceback.extract_stack(limit=10)
            g.sqls.append(
                {
                    "sql": sql,
                    "params": params,
                    "db": self.db,
                    "stacks": [
                        "{fname} {lineno} {name}".format(fname=frame.filename, lineno=frame.lineno, name=frame.name)
                        if isinstance(frame, FrameSummary)
                        else "{fname} {lineno} {name}".format(fname=frame[0], lineno=frame[1], name=frame[2])
                        for frame in extracted_list
                        if (isinstance(frame, (list, tuple)) and not frame[0].find('site-packages') > 0)
                        or (isinstance(frame, FrameSummary) and not frame.filename.find('site-packages') > 0)
                    ],
                }
            )
        self.cur.execute(sql, params)
        return self.cur

    Code is relatively simple, is extracted_list to get information on the stack to do something with the format of the output formatted in a way, after all, the only thing to be looked at. traceback.extract_stack (limit = 10) indicates that up to 10 layers tracking, because the complexity of the code varies. Dizziness looked too deep, not too shallow to see the source of the call.

Request again, see the following figure:

So it is a request, which is a function of conducting a database query. (And later also added some database functions execution time, etc.)

 

Next, we can return, step by step strategy to destroy db query, use Redis replacement, still quite rewarding when you see a page optimized from 30 seconds to 2 seconds. Next content, I will talk next, how we use cache.

 I wish you a nice weekend.