django mysql cannot query new data in the data table (database transaction)


 Background: In the actual django project, there is such a problem. Two processes A and B read and write the MySQL database. After insert m is completed, A notifies B to read MySQL through redis. As shown in the figure below, in the process of process B's permission, after dozens of pieces of data are inserted into the database, process B still cannot query it. As shown in the figure below: There are 6250 in the actual data table, but there are only 6218 queries in the process, and he cannot query the newly added data when the process is started.

              

                                               

 

 

Problem: After A insert is completed, B cannot query the newly added record m. However, the newly added m records can be seen through other client tools, in particular, the B process can also see the record m after restarting.

          Of course, what is more interesting is that this phenomenon does not occur in the development environment, only in the test environment.

 

 

Analysis 1: Due to the django technology used in the project, MySQL query and insert operations are also django encapsulation, and the record does exist in the database. So the suspicion is that django did not actually query the database.

 

Troubleshooting process 1: Did you actually check the database? Since this problem can be reproduced, all use python manage.py shell to check step by step.

              First, the tcpdump packet capture can be done. Second, if it is a test development environment, you can open the MySQL query log and check whether the query is performed through the log.

             Using method 1 here, execute tcpdump port 3306 on the machine of process B. If you want to view detailed information through wireshrk, you can use: tcpdump -w mysql.pcap port 3306 and host xxx , open

             mysql.pcap can see that the query has indeed been performed, as shown in Figure 1:

 

 

           

分析2:看来B是去查询MySQL了。但是MySQL确实没有返回m,看来是django 使用MySQL有些配置上有问题(不要轻易怀疑是MySQL的问题,毕竟它很成熟,一般都是使用上的不当)。这次为了证明这个猜想,需要打开MySQL的日志了。

 

排查过程2:找到my.cnf  在[mysqld] 下增加 log=/var/log/mysql/mysql.log 的配置。重启msyql(注:该操作仅限于在开发测试环境排查问题用,在生产环境慎用影响性能)

            然后我们用tail -f /var/log/mysql/mysql.log 来观察 A 和 B 到底在干嘛。如图二:

           

       通过查询日志可以知道的是,数据确实保存到了数据库中,其他客户端也可以查出。但是进程B查不出,并且B在开始时设置了set autocommit=0表示要手动提交事务。由于数据库的事务隔离

       级别的不同会导致两个事务查询出不同的结果。关于事务隔离级别的话在另一篇文章中有详细的介绍。

       

       可以通过下边的语句来看数据库的隔离级别是什么。当然操作具体表时还需要看表的引擎,比如MyISAM 是不支持事务的,而InnoDB是支持事务的。也就是说不管你的事务隔离机制如何,             MyISAM 类型的表都可以立即查询到已经真实存储在数据库中的记录。

select @@TX_ISOLATION;

 

 

解决方法:

    找到了原因解决方法就方便多了。

(1)如果觉得没有必要用事务,也就是每条语句执行后就可以直接持久化到数据,那就把django 中的 ‘django.middleware.transaction.TransactionMiddleware’去掉

(2)光是查询的话可以使用下边的代码:

from django.db import transaction
with transaction.autocommit():
      #业务代码

(3)第三个就是修改数据库的事务隔离级别(一般不建议)

(4)修改表的数据引擎,刚开始背景中介绍的开发环境不存在,测试环境存在,原因就是表引擎不同。

 

总结:

 

遇到问题后,首先要冷静分析现象出现的场景。什么情况下会复现该问题。尽量能使用差异点来定位问题可能的原因。

 

就像上边这个现象,B在重启后可以查询到m。利用其它客户端也能查询到m,唯独在B运行过程中查询不到,就可以根据B运行中和B重启的差异来分析。

 

当然如果对数据库事务熟悉的同学应该会敏感的想到可能的原因。也就不用去看msyql  log 了。

Guess you like

Origin http://10.200.1.11:23101/article/api/json?id=327073842&siteId=291194637