Generate report memory overflow problem

 

      Recently, it is often found that the memory is not enough in the system. Here is a brief summary of the corresponding scenarios and processing ideas.

Background : Many online customers fail to generate reports. Since the celery and asynchronous solutions are used, they can only check the logs afterwards, but no exceptions are found. In this case, in order to reproduce the scene, the script can only be manually executed in the background to generate it. During the process, top, htop or free are used to observe the status. It is found that the memory has been increasing during the process of converting the pdf, and finally it is not enough to cause failure. Raw pdf has this problem, rtf does not.

Analysis: The generation of pdf uses a third-party toolkit, and the shell command is used to call and execute in the process. His internal logic cannot be controlled and changed, so he can only think of ways from the os level and business.

Option 1: Add physical memory, which can only be dealt with temporarily. And it is more troublesome and cannot solve the problem for a long time. The deliverables are unchanged.

Option 2: Adding swap can only be a temporary solution, but it is relatively simple and convenient. You can do this for urgent and important matters in the first quadrant. The deliverables are unchanged.
         The specific methods are as follows:
 1.

dd if=/dev/zero of=/root/swap bs=1024 count=1024000

    count indicates the number of blocks, generally a block is 1k, which creates a 1G file. However, you need to pay attention to the files in the /tmp directory, which will be cleaned up after the system restarts, so don't put them in /tmp in the end.

 two,

    become swsp partition

mkswap /root/swap

  three,

    make it valid

swapon / root / swap

  Four,

    In order to make it work even after reboot. The information needs to be written to /etc/fstab. The format is as follows:

/root/swap              swap                    swap    defaults        0 0

  Fives,

   Before adding:

            

   After adding:

            

 It is obvious that the free increases by 1G. At this time, it is OK to execute the program to generate the report.

Option 3: Modify the code and provide multiple reports to customers. This will not cause out of memory problems. But customer consent is required, and deliverables change.

Option 4: Long-term plan, split into finer grains, and deliverables change.

From a reasonable point of view. Option 4 is more reasonable, and the others are temporary and urgent modifications. At present, the fourth scheme has been used for improvement.