** python的多线程因为GIL全局解释锁的存在，所以python多线程实质上效率和单线程差不多。要想真正在效率上得到提高，必须使用多进程。**Multiprocess must use the multiprocess class. This blog verifies the infeasibility of IDApython multi-process programming.
Basic concepts of multi-process and multi-thread
Concurrency : Three tasks A, B, and C need to be processed within a period of time.
Parallel : A, B, and C three tasks are processed together at a moment.
Daemon (wire) process : A process that accompanies the execution of the main process. When the main process exits, the daemon process ends immediately.
By default, when a process exits, it will try to terminate all its daemons and their children. Sign: daemon
a.join() method : the main thread is blocked and paused, waiting for the execution of process a to finish before continuing to run.
Multi-process implementation template
# -*- coding:utf-8 -*-
from multiprocessing import Process
import time
def fun1(t):
print 'this is fun1',time.ctime()
time.sleep(t)
print 'fun1 finish',time.ctime()
def fun2(t):
print 'this is fun2',time.ctime()
time.sleep(t)
print 'fun2 finish',time.ctime()
if __name__ == '__main__':
a=time.time()
p1=Process(target=fun1,args=(4,))
p2 = Process(target=fun2, args=(6,))
p1.start()
p2.start()
p1.join()
#p2.join()
b=time.time()
print 'finish',b-a
First look at this classic code, create this code as a python script to run, and find that a pyc file with the same name will be generated in this directory ( do not have Chinese comments in the code, if there are Chinese comments, pyc will not be generated File, the running result is also wrong )
Use process explorer to observe our python script process. In addition to generating one IDLE process, there should also be a layer of two processes defined by us. As shown in the figure, you can see the successful execution of the two processes we defined.
The child process opened in my script executes the print function and does not print to IDLE, but the child process does indeed execute the function specified by the target. When the child process executes the function specified by the target, it will flash back , making people mistakenly think that there is an error in the process of calling multiple processes in the script. You can add a small writelog function to the function specified by the target to record the progress and errors encountered by the child process in executing the function specified by the target by writing a log.
Debug multi-process Target function
p1=Process(target=fun1,args=(4,))
p2 = Process(target=fun2, args=(6,))
As shown in the figure above, this is the statement for the calling process to enable the target functions fun1 and fun2.
In real projects, these two functions fun1 and fun2 may be more complicated. It is possible that the process will fail during the execution of the function, with a second retreat.
When writing the target function, you must remember a principle:
share the function with the parent process, and do not share data with the parent process
Be aware that the function specified by target is run in a newly opened child process. As shown in the explorer, you can clearly see that the third layer pythonw.exe is the process to execute the target function. This requires understanding the memory relationship between the parent process and the child process. The memory of the process is independent of each other, and the data must be independent when writing the target function. But functions can call functions written in the parent script.
The data is independent between the child process and the parent process, python2.7 can use inter-process communication to achieve data sharing. But try not to use it in this way, reproduce the data as much as possible, and try not to share data, because it is said that there are many pits in python2.7 data sharing.
If multiple processes need to communicate with the local server, you also need to consider whether the multiple processes will blow up the local server.
The root cause of multi-process failure
Although the console version idat64.exe runs IDApython, the processed modules and the generated data are not the same, (to ensure that the same data is not accessed, no need to consider thread locks). However, some modules still have abnormal errors when executing IDApython, but when the module is taken out to execute IDApython alone, it is found that the data can be output normally.
The root cause is that IDA does not support multiple processes. When using IDA functions, because there is no waiting in line to call API functions as required, an abnormal error occurs.