Concurrent Programming Summary:
One: Operating System
II: Process
Three: Thread
IV: coroutine
ps; note knowledge
One: Operating System
Summary:
programmer can not put all the hardware details of the operation have learned to manage these hardware and optimizing the use of them is very tedious work,
the tedious work is the operating system to dry, with him, the programmer freed from the tedious work,
only need to consider writing your own applications on it, application software directly operation the system provides the function indirectly using hardware.
Streamlined to say, the operating system is a coordination, management and control of computer hardware resources and control software resources
elaborate words, the operating system should be divided into two parts features
one: hide ugly hardware call interface provides calls for the application programmer better hardware resources, simpler, clearer model (system call interface).
After the application programmer With these interfaces, you do not have to consider the details of the operation of the hardware, to concentrate on developing its own application procedures.
For example: the operating system provides file abstraction, operation of the document is the operation of the disk,
with files we do not need to go on to consider the disk read and write control (such as control disk rotates, moves the head to read and write data and other details),
II: the application of hardware resources of the race to become the order of the request
, for example: many applications actually share a set of computer hardware, for example possible to have three applications at the same time need to apply for a printer to output the contents,
then a competition program resources to the printer prints, then b may be competitive to the printer resource,
It could be c, which led to the disorder, some printers may print the contents of a then went to print c ...,
a function of the operating system that this disorder will become orderly.
Key Knowledge:
# role of an operating system:
1: hide ugly complex hardware interface, providing good abstract interface
2: management, scheduling process, and multiple processes competing hardware become the order
# more than two channel technology:
1. background: for single-core, concurrency
ps:
now hosts are generally multi-core, each core will use multi-channel technology
with four cpu, io encountered obstruction to run a program cpu1, it will wait until the end of io re-scheduling, it is scheduled to four
of any of a cpu, as determined by the operating system scheduling algorithm.
2. spatial multiplexing: the memory as multiple concurrent channel programs
multiplexed on 3. Time: time-multiplexed slice of a cpu
stressed: io encountered cut, also occupied cpu cut too long, the core is cut before the preserved state of the process, so
as to ensure the next time you switch back, can continue to operate based on the last cut taking the position
II: process
process: running programs, CPU smallest resource unit. (The process is alive, a dynamic, temporary)
opened a process will open up a memory space in memory, independent of each other between the process and the process of isolation.
Program: is a pile of code is a static concept, death, permanent
ps: scheduling process :( we need to know there are two)
1, round-robin method
2, the multi-level feedback queue
1, concurrent parallel processes
in parallel: both are performed simultaneously, i.e. simultaneously, for example two races, in constant forward run
concurrently: simultaneously looks like performed, and alternately take turns essence,
the nature of concurrent: switching state of preservation + (also confirms the time characteristics of multi-channel multiplexing technique)
2, running three states:
blocking state: IO operation encountered, sleep , input, output, recv, so are blocking state
ready state: waiting to run
running state: state running programs
for tasks that are submitted by:
synchronization: a task is waiting for the results of another task, do not do anything during the take the results go to the next step
asynchronous: is a task does not need to wait for the results of another task, go to the next step, and so when the results of a callback and then go to the next step by asynchronous.
Sync: Example: For example, in line at the bank to do business;
asynchronous: Just after pumping number, the user can go to rest at the side of the latter (waiting for others to notice) is waiting for asynchronous message notification,
for the state of the program:
blocking: blocking state
non-blocking : the ready state and run state
3, the process of creating a
new process is created by the implementation of an existing process system calls used to create a process that is created.
The system call in windows are: CreateProcess, both the process of creating the process CreateProcess, is also responsible for the correct program into the new process.
For windows systems, from the beginning of the address space of the parent process and the child process is different. That isolation in the physical level
4, quit the process:
1, click ×. 2, the normal exit, Exit 3 mistakes. 4, kill the process (input in cmd tasklist | findstr process name;
taskkill / F / PID process ID;)
5, open process in two ways:
1, first:
the use of process modules to create processes
in a python process openers process, start method and concurrent effects
Import Time
from multiprocessing Import process
DEF F (name):
Print ( '% S child process' name%)
the time.sleep (. 1)
Print ( '% S subprocess end'% name)
IF the __name__ == '__main__':
T = process (target = F, args = ( 'GaN',))
t.start ()
# t.join () method jion, wait for the child ended, the main process is performed Code
the time.sleep (. 1)
Print ( 'end of the main')
2, a second:
their definition of a class to inherit MyProcess Process, initialization,
from multiprocessing Import Process
Import OS
class MyProcess (Process):
DEF the __init __ (Self, name ):
Super () __ .__ the init () # parent class does not pass the initial parameters
the self.name = name
DEF RUN (Self):
Print (os.getpid ()) # obtain three sub-process process ID numbers are not the same
print ( ' % s alley on that, to meet a ... 'the self.name%)
IF the __name__ ==' __main__ ': # windoss following conditions must be placed
p1 = MyProcess (' Chi Master ') the instantiation parameters of the process object #
p2 = MyProcess ( 'Tianqi')
p3 = MyProcess ( 'Stories')
# p1.daemon p1 = True # will be set to stay sprouting process, you must re-start () before setting
p1.start () #start will automatically call the RUN
p2.start ()
p3.start ()
p1.join () # jion concurrent changes to the serial execution, ensuring data security
p2.join ()
p3.join ()
Print ( 'primary')
PS:
1 p.start (): to start the process, and call the child process p.run ()
2 p.run (): method of operation, when the process starts, is it to call the specified target function, we custom classes must be achieved in the method
3 p.terminate (): p forcibly terminate the process, will not carry out any cleaning operation, if p create a child process, the child process to become a zombie process, using this method requires special care this situation. If p then also saved a lock will not be released, leading to a deadlock
4 p.is_alive (): If p is still running and returns True
5 p.join ([timeout]): p wait for the main thread to terminate (emphasis on : is the main thread in the state, etc., and p is in the running state)
3, process object other methods:
1 p.daemon: the default value is False, if set to True, the representative of p is a daemon running in the background, when p when the parent process terminates, p also will be terminated, and after setting True, p can not create their own new process must be set in p.start () before
2 p.name: name of the process
3 p.pid: Process PID
. 4 p.exitcode: processes running to None, if -N, representing the end signal N (to understand)
. 5 p.getpid () Gets the sub process number
6 p.getppid ( ) Get the parent process number
7 print (p.is_alive ()) # survival is determined whether the process result True
. 4, the plurality of opening processes
from process multiprocessing Import
Import OS
DEF F (n-):
Print ( 'sub-process id', os. getpid (), 'parent process ID', os.getppid ())
return * n-n-
IF the __name__ == '__main__':
Print ( 'main process ID')
P_LIST = []
for I in Range (10):
P = process (target = F, args = (I,))
p.start ()
p_list.append (P)
a plurality of processes running simultaneously (note, the order of execution of the child process is not determined according to the starting sequence)
[p.join () for p in p_list ] # list comprehensions join method of application
print ( 'primary')
5, the use of locks between multiple processes
to grab votes in the case of columns:
# concurrent operation, high efficiency, but the competition to write the same file, data writing disorder
from Import Process multiprocessing, Lock
Import Random
DEF Search ():
DIC = the json.load (Open ( 'DB'))
Print ( '\ 033 [43m remaining votes% s \ 033 [0m'% dic [ 'COUNT'])
DEF GET ():
DIC = the json.load (Open ( 'DB'))
the time.sleep (random.random ()) # analog read data network delay
if dic [ 'count']> 0 :
DIC [ 'COUNT'] - =. 1
the time.sleep (random.random ()) # analog network delaying the write data
the json.dump (DIC, Open ( 'DB', 'W'))
Print ( '\ 033 [ 32m ticketing success \ 033 [0m ')
else:
print ( '\ 033 [31m purchase failed \ 033 [0m')
DEF Task (Lock):
Search ()
lock.acquire () grab the lock
GET ()
lock.release () to release the lock
if __name__ == '__main__':
lock = lock () # generates a lock object
for i in range (100): # simulate concurrent client 100 grab votes
P = Process (target = Task, args = (lock,))
p.start ()
# can be locked when multiple processes to ensure that the same piece of data modification, at the same time only one task can be modified,
that is, the serial changes, yes, the speed is slow, but speed is sacrificed to ensure data security.
Although you can share data files to achieve inter-process communication, but the question is:
1. Low efficiency (based on a shared data file, and the file is the data on the hard disk)
2. The need to deal with their own lock
# So we had better find a solution be able to take into account:
1, high efficiency (multiple processes share a memory data)
2, to help us handle locking problems.
This is a message-based IPC communication mechanism mutiprocessing module offers us: queues and pipes.
IPC communication mechanism: queues and pipes
queues and pipes are the data stored in the memory
queue is based on the (pipeline + lock) to achieve, let us free from the lock complex problem,
we should try to avoid using shared data, Whenever possible, use messaging and queuing,
to avoid dealing with complex synchronization and locking problems, but also in the increase in the number of processes can often get a better availability of malleability.
6, mutual communication between processes
1, FIFO queue :()
PS: mutual communication between processes is realized by Queue JinableQueue queue
Queue: Queue Select all communication between the duct + lock general process, to manually lock there will be deadlock
pipeline: take complete data pipeline would be finished
queue queu:
from the queue values increase, q.put () but the value of the queue is full, the time to which value-added, will be in waiting, it can be said obstructive
get the value from the queue, q.get (), but takes the value of the queue is empty, wherein when the value will be in a waiting, it may be said to be blocked
to generate a queue object, Q = queue ()
q.close ()
turn off the queue, more data is added to the queue to prevent
the following three methods is only suitable for use in a single process:
q.qsize ()
Returns the current project queue correct number of
q.empty ()
q is empty If you call this method returns True.
q.full ()
if q is full, otherwise returns True Flase.
queues text columns:
from multiprocessing Import Queue
q = Queue (. 3) # Semi connections
#put, GET, put_nowait, get_nowait, Full, empty
q.put (3)
q.put (3)
q.put (3)
# q.put (3) # If the queue is full, the program will stop here, others waiting for the data to be removed, and then the data is placed in the queue.
# If the data queue has not been removed, the program will stop here forever.
the try:
q.put_nowait (3) # can use put_nowait, if the queue is full will not be blocked, but because the queue is full and error.
except: # so we can use a try statement to handle the error. Such a program would not have blocked it, but will lose the message.
print ( 'queue is full')
before # Therefore, we then add the data, you can look at the state of the queue, if full, will not continue to put up.
print (q.full ()) # full
Print (q.get ())
Print (q.get ())
Print (q.get ())
# Print (q.get ()) # Like the put method, If the queue is empty, then continue to take will be blocked.
the try:
q.get_nowait (3) # can use get_nowait, if the queue is full will not be blocked, but did not get to because of an error value.
except: # so we can use a try statement to handle the error. Such a program would not have been blocking it.
print ( 'queue is empty')
Print (q.empty ()) empty #
2, using the process queue:
Added:
the case of full (), q.empty (), q.get_nowait () do not apply to multi-process
case column: from multiprocessing Import Process, Queue
DEF SCZ (q): Object To queue passed in ** ***
q.put ( 'Hello')
DEF xfz (Q):
Print (q.get ())
IF the __name__ == '__main__':
Q = Queue ()
P = Process (= SCZ target, args = (Q ,))
P1 = process (target = xfz, args = (Q,))
p.start ()
p1.start ()
summary: communication is achieved between the two queues by the processes, the resulting data on queue, make another extraction process
should be noted here that make the queue object as a parameter passed to the two processes are
7, the producer consumer model:
Manufacturer: manufacturing data generated or party
consumer: a party for processing data
problem: the imbalance between supply and demand to solve the problem, use a queue to handle
cases in the column :( can not allow blocking data = q. get () position)
from Import Process multiprocessing, JoinableQueue, Queue
Import Time
Import Random
DEF SCZ (name, Food, Q):
for I in Range (0,10):
Data = ( '% s% s produced buns% s '% (name, Food, I))
the time.sleep (0.5)
q.put (Data)
Print (Data)
DEF xfz (name, Q):
the while True:
Data = q.get ()
IF == None Data: If the data is a break # None, illustrates the data queue has been fetched
print ( '% s% s ate buns'% (name, data))
the time.sleep (random.random ())
q.task_done () # tell the queue is removed from the queue of data, and have all been processed, the task ends *****
IF the __name__ == '__main__':
# Q = Queue ()
Q = JoinableQueue () # *******
P = Process (= SCZ target, args = ( '111', 'beef', Q))
P1 = Process (= SCZ target, args = ( 'dragon', 'cabbage filling', Q))
C = Process (target = xfz, args = ( 'open', Q))
C1 = Process (target = xfz, args = ( 'sesame', Q))
C = True .daemon
c1.daemon = True
p.start ()
p1.start ()
c.start ()
c1.start ()
# this method is a too low
# P.join () # ensure that producers indeed produce finished
# p1.join () # ensure that producers indeed produce finished
# q.put (None) # End producers to produce a value in the queue increases None, when value Description none taken to take over the data queue
# q.put (none) # a consumer with a none, consumers have to use two two none, consumers can receive a none, ending immediately!
# Advanced method
p.join () # ensure that producers indeed produce finished
p1.join ()
q.join () # wait until all the data in the queue is removed
8, process pool and asynchronous callbacks: ****** *
