When talking about the characteristics Twisted asynchronous and non-blocking mode, use the callback function in which naturally becomes indispensable, then we will look at Python Twisted framework used in the Deferred object to manage the use of callback functions.
First throw some of our views when discussing the use of callback programming:
Activation errback is very important. Since errback same functions except block, so the user needs to ensure their presence. They are not optional, but mandatory.
Not the wrong point in time activation with activation callback callback equally important at the right point in time. A typical usage is, callback and errback are mutually exclusive, that can only run one of them.
Using the code callback function somewhat difficult to reconstruct them.
Deferred
Twisted sequence using the Deferred object to manage the callback function. In some cases it may take a series of related functions to the Deferred object to call in sequence (a series of these callback function called a callback function chain) when asynchronous operation is complete; but also there are some functions appear in asynchronous operation to invoke the exception. When the operation completes, a callback to the first call, or when an error occurs, the error will first call the first callback processing, and returns the value of each Deferred object will callback function or error handler callback function passed to the next function in the chain.
Callbacks
a twisted.internet.defer.Deferred objects representative of a moment will produce a function result in the future. We can associate a callback function to the Deferred object, the outcome once the Deferred object, the callback function is called. In addition, Deferred object also allows developers to register it with an error handler callback function. Deferred mechanism for a wide variety of blocked or delayed operations provides a standardized interface for developers.
from twisted.internet import reactor, defer
def getDummyData(inputData):
"""
This function is a dummy which simulates a delayed result and
returns a Deferred which will fire with that result. Don't try too
hard to understand this.
"""
print('getDummyData called')
deferred = defer.Deferred()
# simulate a delayed result by asking the reactor to fire the
# Deferred in 2 seconds time with the result inputData * 3
reactor.callLater(2, deferred.callback, inputData * 3)
return deferred
def cbPrintData(result):
"""
Data handling function to be added as a callback: handles the
data by printing the result
"""
print('Result received: {}'.format(result))
deferred = getDummyData(3)
deferred.addCallback(cbPrintData)
# manually set up the end of the process by asking the reactor to
# stop itself in 4 seconds time
reactor.callLater(4, reactor.stop)
# start up the Twisted reactor (event loop handler) manually
print('Starting the reactor')
reactor.run()
A plurality of callback functions
on a Deferred object can be associated with a plurality of callback, the callback function in the first callback function will result chain Deferred object as a parameter to the call, and the second to the first callback function The results for the parameter to call, and so on. Why do we need such a mechanism? Consider this situation, twisted.enterprise.adbapi returns a Deferred object - a result of a SQL query, there may be a web window will add a callback function on the Deferred object to convert the query results into HTML format and then continue to pass forward the Deferred object, then Twisted will call the callback function and the results returned to the HTTP client. In the case of error or abnormal, callback chain it will not be called.
from twisted.internet import reactor, defer
class Getter:
def gotResults(self, x):
"""
The Deferred mechanism provides a mechanism to signal error
conditions. In this case, odd numbers are bad.
This function demonstrates a more complex way of starting
the callback chain by checking for expected results and
choosing whether to fire the callback or errback chain
"""
if self.d is None:
print("Nowhere to put results")
return
d = self.d
self.d = None
if x % 2 == 0:
d.callback(x * 3)
else:
d.errback(ValueError("You used an odd number!"))
def _toHTML(self, r):
"""
This function converts r to HTML.
It is added to the callback chain by getDummyData in
order to demonstrate how a callback passes its own result
to the next callback
"""
return "Result: %s" % r
def getDummyData(self, x):
"""
The Deferred mechanism allows for chained callbacks.
In this example, the output of gotResults is first
passed through _toHTML on its way to printData.
Again this function is a dummy, simulating a delayed result
using callLater, rather than using a real asynchronous
setup.
"""
self.d = defer.Deferred()
# simulate a delayed result by asking the reactor to schedule
# gotResults in 2 seconds time
reactor.callLater(2, self.gotResults, x)
self.d.addCallback(self._toHTML)
return self.d
def cbPrintData(result):
print(result)
def ebPrintError(failure):
import sys
sys.stderr.write(str(failure))
# this series of callbacks and errbacks will print an error message
g = Getter()
d = g.getDummyData(3)
d.addCallback(cbPrintData)
d.addErrback(ebPrintError)
# this series of callbacks and errbacks will print "Result: 12"
g = Getter()
d = g.getDummyData(4)
d.addCallback(cbPrintData)
d.addErrback(ebPrintError)
reactor.callLater(4, reactor.stop)
reactor.run()
One thing to note is that the processing method gotResults self.d way in. Before Deferred object is the result of an error or activation, this property is set to become None, so Getter instance will no longer hold to be activated Deferred object. This has several advantages, first of all, to avoid the possibility of Getter.gotResults sometimes repeat the same Deferred object activation (this will cause AlreadyCalledError an exception). Second, doing so makes it possible to add on the Deferred object that calls the callback function Getter.getDummyData function without problems. Further, the garbage collector such that Python is easier to detect whether an object to be recovered by reference cycle.
Visualization of the explanation
herein described write pictures 
1. Method request to request data Data Sink, to give Deferred object returned.
2. The method of associating the request to the callback function Deferred object. 
1. When the results are ready, pass it to the Deferred object. If the operation is successful is called Deferred object .callback (result) method, if the operation fails to call .errback (faliure) Deferred object. Note that failure is an instance of class twisted.python.failure.Failure.
2.Deferred object using the result or faliure to activate the callback function to add error handling or before the callback function. Then the following rules to proceed along the chain down callback function:
Results callback function is always used as the first argument is passed to the next callback function, thus forming a chain of processors.
If a callback throws an exception, it goes to a callback function to perform error handling.
If a faliure not addressed, then it would have been passed down along the chain error handling callback functions, it's a bit like the asynchronous version except statement.
If an error handling callback function does not throw an exception or return a twisted.python.failure.Failure instance, then the rest is transferred to a callback function.
Error handling callback
error handling Deferred object model is based on Python-based exception handling. In the absence of error, and all callbacks will be executed, one by one, as mentioned above.
If no callback function but performs error handling callback function (such as DB query error occurred), then a twisted.python.failure.Failure object will be passed to the first error handling callback function (you can add multiple errors handling callback functions, like the callback chain). Error handling functions can callback code blocks except chain as normal Python code.
Unless except code block explicitly raise an error, or the Exception object will be captured and will not continue to spread down, and then began to execute the program properly. The same is true for error handling callback chain, unless you explicitly return a Faliure or re-throw an exception, or an error will stop continues to spread, and then from there it will begin normal callback function chain (error handling the callback function return value). If an error handling callback function returns a Faliure or throws an exception, then the Faliure or abnormal will be passed to the next error handler callback function.
Note that if an error handling callback function does not return anything, then it actually returns None, which means that after the error handling callback function execution will continue callback chain execution. This is probably not what you actually expect, so make sure your error handling callback function returns a Faliure the object (or an object that is passed to it when the Faliure parameters) or a meaningful return value to a callback function to the next .
twisted.python.failure.Failure there is a useful method is called trap, you can make the following code into another form more efficient:
try:
# code that may throw an exception
cookSpamAndEggs()
except (SpamException, EggException):
# Handle SpamExceptions and EggExceptions
...
Can be written as:
def errorHandler(failure):
failure.trap(SpamException, EggException)
# Handle SpamExceptions and EggExceptions
d.addCallback(cookSpamAndEggs)
d.addErrback(errorHandler)
If you pass parameters to faliure.trap not be able to match and Faliure errors, and that it will re-throws this error.
Where there is a need for attention, twisted.internet.defer.Deferred.addCallbacks method functions and addCallback talk on addErrback function is similar, but not exactly the same. Consider the following situation:
# Case 1
d = getDeferredFromSomewhere()
d.addCallback(callback1) # A
d.addErrback(errback1) # B
d.addCallback(callback2)
d.addErrback(errback2)
# Case 2
d = getDeferredFromSomewhere()
d.addCallbacks(callback1, errback1) # C
d.addCallbacks(callback2, errback2)
For Case 1, if an error occurs in callback1 inside, then errback1 will be called. For Case 2, it is called but Yes errback2.
In fact, because, in Case 1, row A process getDeferredFromSomewhere will successfully executed, the line B will occur when getDeferredFromSomewhere process performed when an error or the callback1 line A is performed. While in Case 2, row C of errback1 will only deal with errors generated when getDeferredFromSomewhere execution, and is not responsible for errors callback1 generated.
Unhandled error
if a Deferred object (that is, if it has the next errback will certainly call) when there is an unhandled error is cleared off the garbage collector, then Twisted will this error traceback records to the log file. This means you may still be able to record without adding errback error. But be careful, if you hold the Deferred object references, and it will never be cleaned up garbage collector, then you will never see this error (and your callbacks will mysteriously never executed ). If you are unsure whether the above situation occurs, you should explicitly add a errback after the callback chain, even if just write:
# Make sure errors get logged
from twisted.python import log
d.addErrback(log.err)
Synchronous and asynchronous processing results of
function in some applications, it may also have a synchronized, there will be a function of asynchronous. For example, a user authentication function, if it is to check whether the user has been authenticated from memory so that it can immediately return results; however, if it needs to wait for data on the network, then it should return a data arrival is activated when the Deferred object. That is to say, who wants to check whether the user has authenticated function simultaneously need to be able to accept the result and the Deferred object returned immediately.
The following examples, authenticateUser isValidUser used to authenticate the user:
def authenticateUser(isValidUser, user):
if isValidUser(user):
print("User is authenticated")
else:
print("User is not authenticated")
This function is assumed isValidUser immediately returned, but in fact isValidUser may be asynchronous to authenticate the user and returns a Deferred object. This function is adjusted isValidUser received both synchronous asynchronous receiver can isValidUser is possible. While the synchronization function return value into Deferred object are possible.
In the library function code processing possible Deferred object
which is likely to be transmitted to a user authentication method of synchronizing authenticateUser:
def synchronousIsValidUser(user):
'''
Return true if user is a valid user, false otherwise
'''
return user in ["Alice", "Angus", "Agnes"]
This is an asynchronous method of user authentication, returns a Deferred object:
from twisted.internet import reactor, defer
def asynchronousIsValidUser(user):
d = defer.Deferred()
reactor.callLater(2, d.callback, user in ["Alice", "Angus", "Agnes"])
return d
Our initial hope to achieve authenticateUser isValidUser are synchronized, but now it needs to be changed to handle both synchronous and can handle asynchronous isValidUser achieved. In this regard, you can use the function to call maybeDeferred isValidUser, this function can ensure that the return value of the function is isValidUser a Deferred object, even if isValidUser is a synchronous function:
from twisted.internet import defer
def printResult(result):
if result:
print("User is authenticated")
else:
print("User is not authenticated")
def authenticateUser(isValidUser, user):
d = defer.maybeDeferred(isValidUser, user)
d.addCallback(printResult)
Now isValidUser either synchronous or asynchronous can be had.
The function may be rewritten into synchronousIsValidUser returns a Deferred object, can be found here.
Cancel the callback function
motive: a Deferred object may take a long time to call a callback function, and even never called. Sometimes it may not be so good patience to wait for Deferred return results. Since all of the code to be executed after completion of Deferred in your application or call the library, then you can choose to ignore the results when you have passed a long time before results are received. However, even if you choose to ignore this result, the underlying operating Deferred object of this work is still generated in the background, and consume machine resources, such as CPU time, memory, network bandwidth, and even disk capacity. Therefore, when the user closes the window, click the Cancel button, disconnect or send a "stop" command from your server, then you need to explicitly declare before you the results of the original operation is no longer interested, so that the original Deferred object can do some clean-up work and free up resources.
This is a simple example, you want to connect to an external machine, but the machine is too slow, so you need to add a Cancel button to terminate the connection attempt in the application, so that users can connect to another machine. This is probably the logic of such an application:
def startConnecting(someEndpoint):
def connected(it):
"Do something useful when connected."
return someEndpoint.connect(myFactory).addCallback(connected)
# ...
connectionAttempt = startConnecting(endpoint)
def cancelClicked():
connectionAttempt.cancel()
Obviously, startConnecting by some UI element is used to allow the user to select which machine is connected. Then a Cancel button is accompanied to the cancelClicked function.
When connectionAttempt.cancel is called, the following actions occur:
Resulting in a potential connection operation is terminated, if it is still in progress, then
in any case, make connectionAttempt the Deferred object is completed in a timely manner
may result in connectionAttempt the Deferred object because the error handler CancelledError wrong call
even if the operation has been canceled so that the underlying expressed operation needs to stop, but the underlying operations are unlikely to immediately react. Even in this simple example, there is a not interrupted operations: DNS, and therefore need to be performed in a thread; join operation if the application can not be while waiting for the cancellation of the domain name resolution. So you want to cancel the Deferred object may not invoke the callback function immediately or error handling callback function.
A Deferred object may wait for the completion of another Deferred object at any point in the execution of its callback chain. There is no way to know whether everything is ready at a certain point callback chain. Since there may be a lot will function on the level of a callback chain wish to cancel with a Deferred object, so any level of function are likely to call .cancel () function at any time on the chain. .cancel () function never throwing an exception or returns no value. You can recall it, even if the Deferred object has been activated, and it has no remaining a callback function.
In the example of a Deferred object at the same time, you can give it a cancel function (constructor Deferred object is DEF the init (Self, canceller = None): (Source)), the canceller can do anything. Ideally, it would do everything to stop the operation before you request, but does not always guarantee this. So cancel the Deferred object just do my best. There are several reasons:
the Deferred object does not know how to cancel the bottom.
Underlying the operation has been performed to a state of irrevocable, because it may have performed some irreversible action.
Deferred object may have been the result, so there is nothing to cancel up.
Call cancel () after the function, regardless of whether it can be canceled, there will always get a successful result, error situation does not occur. In the first and second case, the underlying operation continues, the Deferred large objects may be invoked as a parameter twisted.internet.defer.CancelledError its errback.
If you cancel the Deferred object is waiting for another Deferred object, then cancel the operation will be passed forward to this Deferred object.
You can refer to the API.
The default behavior to cancel
All objects support Deferred canceled, but only provides a very simple act, nor release any resources.
Consider the following example:
operation = Deferred()
def x(result):
print("Hooray, a result:" + repr(x))
operation.addCallback(x)
# ...
def operationDone():
operation.callback("completed")
If you need to cancel the Deferred object of this operation, but did not cancel the operation a canceller function, it will produce results in one of two of the following:
If operationDone has been called, that is the object operation has been completed, then nothing will change. operation still have a result, but since there is no other callback functions, so there is no change in behavior on what can be seen.
If operationDone has not yet been called, then the operation will be immediately activated as a parameter to CancelledError errback.
Under normal circumstances, if a Deferred object has come to call a callback function to call callback will lead to a AlreadyCalledError. Therefore, callback may have been canceled, but the Deferred object is not called once again canceller, will only lead to a no-op. If you repeatedly call the callback, you will get a AlreadyCalledError exception.
Creating can cancel the Deferred object: Custom cancel function
Suppose you implement an HTTP client, the server returns returns a response will be activated when the Deferred object. Cancel is best to close the connection. In order to cancel the function to do so, a function that can be passed to the constructor Deferred object as a parameter (when the Deferred object is canceled will call this function):
class HTTPClient(Protocol):
def request(self, method, path):
self.resultDeferred = Deferred(
lambda ignore: self.transport.abortConnection())
request = b"%s %s HTTP/1.0\r\n\r\n" % (method, path)
self.transport.write(request)
return self.resultDeferred
def dataReceived(self, data):
# ... parse HTTP response ...
# ... eventually call self.resultDeferred.callback() ...
Now if you call cancel the Deferred object HTTPClient.request () return () function, the HTTP request is canceled (if not late already). Note that, even in a has been canceled, call the callback () on a Deferred object with the canceller.
DeferredList
Sometimes you want to be notified after several different events have taken place, but not everyone will be informed about the incident. For example, you want to wait for a list of all the connections are closed. twisted.internet.defer.DeferredList would apply in this case.
Deferred multiple objects to create a DeferredList, simply pass the list to the one you want to wait for the Deferred object:
Creates a DeferredList
dl = defer.DeferredList([deferred1, deferred2, deferred3])
Now you can use that as an ordinary Deferred DeferredList to look up, for example, you can also call addCallbacks and so on. This DeferredList will only call its callback function after all of the Deferred object are completed. This callback parameter is the object of all this DeferredList Deferred object contains a list of return results, such as:
# A callback that unpacks and prints the results of a DeferredList
def printResult(result):
for (success, value) in result:
if success:
print('Success:', value)
else:
print('Failure:', value.getErrorMessage())
# Create three deferreds.
deferred1 = defer.Deferred()
deferred2 = defer.Deferred()
deferred3 = defer.Deferred()
# Pack them into a DeferredList
dl = defer.DeferredList([deferred1, deferred2, deferred3], consumeErrors=True)
# Add our callback
dl.addCallback(printResult)
# Fire our three deferreds with various values.
deferred1.callback('one')
deferred2.errback(Exception('bang!'))
deferred3.callback('three')
# At this point, dl will fire its callback, printing:
# Success: one
# Failure: bang!
# Success: three
# (note that defer.SUCCESS == True, and defer.FAILURE == False)
DeferredList does not call normally errback, but unless the cousumeErrors set to True, otherwise an error generated in the Deferred object will still activate the Deferred object for each respective errback.
Note that if you want to add to the application callback function on the Deferred object DeferredList to go, then you need to pay attention to time to add a callback function. Add a Deferred object to DeferredList will lead also to the Deferred object is added to a callback (when the callback function to run, its function is to check whether DeferredList has been completed). Most importantly, this variable callback function to record the return value of a Deferred object and pass this value to the final DeferredList to the callback function as a parameter list.
So, if you're offering the Deferred object is added to a callback function after a Deferred added to DeferredList, then the return value will not be passed to the callback function DeferredList callback function newly added. To avoid this situation, it is recommended not to give the Deferred add callback after adding a Deferred object to DeferredList in.
def printResult(result):
print(result)
def addTen(result):
return result + " ten"
# Deferred gets callback before DeferredList is created
deferred1 = defer.Deferred()
deferred2 = defer.Deferred()
deferred1.addCallback(addTen)
dl = defer.DeferredList([deferred1, deferred2])
dl.addCallback(printResult)
deferred1.callback("one") # fires addTen, checks DeferredList, stores "one ten"
deferred2.callback("two")
# At this point, dl will fire its callback, printing:
# [(True, 'one ten'), (True, 'two')]
# Deferred gets callback after DeferredList is created
deferred1 = defer.Deferred()
deferred2 = defer.Deferred()
dl = defer.DeferredList([deferred1, deferred2])
deferred1.addCallback(addTen) # will run *after* DeferredList gets its value
dl.addCallback(printResult)
deferred1.callback("one") # checks DeferredList, stores "one", fires addTen
deferred2.callback("two")
# At this point, dl will fire its callback, printing:
# [(True, 'one), (True, 'two')]
DeferredList accept the three key parameters to customize its behavior: fireOnOneCallback, fireOnOneErrback and cousumeErrors. If you set fireOnOneCallback, as long as there is a Deferred object invokes its callback function, DeferredList will call its callback function immediately. Similarly, if you set the fireOnOneErrback, as long as there is a Deferred called errback, DeferredList will call it errback. Note, DeferredList just a one-time, so after a callback or errback call, it will do nothing (it will ignore the Deferred pass it to all of its results).
fireOnOneErrback option in case you want to wait for the successful implementation of all the things, but also need to know immediately when an error is very useful.
Deferred object consumeErrors error will DeferredList parameters contained in the produced after the establishment of DeferredList object will not be passed to each of the original Deferred object respective errbacks. After creating DeferredList objects, any errors in a single Deferred object generated will be converted into results None of callback calls. With this option prevents Deferred it contains the "Unhandled error in Deferred" warning, without adding extra errbacks (otherwise, it should eliminate the need to add a warning for each errback Deferred object). True to consumeErrors pass a parameter does not affect the behavior of fireOnOneCallback and fireOnOneErrback. You should always use this parameter, unless you want to add callbacks or errbacks to these lists in the future Deferred object, or unless you know they will not generate an error. Otherwise, it will lead to error is a Twisted logged in "unhandled error".
DeferredList is a common use of some asynchronous operation result of the parallel combination together. If all the operations are successful, it can operate successfully, if there is one operation fails, then the operation fails. twisted.internet.defer.gatherResults is a shortcut:
from twisted.internet import defer
d1 = defer.Deferred()
d2 = defer.Deferred()
d = defer.gatherResults([d1, d2], consumeErrors=True)
def cbPrintResult(result):
print(result)
d.addCallback(cbPrintResult)
d1.callback("one")
# nothing is printed yet; d is still awaiting completion of d2
d2.callback("two")
# printResult prints ["one", "two"]
Chain of Deferred
if you need to wait for a Deferred object to execute another Deferred object, you have to do is return a Deferred object from its callback function callback function in the chain. In particular point, if you return a Deferred object B from a Deferred callback object A, then A callback function chain will wait before the function call in the callback B (). In this case, a first callback function parameters A, B is the last callback function returns a result.
Note that if an Deferredobject is in its callback function directly or indirectly returned to itself, then such behavior is not defined. The code attempts to detect this situation and give a warning. In the future it may have a direct throw.
If this seems a little complicated, do not worry - when you encounter this situation, you might recognize and know why it directly produce such a result. If you need to manually put Deferred objects
are linked, there is a convenient way: chainDeferred(otherDeferred)
The last word of mouth to recommend a good python gathering [ click to enter ], there are a lot of old-timers learning skills, learning experience, interview skills, workplace share experiences and so on, the more we carefully prepared the zero-based introductory information, information on actual projects, the timing has to explain the Python programmer technology every day, share some learning methods and the need to pay attention to the small details
summary
we recognize deferred how to help us solve these problems are:
we can not ignore errback, asynchronous programming in any of the API need it. Deferred support errbacks.
Activate callback times may lead to very serious problems. Deferred can only be activated once, which is similar to the processing method of synchronizing programming try / except the.
The program contains a callback rather difficult at the time of reconstruction. With deferred, we have to reconstruct the program by modifying callback chain.
