twisted reactor implementation source code parsing


twisted reactor implementation source code parsing


1. reactor Source resolve

1.1 Case Study Code:

from twisted.internet import protocol
from twisted.internet.protocol import Factory
from twisted.protocols.basic import LineReceiver
from twisted.internet import reactor

class Chat(LineReceiver):

    def __init__(self, users):
        self.users = users = None
self.state = "GETNAME"

def connectionMade(self):
        self.sendLine(b'Whats your name')

    def connectionLost(self, reason):
        if in self.users:
            del self.users[]

    def lineReceived(self, line):
        if self.state == "GETNAME":

    def handle_GETNAME(self, name):
        if name in self.users:
            self.sendLine(b"Name taken, please choose another.")
self.sendLine(b"Welcome, %s!" % (name,)) = name
        self.users[name] = self
        self.state = "CHAT"

def handle_CHAT(self, message):
        message = "<%s> %s" % (, message)
        for name, protocol in self.users.items():
            if protocol != self:

class ChatFactory(Factory):

    def __init__(self):
        self.users = {} # maps user names to Chat instances

def buildProtocol(self, addr):
        return Chat(self.users)

def main():
    # 监听端口
reactor.listenTCP(8123, ChatFactory())

if __name__ == '__main__':


1.2. Reactor code analysis

Documents have been traced to (), so the front part is not repeated here.


reactor has been found, the following look at what it does after running:

res =


<bound method of <twisted.internet.selectreactor.SelectReactor object at 0x000000C61740B6D8>> <class 'method'>

Look at github get: twisted / internet /

class _SignalReactorMixin(object):

    def startRunning(self, installSignalHandlers=True):


        PosixReactorBase _SignalReactorMixin parent class and has the function ReactorBase, but _SignalReactorMixin front mounting mro order, then, will the first call _SignalReactorMixin.


        self._installSignalHandlers = installSignalHandlers



    def run(self, installSignalHandlers=True):


        self.mainLoop ()


    def MAINLOOP (self):

        while self._started:


                while self._started:

                    # Advance simulation time in delayed event

                    # processors.


                    t2 = self.timeout()

                    t = self.running and t2

                    self.doIteration (t) # monitor cycle to achieve Affairs



                log.msg("Unexpected error in main loop.")



                log.msg('Main loop terminated.')


Here is the main loop.



The main loop to do a few things:

  1. Listener: self.doIteration (t)


Monitor realization: self.doIteration (t)


class SelectReactor(posixbase.PosixReactorBase, _extraBase):
    A select() based reactor - runs on all POSIX platforms and on Win32.
    @ivar _reads: A set containing L{FileDescriptor} instances which will be
        checked for read events.
    @ivar _writes: A set containing L{FileDescriptor} instances which will be
        checked for writability.

def __init__(self):
        Initialize file descriptor tracking dictionaries and the base class.
self._reads = set()
        self._writes = set()

    def doSelect(self, timeout):
        Run one iteration of the I/O monitor loop.
        This will run all selectables who had input or output readiness
        waiting for them.
            r, w, ignored = _select(self._reads,
                                    [], timeout)
        except ValueError:
            # Possibly a file descriptor has gone negative?
            # Something *totally* invalid (object w/o fileno, non-integral
            # result) was passed
(select.error, socket.error, IOError) as se:
            # select(2) encountered an error, perhaps while calling the fileno()
            # method of a socket.  (Python 2.6 socket.error is an IOError
            # subclass, but on Python 2.5 and earlier it is not.)
if se.args[0] in (0, 2):
                # windows does this if it got an empty list
if (not self._reads) and (not self._writes):
se.args[0] == EINTR:
se.args[0] == EBADF:
                # OK, I really don't know what's going on.  Blow up.

_drdw = self._doReadOrWrite
        _logrun = log.callWithLogger
        for selectables, method, fdset in ((r, "doRead", self._reads),
                                           (w,"doWrite", self._writes)):
            for selectable in selectables:
                # if this was disconnected in another thread, kill it.
                # ^^^^ --- what the !@#*?  serious!  -exarkun
if selectable not in fdset:
# This for pausing input when we're not ready for more.
_logrun(selectable, _drdw, selectable, method)

    doIteration = doSelect


def _doReadOrWrite(self, selectable, method):
        why = getattr(selectable, method)()
        why = sys.exc_info()[1]
    if why:
        self._disconnectSelectable(selectable, why, method == "doRead")


Experimental platform is win so here call is selectReactor

There is a package, performed in the win platform doIteration = doSelect

In the Unix platform is doIteration = doPoll

Of course, you can manually choose, as long as the platform support.


Further _select essence is a package, as long as the selector, and a final bottom are (), particularly visible twisted / internet /

    r, w, e =, w, w, timeout)

    return r, w + e, []


Here, in essence, it was returned to readable, writable queue to perform read and write.

So if the client requests a connection, doRead method will be invoked;

Here, selectable substantially Port class instance


class Port(base.BasePort, _SocketCloser):
    A TCP server port, listening for connections.

def doRead(self):
        Called when my socket is ready for reading.
        This accepts a connection and calls self.protocol() to handle the
        wire-level protocol.
            if platformType == "posix":
                numAccepts = self.numberAccepts
                # win32 event loop breaks if we do more than one accept()
                # in an iteration of the event loop.
numAccepts = 1

            with _BuffersLogs(self._logger.namespace,
                     as bufferingLogger:
                accepted = 0
                clients = _accept(bufferingLogger,

                for accepted, (skt, addr) in enumerate(clients, 1):

                    if len(addr) == 4:
                        # IPv6, make sure we get the scopeID if it
                        # exists
host = socket.getnameinfo(
                            socket.NI_NUMERICHOST | socket.NI_NUMERICSERV)
                        addr = tuple([host[0]] + list(addr[1:]))

                    protocol = self.factory.buildProtocol(
                    if protocol is None:
s = self.sessionno
                    self.sessionno = s + 1
                    transport = self.transport(
                        skt, protocol, addr, self, s, self.reactor)

            # Scale our synchronous accept loop according to traffic
            # Reaching our limit on consecutive accept calls indicates
            # there might be still more clients to serve the next time
            # the reactor calls us.  Prepare to accept some more.
if accepted == self.numberAccepts:
                self.numberAccepts += 20
            # Otherwise, don't attempt to accept any more clients than
            # we just accepted or any less than 1.
                self.numberAccepts = max(1, accepted)
        except BaseException:
            # Note that in TLS mode, this will possibly catch SSL.Errors
            # raised by self.socket.accept()
            # There is no "except SSL.Error:" above because SSL may be
            # None if there is no SSL support.  In any case, all the
            # "except SSL.Error:" suite would probably do is log.deferr()
            # and return, so handling it here works just as well.



clients = _accept(bufferingLogger,range(numAccepts),self.socket,_reservedFD)

Returns the client, addr.

def _accept(logger, accepts, listener, reservedFD):
    Return a generator that yields client sockets from the provided
    listening socket until there are none left or an unrecoverable
    error occurs.


    client, address = listener.accept()


yield client, address


transport = self.transport(skt, protocol, addr, self, s, self.reactor)



The actual transport is to call a method


def _makeTransport(self):
    Create a L{Client} bound to this L{Connector}.
    @return: a new L{Client}
    @rtype: L{Client}
return Client(, self.port, self.bindAddress, self, self.reactor)

In summary, the CLIENT Transport point, it is an example of Port, comprising host, port, bindaddress socket information and the like.

Further, reactor.listenTCP (8123, ChatFactory ())
It will call its def startListening (self):
In which calls self.startReading (), the port is added to the instance of the read sequence reactor, as summarized below section



class _BaseTCPClient(object):
    Code shared with other (non-POSIX) reactors for management of outgoing TCP
    connections (both TCPv4 and TCPv6).
    @note: In order to be functional, this class must be mixed into the same
        hierarchy as L{_BaseBaseClient}.  It would subclass L{_BaseBaseClient}
        directly, but the class hierarchy here is divided in strange ways out
        of the need to share code along multiple axes; specifically, with the
        IOCP reactor and also with UNIX clients in other reactors.
    @ivar _addressType: The Twisted _IPAddress implementation for this client
    @type _addressType: L{IPv4Address} or L{IPv6Address}
    @ivar connector: The L{Connector} which is driving this L{_BaseTCPClient}'s
        connection attempt.
    @ivar addr: The address that this socket will be connecting to.
    @type addr: If IPv4, a 2-C{tuple} of C{(str host, int port)}.  If IPv6, a
        4-C{tuple} of (C{str host, int port, int ignored, int scope}).
    @ivar createInternetSocket: Subclasses must implement this as a method to
        create a python socket object of the appropriate address family and
        socket type.
    @type createInternetSocket: 0-argument callable returning

_addressType = address.IPv4Address

    def __init__(self, host, port, bindAddress, connector, reactor=None):
        # BaseClient.__init__ is invoked later
self.connector = connector
        self.addr = (host, port)

        whenDone = self.resolveAddress
        err = None
skt = None

            self._requiresResolution = False
            self._requiresResolution = False
self.addr = _resolveIPv6(host, port)
            self.addressFamily = socket.AF_INET6
            self._addressType = address.IPv6Address
            self._requiresResolution = True
            skt = self.createInternetSocket()
        except socket.error as se:
            err = error.ConnectBindError(se.args[0], se.args[1])
            whenDone = None
whenDone and bindAddress is not None:
                if abstract.isIPv6Address(bindAddress[0]):
                    bindinfo = _resolveIPv6(*bindAddress)
                    bindinfo = bindAddress
            except socket.error as se:
                err = error.ConnectBindError(se.args[0], se.args[1])
                whenDone = None
self._finishInit(whenDone, skt, err, reactor)


Look again at the protocol


protocol = self.factory.buildProtocol(self._buildAddr(addr))

protocol used here is LineReceiver, look at its makeConnection

# /twisted/internet/

class BaseProtocol:
    This is the abstract superclass of all protocols.
    Some methods have helpful default implementations here so that they can
    easily be shared, but otherwise the direct subclasses of this class are more
    interesting, L{Protocol} and L{ProcessProtocol}.
connected = 0
    transport = None

makeConnection(self, transport):
        Make a connection to a transport and a server.
        This sets the 'transport' attribute of this Protocol, and calls the
        connectionMade() callback.
self.connected = 1
        self.transport = transport

Call self.connectionMade ()

This is to write the code, send a message.

def connectionMade(self):
    self.sendLine(b'Whats your name')



# /twisted/protocols/

class LineReceiver(protocol.Protocol, _PauseableMixin):
    A protocol that receives lines and/or raw data, depending on mode.
def sendLine(self, line):
        Sends a line to the other end of the connection.
        @param line: The line to send, not including the delimiter.
        @type line: C{bytes}
return self.transport.write(line + self.delimiter)


1.3.    listenTCP()

Back to the example.


DEF main ():
    # listening port
reactor.listenTCP (8123, ChatFactory ()) ()

reactor.listenTCP () registered a listener event, it is the parent class PosixReactorBase methods.

# twisted/internet/
@implementer(IReactorTCP, IReactorUDP, IReactorMulticast)
class PosixReactorBase(_SignalReactorMixin, _DisconnectSelectableMixin,
    A basis for reactors that use file descriptors.
    @ivar _childWaker: L{None} or a reference to the L{_SIGCHLDWaker}
        which is used to properly notice child process termination.
    # IReactorTCP

def listenTCP(self, port, factory, backlog=50, interface=''):
        p = tcp.Port(port, factory, backlog, interface, self)
        return p

# twisted/internet/
class Port(base.BasePort, _SocketCloser):
    A TCP server port, listening for connections.
def startListening(self):
        """Create and bind my socket, and begin listening on it.
        This is called on unserialization, and must be called after creating a
        server to begin listening on the specified port.
        if self._preexistingSocket is None:
            # Create a new socket and make it listen
                # 声明一个非阻塞socket
skt = self.createInternetSocket()
                if self.addressFamily == socket.AF_INET6:
                    addr = _resolveIPv6(self.interface, self.port)
                    addr = (self.interface, self.port)
                # 绑定
            except socket.error as le:
                raise CannotListenError(self.interface, self.port, le)
            # 监听
            # Re-use the externally specified socket
skt = self._preexistingSocket
            self._preexistingSocket = None
# Avoid shutting it down at the end.
self._shouldShutdown = False

# Make sure that if we listened on port 0, we update that to
        # reflect what the OS actually assigned us.
self._realPortNumber = skt.getsockname()[1]

        log.msg("%s starting on %s" % (
                self._getLogPrefix(self.factory), self._realPortNumber))

        # The order of the next 5 lines is kind of bizarre.  If no one
        # can explain it, perhaps we should re-arrange them.
        self.connected = True
self.socket = skt
        self.fileno = self.socket.fileno
        self.numberAccepts = 100


    def createInternetSocket(self):
        """(internal) Create a non-blocking socket using
        self.addressFamily, self.socketType.
s = socket.socket(self.addressFamily, self.socketType)
        return s

The whole logic is simple, and the server-side as normal, creates a socket, bind, listen. Except that socket descriptor added to the read set of reactor. So if the connection with the client over the words, reactor will be monitored, then the event handler is triggered.


Look self.startReading ()

# /twisted/internet/
    interfaces.IPushProducer, interfaces.IReadWriteDescriptor,
    interfaces.IConsumer, interfaces.ITransport,
class FileDescriptor(_ConsumerMixin, _LogOwner):
    An object which can be operated on by select().
    This is an abstract superclass of all objects which may be notified when
    they are readable or writable; e.g. they have a file-descriptor that is
    valid to be passed to select(2).
def startReading(self):
        """Start waiting for read availability.


def addReader(self, reader):
    Add a FileDescriptor for notification of data available to read.

In this sequence will transport added _reads

SOCKET and then the corresponding message will be treated at each select, is to call the doRead transport () method in class

# twisted/internet/
@implementer(interfaces.ITCPTransport, interfaces.ISystemHandle)
class Connection(_TLSConnectionMixin, abstract.FileDescriptor, _SocketCloser,

    def doRead(self):
            # 接收数据
data = self.socket.recv(self.bufferSize)
        except socket.error as se:
            if se.args[0] == EWOULDBLOCK:
                return main.CONNECTION_LOST

        return self._dataReceived(data)

    def _dataReceived(self, data):
        if not data:
            return main.CONNECTION_DONE
        # 调用我们自定义protocol的dataReceived方法处理数据
rval = self.protocol.dataReceived(data)
        if rval is not None:
            offender = self.protocol.dataReceived
            warningFormat = (
                'Returning a value other than None from %(fqpn)s is '
                'deprecated since %(version)s.'
            warningString = deprecate.getDeprecationWarningString(
                offender, versions.Version('Twisted', 11, 0, 0),
            deprecate.warnAboutFunction(offender, warningString)
        return rval

In which the calling custom protocol

rval = self.protocol.dataReceived(data)


At this point, the whole cycle be run up.


2. Other

2.1. implementer(IReactorFDSet)

This is a decorator interface, concrete and tangible

# twisted/internet/
class SelectReactor(posixbase.PosixReactorBase, _extraBase)

implementer expressed SelectReactor implements methods IReactorFDSet interface used here zope.interface , it is the interface in python.

IReactorFDSet interfaces mainly acquisition descriptor, add, or delete method of operation. These methods will be able to see the name to know the meaning of, Skip.

# twisted/internet/
class IReactorFDSet(Interface):
    def addReader(reader):
    def addWriter(writer):
    def removeReader(reader):
    def removeWriter(writer):
    def removeAll():
    def getReaders():
    def getWriters():


3. Summary

Simply described, then, twisted to create two socket, respectively, port number port, port + 1, select the use of the system (or other IO multiplexing function) as a polling body;

Then, the upper layer processor encapsulated form and protocol factory, reactor cycle responsible.

PS: do not read the code architecture diagram of the pit too. . . . .


Guess you like