Design and implementation of muduo library learning 04-EventLoop::runInLoop()

Dongyang's study notes

一、EventLoop::runInLoop()

runInLoop: Execute a user task callback in the IO thread.

• If the user calls this function in the current IO thread, the callback will be performed synchronously;
• If the user calls in other threads, cb will be added to the queue, and the IO thread will be awakened to call this Functor
• This feature allows us 轻易地跨线程调配任务, and不用加锁

void EventLoop::runInLoop(const Functor& cb)
{
     
     
 if (isInLoopThread())
 {
     
     
   cb();
 }
 else                   // 不在当前loop，则加入队列，并在
 {
     
     
   queueInLoop(cb);
 }
}

Second, how to wake up the IO thread to let it execute the newly added callback

The IO thread is usually blocked in the event loop EventLoop::loop() 中 poll(2)调用中. In order to make the IO thread execute the newly added user callback immediately, we need to try to wake it up.

• The traditional method is to use pipe(2). The IO thread always monitors the readable event of this pipeline. When it needs to wake up, other threads write a byte into the pipeline, so that the IO thread returns from the IO multiplexing blocking call. (HTTP long polling?)
• Now that Linux has it eventfd(2), it can wake up more efficiently because it does not have to manage buffers.

3. New members of EventLoop

• wakeupChannel_ for handling readable events on wakeupFd_, the event will be distributed handleRead()function. Which only pendingFunctors_exposed to other threads, so it is necessary mutexto protect

private:

  void abortNotInLoopThread();
+ void handleRead();  // waked up
+ void doPendingFunctors();

  typedef std::vector<Channel*> ChannelList;

  bool looping_; /* atomic */
  bool quit_; /* atomic */
+ bool callingPendingFunctors_; /* atomic */
  const pid_t threadId_;
  Timestamp pollReturnTime_;
  boost::scoped_ptr<Poller> poller_;
  boost::scoped_ptr<TimerQueue> timerQueue_;
+ int wakeupFd_;
  // unlike in TimerQueue, which is an internal class,
  // we don't expose Channel to client.
+ boost::scoped_ptr<Channel> wakeupChannel_;
  ChannelList activeChannels_;
+ MutexLock mutex_;
+ std::vector<Functor> pendingFunctors_; // @GuardedBy mutex_
};

3.1 EventLoop::queueInLoop()

Put cb into the queue and 必要时wake up the IO thread.
When necessary, there are two situations:

• If the caller is not an IO thread, then wakeup is necessary
• If it is called in the IO thread, but it is being called at the same time pending functor, it must also be awakened 不然新加的 cb 不能被及时调用(the reason for swap)

void EventLoop::queueInLoop(const Functor& cb)
{
     
     
{
     
     
MutexLockGuard lock(mutex_);
pendingFunctors_.push_back(cb);
}

if (!isInLoopThread() || callingPendingFunctors_)
{
     
     
  wakeup();
}
}

Note the call point of doPendingFunctors() below:

• Add a line of code to loop() to execute the task callback in pendingFunctors_.

 while (!quit_)
 {
     
     
   activeChannels_.clear();
   pollReturnTime_ = poller_->poll(kPollTimeMs, &activeChannels_);
   for (ChannelList::iterator it = activeChannels_.begin();
       it != activeChannels_.end(); ++it)
   {
     
     
     (*it)->handleEvent();
   }
+  doPendingFunctors();
 }

3.2 EventLoop::doPendingFunctors()

doPendingFunctors() does not call Functor in sequence in the critical section. Instead, swap() the callback list into the local variable functors.

• On the one hand 减小了临界区的长度(meaning it will not block)
• On the other hand 避免了死锁(because Functor may call queueInLoop again)
• 没有反复执行 Until pendingFunctors_ is empty, this is intentional, otherwise the IO thread may fall into an infinite loop.

void EventLoop::doPendingFunctors()
{
     
     
	//
 std::vector<Functor> functors;
 callingPendingFunctors_ = true;

 {
     
     
 MutexLockGuard lock(mutex_);
 functors.swap(pendingFunctors_);
 }

 for (size_t i = 0; i < functors.size(); ++i)
 {
     
     
   functors[i]();
 }
 callingPendingFunctors_ = false;
}

3.3 EventLoop::quit()

When non-IO thread quit(), wakeup() is required.

• Why is wakeup not needed in the IO thread quit?

void EventLoop::quit()
{
     
     
quit_ = true;
+ if (!isInLoopThread())
+ {
     
     
+  wakeup();
+ }
}

3.4 EventLoop::wakeup() 和 EventLoop::handleRead()

Wakeup() and handleRead() in EventLoop write data and read data to wakeupFd_ respectively, the code is omitted.

Fourth, improve the thread safety of TimerQueue

As mentioned earlier, TimerQueue::addTimer() can only be called on the IO thread, therefore EventLoop::runAfer() 系列函数不是线程安全的.

• Calling EventLoop::runAfter() in a non-IO thread will cause the program to crash

void threadFunc()
{
     
     
	g_loop->runAfter(1.0, print);
}

int main()
{
     
     
	muduo::EventLoop loop;
	g_loop = &loop;
	muduo::Thread t(threadFunc);
	t.start()
	loop.loop();
}

4.1 Thread-safe addTimer()

With EventLoop::runInLoop(), we can easily make TimerQueue::addTimer() thread-safe,而且无须用锁

• Split addTimer() into two parts:
  • addTimer()(After splitting): Only responsible for forwarding
  • addTimerInLoop(): Complete the task of modifying the timer

In this way, it is safe to call in which thread.

TimerId TimerQueue::addTimer(const TimerCallback& cb,
                            Timestamp when,
                            double interval)
{
     
     
 Timer* timer = new Timer(cb, when, interval);
+  loop_->runInLoop(
+     boost::bind(&TimerQueue::addTimerInLoop, this, timer));
+ return TimerId(timer);
+}

void TimerQueue::addTimerInLoop(Timer* timer)
+{
     
     
 loop_->assertInLoopThread();
 bool earliestChanged = insert(timer);

 if (earliestChanged)
 {
     
     
   resetTimerfd(timerfd_, timer->expiration());
 }
}

五、EventLoopThread class

The IO thread is not necessarily the main thread, we can create and run EventLoop in any thread.

• A thread can also have more than one IO thread
• We can assign different sockets to different IO threads according to priority,避免优先级反转。

5.1 EventLoopThread::startLoop()

EventLoopThread will start its own thread and run EventLoop::loop() in it. The key startLoop() is defined as follows:

• This function will return to the new thread EventLoop对象的地址
• 这里是隐式返回的（使用了 Thread类）

EventLoop* EventLoopThread::startLoop()
{
     
     
 assert(!thread_.started());
 thread_.start();

 {
     
     
   MutexLockGuard lock(mutex_);
   while (loop_ == NULL)
   {
     
     
     cond_.wait();
   }
 }

 return loop_;
}

5.2 EventLoopThread::threadFunc()

The thread main function defines the EventLoop object on the stack, then assigns its address to the loop_ member variable, and finally notify() wakes up startLoop().

• The lifetime of EventLoop is the same as the scope of the thread's main function, so this pointer becomes invalid after threadFunc() exits.
• However, the server generally does not require normal exit, which is not a big problem

void EventLoopThread::threadFunc()
{
     
     
 EventLoop loop;

 {
     
     
   MutexLockGuard lock(mutex_);
   loop_ = &loop;
   cond_.notify();
 }

 loop.loop();
 //assert(exiting_);
}