Controller analysis of SharedIndexInformer (k8s client-go)

Preface

Controller, the Chinese translation is a controller. The Controller of this article is the controller of SharedIndexInformer, not the controller of kube-controller-manager, so don't confuse the concept, although they are called controllers.

Since it is called a controller, what does it control? Remember this classic picture?

Controller is responsible for the process of taking over Deltas from Reflector and sending them to DeltaFIFO. It seems that it does control the flow of Deltas. The red controller in this picture is kube-controller, not the Controller of this article.

This article uses the release-1.20 branch of kubenete, the latest Kubernetes version document link: https://github.com/jindezgm/k8s-src-analysis/blob/master/client-go/tools/cache/Controller.md .

Controller definition

Source connection: https://github.com/kubernetes/client-go/blob/release-1.20/tools/cache/controller.go#L98

type Controller interface {
    // Run主要做两件事情：1）创建并运行Reflector，从Config.WatcherLister列举/监视并发送到Config.Queue，可能周期性的调用Queue.Resync()
    // 2）持续从Queue弹出对象增量并调用Config.ProcessFunc。这两个事情都是通过独立的协程运行直到stopC关闭
    // 关于Config后文有注释
    Run(stopCh <-chan struct{})

    // 这个函数会给SharedIndexInformer使用，目的是判断当前是否已经同步完成了。同步的意思就是至少完成了一次全量列举
    HasSynced() bool

    // 获取上一次同步的版本，这个版本是列举全量对象的最大版本号
    LastSyncResourceVersion() string
}

Purely from the definition of Controller, it does play the role of the controller, at least it is driving the flow of data, including full enumeration and monitoring increments that have been synchronized regularly.

Controller implementation

Config

Config is the configuration of the Controller, which is used when creating the Controller to configure the Controller. Source connection: https://github.com/kubernetes/client-go/blob/release-1.20/tools/cache/controller.go#L39

type Config struct {
    // 队列，当前实现为DeltaFIFO
    Queue

    // ListerWatcher用来创建Reflector
    ListerWatcher

    // 处理从Queue中弹出的增量(Deltas)的函数，ProcessFunc下面有注释
    Process ProcessFunc

    // 对象类型，比如v1.Pod，说明SharedIndexInformer是每个API类型都需要创建一个
    ObjectType runtime.Object

    // 在同步的周期，再同步不是同步apiserver，是Queue与Indexer、ResourceEventHandler之间的在同步
    FullResyncPeriod time.Duration

    // 是否需要再同步的函数，如果没有任何ResourceEventHandler设置resyncPeriod，则不需要在同步
    // ShouldResyncFunc下面有注释
    ShouldResync ShouldResyncFunc

    // 如果为true，当调用Process()返回错误时，重新放回Queue
    RetryOnError bool

    // 用来创建Reflector，每个Reflector.ListAndWatch()断开连接并出现错误时调用这个函数
    WatchErrorHandler WatchErrorHandler

    // 初始化监视列表和重新列举全量对象的请求块大小，说白了就是用来分页列举的，与SQL中的limit类似
    WatchListPageSize int64
}

// ShouldResyncFunc是一种函数类型，该类型的函数需要告知是否需要再同步，函数实际指向了sharedProcessor.shouldResync().
// 关于sharedProcessor.shouldResync()请参看SharedIndexInformer的文档
type ShouldResyncFunc func() bool

// ProcessFunc从定义上看是处理一个对象的函数类型，函数实际指向了sharedIndexInformer.HandleDeltas()，所以对象就是Deltas
// 关于sharedIndexInformer.HandleDeltas()请参看SharedIndexInformer的文档
type ProcessFunc func(obj interface{}) error

From the member variables of Config, it can be seen that Controller needs ListerWatcher, Queue, ProcessFunc, which is basically the data flow of SharedIndexInformer, from ListerWatcher->Queue->ProcessFunc, so it is called the controller worthy of its name.

controller

Controller is the realization of Controller. This way of definition is already an unwritten rule of golang. Source connection: https://github.com/kubernetes/client-go/blob/release-1.20/tools/cache/controller.go#L89

type controller struct {
    // Config不用多说了，上一个章节介绍过了
    config         Config
    // Reflector请参看Reflector的文档
    reflector      *Reflector
    // 后面两个应该没什么好解释的了
    reflectorMutex sync.RWMutex
    clock          clock.Clock
}

From the definition of controller, it is basically equal to Config+Reflector, and it cannot be more.

Run() interface implementation

Run() is a very core function of the controller, because the core functions of the controller are all started here, that is to say, there is not much work done in the constructor of the controller, and the reader can understand it by himself. Because the controller is the core of SharedIndexInformer , controller.Run() must be executed in SharedIndexInformer.Run().

Source connection: https://github.com/kubernetes/client-go/blob/release-1.20/tools/cache/controller.go#L127

func (c *controller) Run(stopCh <-chan struct{}) {
    defer utilruntime.HandleCrash()
    // 如果收到停止信号，需要把Queue关闭掉，此处有一个疑问：有必要卡一个协程等待信号然后关闭Queue么？
    // 直接在函数的结尾关闭不行么？毕竟函数需要等待停止型号并且Run()退出后才结束。
    // 其实在DeltaFIFO的文档中介绍过了，DeltaFIFO.Pop()通过sync.Cond阻塞协程，
    // 此时stopCh关闭也不会激活阻塞的协程，除非有新的对象或者关闭Queue
    // 所以常见一个协程关闭Queue是非常有必要的，否则就和controller.processLoop(后面章节会介绍)形成死锁了
    go func() {
        <-stopCh
        c.config.Queue.Close()
    }()
    // 创建Reflector，需要注意的是传入了ListerWatcher、Queue、FullResyncPeriod，在Reflector文章中提到了：
    // 1.通过ListerWatcher同步apiserver的对象；
    // 2.定期的调用Queue.Resync()；
    r := NewReflector(
        c.config.ListerWatcher,
        c.config.ObjectType,
        c.config.Queue,
        c.config.FullResyncPeriod,
    )
    // 感情Config中的大部分参数都是给Reflector的...
    r.ShouldResync = c.config.ShouldResync
    r.WatchListPageSize = c.config.WatchListPageSize
    r.clock = c.clock
    if c.config.WatchErrorHandler != nil {
        r.watchErrorHandler = c.config.WatchErrorHandler
    }

    // 记录Reflector
    c.reflectorMutex.Lock()
    c.reflector = r
    c.reflectorMutex.Unlock()

    var wg wait.Group
    // 启动协程运行Reflector.Run()
    wg.StartWithChannel(stopCh, r.Run)
    // 启动协程运行processLoop()函数，虽然wait.Util()是每一秒执行一次processLoop()，但是processLoop()内部是一个死循环直到Queue关闭。
    // 初期的设计应该是遇到错误一秒后再重试，直到收到停止信号，现在来看只有Queue关闭的错误processLoop()才会退出。
    // 其他的错误当Config.RetryOnError为true时，会重新放入Queue，否则就丢弃，所以用wait.Until()当前的版本来看没有意义。难道Queue关闭了还会重新创建不成？
    wait.Until(c.processLoop, time.Second, stopCh)
    // 当前来看，只等待运行Reflector.Run()函数的协程退出
    wg.Wait()
}

Other interface implementation

Source connection: https://github.com/kubernetes/client-go/blob/release-1.20/tools/cache/controller.go#L158

// HasSynced实现了Controller.HasSynced()，无非是对Queue.HasSynced()的在封装
func (c *controller) HasSynced)() bool {
    return c.config.Queue.HasSynced()
}

// LastSyncResourceVersion无非是对Reflector.LastSyncResourceVersion()的再封装
func (c *controller) LastSyncResourceVersion() string {
    c.reflectorMutex.RLock()
    defer c.reflectorMutex.RUnlock()
    if c.reflector == nil {
        return ""
    }
    return c.reflector.LastSyncResourceVersion()
}

Controller's core processing function processLoop()

Source connection: https://github.com/kubernetes/client-go/blob/release-1.20/tools/cache/controller.go#L181

// processLoop才是controller真正所事情的函数
func (c *controller) processLoop() {
    for {
        // 从Queue中弹出对象并交给Config.ProcessFunc()处理
        obj, err := c.config.Queue.Pop(PopProcessFunc(c.config.Process))
        if err != nil {
            // 如果是队列关闭错误，直接退出，因为队列关闭可能收到了停止信号，所以需要退出
            // 如果没有收到停止信号关闭了Queue也没问题，processLoop是周期性调用的，1秒过后还会被调用
            if err == ErrFIFOClosed {
                return
            }
            // 如果配置了错误重试，那么就把对象重新放回队列
            if c.config.RetryOnError {
                c.config.Queue.AddIfNotPresent(obj)
            }
        }
    }
}

Just that's a short answer, is there a feeling of saying that you are the core but not doing anything?

to sum up

1. Controller is the controller of SharedIndexInformer, which is the core module. The control API object is from WatcherLister->Queue->ProcessFunc;
2. Although Controller seems to control the entire data flow, WatcherLister->Queue is implemented by Reflector, and Controller is only responsible for creating Reflector and running it;
3. Controller actually only implements Queue->ProcessFunc;
4. Controllers everywhere are all controllers. The key point depends on the control. Don't confuse it with kube-controller. Be careful in naming! I have a former colleague who likes to name various Managers, and I also take it...