foreword
In the previous article, we sorted out the main process structure of Influxdb's code. From this article, we will analyze various services in Influxdb. Due to space limitations, this article will first introduce MonitorService. In the above, we know that Influxdb registers the Service in the following way.
//cmd/influxd/run/server.go 371
s.appendMonitorService()
s.appendPrecreatorService(s.config.Precreator)
s.appendSnapshotterService()
s.appendContinuousQueryService(s.config.ContinuousQuery)
s.appendHTTPDService(s.config.HTTPD)
s.appendStorageService(s.config.Storage)
s.appendRetentionPolicyService(s.config.Retention)And call it sequentially through the following code
//cmd/influxd/run/server.go 434
for _, service := range s.Services {
if err := service.Open(); err != nil {
return fmt.Errorf("open service: %s", err)
}
}Then we can start from here and strip each Service.
MonitorService resolution
Let's first look at the core code of MonitorService. We can find that the initialization of MonitorService is more to ensure that MonitorService is a singleton in the running state, and some diagnostic instances are registered. Finally, it is judged whether the configuration supports storage. If it supports storage, goroutine will be started. Periodically store data asynchronously in TSM Engine.
//influxdb/monitor/service.go 73行
// New returns a new instance of the monitor system.
func New(r Reporter, c Config) *Monitor {
return &Monitor{
globalTags: make(map[string]string),
diagRegistrations: make(map[string]diagnostics.Client),
reporter: r,
storeEnabled: c.StoreEnabled,
storeDatabase: c.StoreDatabase,
storeInterval: time.Duration(c.StoreInterval),
storeRetentionPolicy: MonitorRetentionPolicy,
Logger: zap.NewNop(),
}
}
// open returns whether the monitor service is open.
func (m *Monitor) open() bool {
m.mu.Lock()
defer m.mu.Unlock()
return m.done != nil
}
// Open opens the monitoring system, using the given clusterID, node ID, and hostname
// for identification purpose.
func (m *Monitor) Open() error {
if m.open() {
m.Logger.Info("Monitor is already open")
return nil
}
m.Logger.Info("Starting monitor service")
// Self-register various stats and diagnostics.
m.RegisterDiagnosticsClient("build", &build{
Version: m.Version,
Commit: m.Commit,
Branch: m.Branch,
Time: m.BuildTime,
})
m.RegisterDiagnosticsClient("runtime", &goRuntime{})
m.RegisterDiagnosticsClient("network", &network{})
m.RegisterDiagnosticsClient("system", &system{})
m.mu.Lock()
m.done = make(chan struct{})
m.mu.Unlock()
// If enabled, record stats in a InfluxDB system.
if m.storeEnabled {
hostname, _ := os.Hostname()
m.SetGlobalTag("hostname", hostname)
// Start periodic writes to system.
m.wg.Add(1)
go m.storeStatistics()
}
return nil
}The following focuses on what is done in storeStatistics. In storeStatistics, stats are mainly obtained through the Statistics method, and then written to the corresponding store through the writePoints method.
//influxdb/monitor/service.go 418行
func (m *Monitor) storeStatistics() {
defer m.wg.Done()
m.Logger.Info("Storing statistics", logger.Database(m.storeDatabase), logger.RetentionPolicy(m.storeRetentionPolicy), logger.DurationLiteral("interval", m.storeInterval))
// Wait until an even interval to start recording monitor statistics.
// If we are interrupted before the interval for some reason, exit early.
if err := m.waitUntilInterval(m.storeInterval); err != nil {
return
}
tick := time.NewTicker(m.storeInterval)
defer tick.Stop()
for {
select {
case now := <-tick.C:
now = now.Truncate(m.storeInterval)
func() {
m.mu.Lock()
defer m.mu.Unlock()
m.createInternalStorage()
}()
stats, err := m.Statistics(m.globalTags)
if err != nil {
m.Logger.Info("Failed to retrieve registered statistics", zap.Error(err))
return
}
// Write all stats in batches
batch := make(models.Points, 0, 5000)
for _, s := range stats {
pt, err := models.NewPoint(s.Name, models.NewTags(s.Tags), s.Values, now)
if err != nil {
m.Logger.Info("Dropping point", zap.String("name", s.Name), zap.Error(err))
return
}
batch = append(batch, pt)
if len(batch) == cap(batch) {
m.writePoints(batch)
batch = batch[:0]
}
}
// Write the last batch
if len(batch) > 0 {
m.writePoints(batch)
}
case <-m.done:
m.Logger.Info("Terminating storage of statistics")
return
}
}
}What is more interesting is that the Statistics method not only returns monitoring indicators of type []*Statistic, but also exposes the data through the API through expvar. The Go standard library has an expvar package. This package exposes metrics from your application and the Go runtime via a JSON-formatted HTTP API, somewhat similar to the data format that prometheus' API scrapes. The data source of MonitorService is obtained through the runtime package. Importantly include related system metrics such as memory and goroutines.
//influxdb/monitor/service.go 324行
statistic.Values = map[string]interface{}{
"Alloc": int64(rt.Alloc),
"TotalAlloc": int64(rt.TotalAlloc),
"Sys": int64(rt.Sys),
"Lookups": int64(rt.Lookups),
"Mallocs": int64(rt.Mallocs),
"Frees": int64(rt.Frees),
"HeapAlloc": int64(rt.HeapAlloc),
"HeapSys": int64(rt.HeapSys),
"HeapIdle": int64(rt.HeapIdle),
"HeapInUse": int64(rt.HeapInuse),
"HeapReleased": int64(rt.HeapReleased),
"HeapObjects": int64(rt.HeapObjects),
"PauseTotalNs": int64(rt.PauseTotalNs),
"NumGC": int64(rt.NumGC),
"NumGoroutine": int64(runtime.NumGoroutine()),
}We can look at the content stored in the TSM Engine
Careful students may have found that the _internal library also contains a lot of other monitoring data, so where do these data come from? In fact, in the above, we ignored an important object, Reportor, when instantiating Monitor Pass an instance r that implements the Reporter interface. The Reporter interface defines a Statistics method. We can find that QueryExecutor, TSDBStore, PointsWriter, Subscriber and other core components implement the Statistics method.
// cmd/influxd/run/server.go 210行
// Statistics returns statistics for the services running in the Server.
func (s *Server) Statistics(tags map[string]string) []models.Statistic {
var statistics []models.Statistic
statistics = append(statistics, s.QueryExecutor.Statistics(tags)...)
statistics = append(statistics, s.TSDBStore.Statistics(tags)...)
statistics = append(statistics, s.PointsWriter.Statistics(tags)...)
statistics = append(statistics, s.Subscriber.Statistics(tags)...)
for _, srv := range s.Services {
if m, ok := srv.(monitor.Reporter); ok {
statistics = append(statistics, m.Statistics(tags)...)
}
}
return statistics
}In the gatherStatistics method, the reporter will be called, and the corresponding data will be merged with the data of the diagnosticClient such as runtime, and written to Influxdb.
//influxdb/monitor/service.go 347行
func (m *Monitor) gatherStatistics(statistics []*Statistic, tags map[string]string) []*Statistic {
m.mu.RLock()
defer m.mu.RUnlock()
if m.reporter != nil {
for _, s := range m.reporter.Statistics(tags) {
statistics = append(statistics, &Statistic{Statistic: s})
}
}
return statistics
}Summarize
At this point, the monitoring service part of Influxdb is relatively clear. First, monitoring is divided into system-level monitoring and component-level monitoring. System-level monitoring will be obtained through runtime, and component-level monitoring will be implemented through the implementation of the Reportor interface by their respective components. , Influxdb will periodically store monitoring data to the _internal library of TSM Engine. Influxdb also exposes monitoring data to the API through expvar.