1 Overview:
1.1 Source code environment
The version information is as follows:
a. Thanos component version: v0.16.0
1.2 The role of Thanos Query
The Thanos Query component is an http server + grpc server. Its data source is a downstream component that implements STORE API (such as Thanos Sidecar component, Thanos Store component, Thanos Ruler component), and it also implements the official HTTP API of Prometheus. After the Thanos Query component obtains the data from the downstream, it can perform operations such as merging and de-duplication, and finally return the result to the external client. Therefore, Thanos Query is the role of database middleware.
2 Brief analysis of source code:
Use the github.com/oklog/run package to start a set of coroutines. The logic of these coroutines is mainly to start the http server, grpc server, and dynamically discover downstream components that implement the STORE API.
2.1 main method
The startup command format of Thanos is as follows, the format starts with thanos (because it is the same executable binary file). Which component to start depends on the first parameter, which is query in this example, so this command is the logic to start the query component.
thanos query \
--log.level=debug \
--query.auto-downsampling \
--grpc-address=0.0.0.0:10901 \
--http-address=0.0.0.0:9090 \
--query.partial-response \
--query.replica-label=prometheus_replica \
--query.replica-label=rule_replica \
--store=dnssrv+_grpc._tcp.prometheus-headless.thanos.svc.cluster.local \
--store=dnssrv+_grpc._tcp.thanos-rule.thanos.svc.cluster.local \
--store=dnssrv+_grpc._tcp.thanos-store.thanos.svc.cluster.local
Let's take a look at the main method in detail. Create an app object. The app object contains the startup functions of all Thanos components, but only one function is taken from the map to start when it is actually started. Which function is taken out depends on the startup command.
func main() {
/*
其他代码
*/
app := extkingpin.NewApp(kingpin.New(filepath.Base(os.Args[0]), "A block storage based long-term storage for Prometheus").Version(version.Print("thanos")))
/*
其他代码
*/
// 把所有组件的启动逻辑都放进app对象中的setups列表中
registerSidecar(app)
registerStore(app)
registerQuery(app)
registerRule(app)
registerCompact(app)
registerTools(app)
registerReceive(app)
registerQueryFrontend(app)
// 根据命令行的信息，从app对象的setups列表中取出一个组件逻辑
cmd, setup := app.Parse()
logger := logging.NewLogger(*logLevel, *logFormat, *debugName)
/*
其他代码
*/
var g run.Group
var tracer opentracing.Tracer
/*
tracing相关的代码
*/
reloadCh := make(chan struct{}, 1)
// 启动特定的一个组件(sidecar、query、store等组件中的一种)，底层还是执行g.Add(...)
if err := setup(&g, logger, metrics, tracer, reloadCh, *logLevel == "debug"); err != nil {
os.Exit(1)
}
// 监听来自系统的杀死信号.
{
cancel := make(chan struct{})
g.Add(func() error {
return interrupt(logger, cancel)
}, func(error) {
close(cancel)
})
}
// 监听来配置重载的信号
{
cancel := make(chan struct{})
g.Add(func() error {
return reload(logger, cancel, reloadCh)
}, func(error) {
close(cancel)
})
}
// 阻塞地等待所有协程中的退出
// 有一个协程返回，其他协程也会返回
if err := g.Run(); err != nil {
level.Error(logger).Log("err", fmt.Sprintf("%+v", errors.Wrapf(err, "%s command failed", cmd)))
os.Exit(1)
}
// 到达此处，说明整个程序结束了。
level.Info(logger).Log("msg", "exiting")
}
2.2 registerQuery method
func registerQuery(app *extkingpin.App) {
cmd := app.Command(comp.String(), "query node exposing PromQL enabled Query API with data retrieved from multiple store nodes")
/*
解析命令行参数
secure := cmd.Flag("grpc-client-tls-secure", "Use TLS when talking to the gRPC server").Default("false").Bool()
等等诸如此类
*/
//Setup()的入参方法，会被放入app对象的setups列表中
//最核心的是runQuery()方法
cmd.Setup(func(g *run.Group, logger log.Logger, reg *prometheus.Registry, tracer opentracing.Tracer, _ <-chan struct{}, _ bool) error {
return runQuery(
g,
logger,
reg,
tracer,
*requestLoggingDecision,
*grpcBindAddr,
time.Duration(*grpcGracePeriod),
*grpcCert,
*grpcKey,
*grpcClientCA,
/*
其他代码
*/
)
)
}
2.3 runQuery method
Use the run.Group object to start the http server, grpc server, and service discovery coroutine.
func runQuery(
g *run.Group, //其实来自main()方法
logger log.Logger,
reg *prometheus.Registry,
tracer opentracing.Tracer,
requestLoggingDecision string,
grpcBindAddr string,
grpcGracePeriod time.Duration,
grpcCert string,
grpcKey string,
grpcClientCA string,
/*
其他代码
*/
) error {
var (
// stores对象的类型StoreSet。它包含了一组store组件（位于下游的实现Store API的组件），这一组store组件是可以动态变化的
/*
type StoreSet struct {
//其他属性
stores map[string]*storeRef
}
*/
stores = query.NewStoreSet(...)
// proxy对象，即下游的Store API组件的代理
// 下游的Store API组件的列表，其实就是构造方法的入参stores.Get这个方法来获取
proxy = store.NewProxyStore(logger, reg, stores.Get, component.Query, selectorLset, storeResponseTimeout)
rulesProxy = rules.NewProxy(logger, stores.GetRulesClients)
/*
queryableCreator是一个方法，用于创建一个querier结构体对象;
querier结构体的属性proxy就是proxy对象，它包含了一组会动态变化的thanos store组件(动态变化是因为启动了一些额外的专门的协程来动态地修改这个切片);
*/
queryableCreator = query.NewQueryableCreator(
logger,
extprom.WrapRegistererWithPrefix("thanos_query_", reg),
proxy,
maxConcurrentSelects,
queryTimeout,
)
/*
这一段代码都是启动一些协程，定时发现和动态发现Store API组件的变化，随即更新stores对象中的类型为map[string]*storeRef的属性
*/
// 创建http server，注册http handler，并启动server
{
router := route.New()
//新建QueryAPI结构体对象
api := v1.NewQueryAPI(
logger,
stores,
engine,
queryableCreator,
rules.NewGRPCClientWithDedup(rulesProxy, queryReplicaLabels),
enableAutodownsampling,
enableQueryPartialResponse,
enableRulePartialResponse,
queryReplicaLabels,
flagsMap,
instantDefaultMaxSourceResolution,
defaultMetadataTimeRange,
gate.New(
extprom.WrapRegistererWithPrefix("thanos_query_concurrent_", reg),
maxConcurrentQueries,
),
)
// 为router对象注册http方法
api.Register(router.WithPrefix("/api/v1"), tracer, logger, ins, logMiddleware)
srv := httpserver.New(logger, reg, comp, httpProbe,
httpserver.WithListen(httpBindAddr),
httpserver.WithGracePeriod(httpGracePeriod),
)
// http服务器使用router对象
srv.Handle("/", router)
g.Add(func() error {
statusProber.Healthy()
// 启动http server
return srv.ListenAndServe()
}, func(err error) {
statusProber.NotReady(err)
defer statusProber.NotHealthy(err)
srv.Shutdown(err)
})
}
// 创建gprc server，注册grpc handler，并启动server
{
tlsCfg, err := tls.NewServerConfig(log.With(logger, "protocol", "gRPC"), grpcCert, grpcKey, grpcClientCA)
if err != nil {
return errors.Wrap(err, "setup gRPC server")
}
s := grpcserver.New(logger, reg, tracer, comp, grpcProbe,
grpcserver.WithServer(store.RegisterStoreServer(proxy)), // 注册grpc handler
grpcserver.WithServer(rules.RegisterRulesServer(rulesProxy)), // 注册grpc handler
grpcserver.WithListen(grpcBindAddr),
grpcserver.WithGracePeriod(grpcGracePeriod),
grpcserver.WithTLSConfig(tlsCfg),
)
g.Add(func() error {
statusProber.Ready()
// 启动grpc server
return s.ListenAndServe()
}, func(error) {
statusProber.NotReady(err)
s.Shutdown(err)
})
}
// 至此，http server和grpc server都启动了。
level.Info(logger).Log("msg", "starting query node")
return nil
)
}
2.4 QueryAPI structure and its methods
// QueryAPI is an API used by Thanos Query.
type QueryAPI struct {
baseAPI *api.BaseAPI
logger log.Logger
gate gate.Gate
// 构造方法，用于创建一个querier结构体对象
queryableCreate query.QueryableCreator
queryEngine *promql.Engine
ruleGroups rules.UnaryClient
/*
其他代码
*/
replicaLabels []string
storeSet *query.StoreSet
}
func (qapi *QueryAPI) Register(r *route.Router, tracer opentracing.Tracer, logger log.Logger, ins extpromhttp.InstrumentationMiddleware, logMiddleware *logging.HTTPServerMiddleware) {
qapi.baseAPI.Register(r, tracer, logger, ins, logMiddleware)
instr := api.GetInstr(tracer, logger, ins, logMiddleware)
/*
其他代码
*/
// 把qapi.query、qapi.series、 qapi.stores注册到入参r，从而完成http handler的注册
// 不管是/query接口和/series接口，每次请求到达都会创建querier对象，而querier对象内含了一组的Store API组件
r.Get("/query", instr("query", qapi.query))
r.Get("/series", instr("series", qapi.series))
r.Get("/stores", instr("stores", qapi.stores))
}
Take a look at qapi.series.
//返回指标数据
func (qapi *QueryAPI) series(r *http.Request) (interface{}, []error, *api.ApiError) {
/*
其他代码
*/
// 创建一个querier对象
// querier对象的属性proxy则包含了一组thanos store组件
q, err := qapi.queryableCreate(enableDedup, replicaLabels, storeDebugMatchers, math.MaxInt64, enablePartialResponse, true).
Querier(r.Context(), timestamp.FromTime(start), timestamp.FromTime(end))
/*
其他代码
*/
var (
metrics = []labels.Labels{}
sets []storage.SeriesSet
)
for _, mset := range matcherSets {
// 调用querier对象的Select()方法获取指标
sets = append(sets, q.Select(false, nil, mset...))
}
set := storage.NewMergeSeriesSet(sets, storage.ChainedSeriesMerge)
for set.Next() {
metrics = append(metrics, set.At().Labels())
}
return metrics, set.Warnings(), nil
}
2.5 querier structure and its method
The Querier interface (github.com/prometheus/prometheus/storage/interface.go) is implemented. The core method of this interface is Select(...). This method is used in interfaces such as /query and /series.
type querier struct {
ctx context.Context
logger log.Logger
cancel func()
mint, maxt int64
replicaLabels map[string]struct{}
storeDebugMatchers [][]*labels.Matcher
// proxy包含了一组动态的thanos store组件
proxy storepb.StoreServer
deduplicate bool
maxResolutionMillis int64
partialResponse bool
skipChunks bool
selectGate gate.Gate
selectTimeout time.Duration
}
func (q *querier) Select(_ bool, hints *storage.SelectHints, ms ...*labels.Matcher) storage.SeriesSet {
/*
其他代码
*/
promise := make(chan storage.SeriesSet, 1)
go func() {
defer close(promise)
var err error
/*
其他代码
*/
//获取到指标数据
set, err := q.selectFn(ctx, hints, ms...)
if err != nil {
// 把错误送至管道，并退出本协程
promise <- storage.ErrSeriesSet(err)
return
}
//将指标数据送至管道
promise <- set
}()
// 返回指标的封装
return &lazySeriesSet{
create: func() (storage.SeriesSet, bool) {
/*
其他代码
*/
// 从管道中读取指标
set, ok := <-promise
return set, set.Next()
}
}
}
// 获取指标，调用的是属性proxy的Series(...)方法
func (q *querier) selectFn(ctx context.Context, hints *storage.SelectHints, ms ...*labels.Matcher) (storage.SeriesSet, error) {
/*
其他代码
*/
// seriesServer结构体重写了Send()方法，在Sender()方法中将gprc返回的数据数据存储到它的seriesSet属性
resp := &seriesServer{ctx: ctx}
// q.proxy的实现是ProxyStore结构体
// q.proxy.Series()是grpc方法(流式)
// q.proxy.Series()调用完毕后，resp的seriesSet属性的值会被填充
if err := q.proxy.Series(&storepb.SeriesRequest{
MinTime: hints.Start,
MaxTime: hints.End,
Matchers: sms,
/*
其他代码
*/
}, resp); err != nil {
return nil, errors.Wrap(err, "proxy Series()")
}
/*
其他代码
*/
set := &promSeriesSet{
mint: q.mint,
maxt: q.maxt,
set: newStoreSeriesSet(resp.seriesSet), // 把resp的seriesSet属性抽出来
aggrs: aggrs,
warns: warns,
}
// set就是指标
return newDedupSeriesSet(set, q.replicaLabels, len(aggrs) == 1 && aggrs[0] == storepb.Aggr_COUNTER), nil
}
2.6 ProxyStore object
// ProxyStore implements the store API that proxies request to all given underlying stores.
type ProxyStore struct {
logger log.Logger
// 返回位于下游的实现Store API接口的组件，查询指标时会用到此属性
stores func() []Client
component component.StoreAPI
selectorLabels labels.Labels
responseTimeout time.Duration
metrics *proxyStoreMetrics
}
When querying metrics, the metrics will be queried from all downstream Store API components, combined, and deduplicated (if set)
/*
根据客户端的请求，从下游的所有的Store API的组件中查询指标以及进行合并、去重，最后将指标传输给入参srv.
这是一个gprc流式接口。
*/
func (s *ProxyStore) Series(r *storepb.SeriesRequest, srv storepb.Store_SeriesServer) error {
/*
其他代码
*/
g, gctx := errgroup.WithContext(srv.Context())
respSender, respCh := newCancelableRespChannel(gctx, 10)
// 生产者协程
g.Go(func() error {
/*
本协程会从后端的thanos store组件中获取指标，并进行指标合并操作。
本协程的关闭，消费者协程也会关闭。
*/
var (
seriesSet []storepb.SeriesSet
storeDebugMsgs []string
wg = &sync.WaitGroup{}
)
defer func() {
wg.Wait()
//close()方法会引发消费者协程退出
close(respCh)
}()
// 遍历后端的Store API组件
for _, st := range s.stores() {
/*
其他代码
*/
sc, err := st.Series(seriesCtx, r)
seriesSet = append(seriesSet, startStreamSeriesSet(seriesCtx, s.logger, closeSeries,
wg, sc, respSender, st.String(), !r.PartialResponseDisabled, s.responseTimeout, s.metrics.emptyStreamResponses))
/*
其他代码
*/
// 获得合并后的指标，再发送给respCh管道
mergedSet := storepb.MergeSeriesSets(seriesSet...)
for mergedSet.Next() {
lset, chk := mergedSet.At()
// respSender.send(...)其实是将指标发送给respCh管道
respSender.send(storepb.NewSeriesResponse(&storepb.Series{Labels: labelpb.ZLabelsFromPromLabels(lset), Chunks: chk}))
}
return mergedSet.Err()
})
// 消费者协程
g.Go(func() error {
// 响应(已被merged)被本协程获取，并将响应输送给方法入参srv.
for resp := range respCh {
if err := srv.Send(resp); err != nil {
return status.Error(codes.Unknown, errors.Wrap(err, "send series response").Error())
}
}
return nil
})
// 等待生产者协程和消费者协程结束
if err := g.Wait(); err != nil {
return err
}
return nil
}
3 Summary:
This article analyzes the outline of the code, and there are many details that have not been mentioned, but the code structure of the Thanos Query component is clear and easy to understand. The github.com/oklog/run package is used to start a set of coroutines, and write http server and grpc server The idea and the dynamic discovery of downstream Store API components are all worthy of imitation.