Introduction and practice of Thanos

 Luyuan  360 Cloud Computing 

Heroine declaration

With the growing scale of Openstack clusters, the monitoring data has shown exponential growth , which has brought a great test to the subsequent expansion of computing and storage resources. How to store monitoring data stably and permanently, and quickly query hot data and historical data has always been a problem in large-scale cloud computing clusters. Of course, the Ceilometer, Gnocchi, and Aodh projects of the Openstack community have not been able to solve our current problems well. Here The author will introduce the CNCF big killer, the concept and use of the Thanos + Prometheus TP combination (PS: not a silver bullet) in the Openstack and ceph clusters, and will give an effective answer to the above questions.

PS: rich first-line technology, a wide range of forms, all in " 3 60 cloud computing " point of concern Oh!

1

What is Thanos

British game technology company Improbable has open sourced their Prometheus high-availability solution. A simple and easy-to-understand English introduction on the homepage is as follows: Open source, highly available Prometheus setup with long term storage capabilities. Open source, highly available Prometheus settings, and provide long-term storage capabilities.

2

What are the characteristics of Thanos

1. Cross Prometheus services and provide a unified query interface.
2. Indefinite storage monitoring indicators. Currently, it supports object storage systems such as S3, Microsoft Azure, Tencent COS, Google GCP, and Openstack Swift.
   
3. Compatible with existing Prometheus API interfaces, such as Grafana or support tools such as Pormetheus Query API.
4. Provide data compression function and reduced standard sampling to improve query speed.
5. Data deduplication and merging, and collecting metrics from the Pormetheus HA cluster.

3

Thanos architecture

4

Components in the Thanos architecture

Compact

Compac provides data reduction and compression functions. It is mainly responsible for compressing objects in S3 storage. It can merge and compress blocks in historical data into large file objects. In fact, the reduced standard compression does not save any space, and it will add 2 blocks to the original Block, but it will increase the query speed when querying historical data. The last thing to note is that since the intermediate data is processed while the process is running, sufficient disk space is needed locally. As the data increases, the space demand is getting larger and larger. At present, we reserve 300GB of local space for the processing of compressed intermediate data, and Compression is performed every three days.

Querier

The query component implements the Pormetheus HTTP v1 API function. The component is responsible for querying and collecting data after receiving the HTTP PromSQL query request. It is a stateless service that supports horizontal expansion.

SideCar

This component needs to be deployed together with the Pormetheus instance. It mainly plays two roles. The first agent Querier component reads local Prometheus data; the second is to upload Prometheus local monitoring data to the object storage through the object storage interface. Finally, the sidecar will monitor the local storage of Prometheus. If new monitoring data is found to be saved to the disk, it will upload the monitoring data to the object storage.

Store

Store mainly provides the function of querying historical data. When the Querier component calls the Stroe interface, Stroe then obtains the data through the object storage interface, and converts the stored data into the data format required by Querier.

Bucket

The command used to check the data in the object storage is usually run as an independent command and helps us with troubleshooting. It supports viewing the current number of buckets through the Web UI.

Check

Through Thanos check, you can check and verify whether Pormetheus Rules are correct. The implementation function is as follows.

//定义检查Rules函数func checkRules(logger log.Logger, filename string) (int, errors.MultiError) {//记录日志，返回检测的文件名称和详细的日志信息  level.Info(logger).Log("msg", "checking", "filename", filename)  checkErrors := errors.MultiError{}
  b, err :=//读取Rules文件  ioutil.ReadFile(filename)  if err != nil {    checkErrors.Add(err)    return 0, checkErrors  }//由于rules 格式需要纯Yaml格式，需要验证Yaml 格式是否正确  var rgs ThanosRuleGroups  if err := yaml.UnmarshalStrict(b, &rgs); err != nil {    checkErrors.Add(err)    return 0, checkErrors  }
// We need to convert Thanos rules to Prometheus rules so we can use their validation.  promRgs := thanosRuleGroupsToPromRuleGroups(rgs)  if errs := promRgs.Validate(); errs != nil {    for _, e := range errs {      checkErrors.Add(e)    }    return 0, checkErrors  }
  numRules := 0  for _, rg := range rgs.Groups {    numRules += len(rg.Rules)  }//函数结尾返回检查的rules 数量和错误的数量及错误信息  return numRules, checkErrors}

5

Problems we encountered in Thanos practice

Since Thanos Store loads accessible data when it starts, it loads a small amount of object storage block information on the local disk or memory, which will cause the local disk and memory to become full over time, causing cluster exceptions, and introducing the following Multiple questions. A large number of slow queries caused a surge in memory and Store OOM appeared. In the early stage, we used the POD method to deploy the Thanos cluster. As the IP changed after the POD was changed, the cluster split brain and collapsed, and finally failed to query historical data. Considering that the Stroe component consumes more resources, we transfer it to a physical machine, and put Sidecar and Pormetheus into the POD. Due to the poor performance of the earlier version, we have also upgraded the version and enabled the compression function. After enabling compression:

Monitoring data volume generated from September 28 to November 07:

The current integrated monitoring scenarios are as follows:

1. Ceph / Cephfs, Lvs, Openstack, Etcd, K8s, Istio, Openstack virtual machine monitoring.
2. Provide API query interface and StackStorm linkage to process specified event information.

6

to sum up

The Thanos solution itself does not have any strong intrusion into Prometheus and strengthens the shortcomings of Prometheus. Finally, Thanos relies on the object storage system, and this part of the resources should be considered as much as possible. Currently, there are about 40+ sets of Openstack, 70+ sets of Ceph clusters, about 10,000+ OSD nodes, and about 50G of monitoring data is generated every day.

What problems did Thanos help solve?

1. Due to the limitation of storage size, the issue of the time of historical data storage is First Blood.
2. The number of clusters is increasing, and the query performance of Prometheus is stuck with double kill.
3. There are a large number of Openstack and ceph clusters, and it is impossible to query data and alarm triple kill through a unified interface.