Message Queuing using App.Metrics monitoring message queue monitoring App.Metrics

Monitoring the message queue using App.Metrics

I. Introduction

App Metrics is an open source and cross-platform .NET library for index recording applications. App Metrics can also support the run or on the full .NET Framework .NET 4.5.2 of the .NET Core.

By sampling the App Metrics and polymerized in memory, and provides scalability point at specified intervals to refresh the index repository, which abstracts repository Metrics basis, e.g. InfluxDB, Prometheus, Graphite, Elasticsearch like.

App Metrics provides a variety of metrics to measure the types of things, such as request rate, calculated over a period of several user login time, measure the time it takes to execute database queries, measure the amount of available memory, and so on. Index types supported are Apdex, meters, counters, meters, timers, and histograms.

App Metrics also offers a health check system that allows you to monitor the health of applications through a user-defined checks.

Use App Metrics, you can:

• Capture application index any type of .NET application, such as Windows Service, MVC Site, Web API, etc.
• Automatic measurement MVC or Web API project for each endpoint performance and error
• When using OAuth2 protection API, automatically measure each client's requests and error rates
• Select Save position of the captured metrics and the desired instrument panel used to visualize these indicators
• Extended support format indicators based on the disclosure and push pull metrics collection, and by a via HTTP to request specific report TSDB
• Support in the required format index pushed to a custom HTTP endpoint
• Support in the required format index file written proxy indicators for the collection,

 

More use to directly access the official website: https://www.app-metrics.io/

 

Second, the actual business scenarios

The above describes the scene as well as App.Metrics it supports, but reading you will feel very abstract, yes me too. If not with actual business scenarios to see these things, but it is still a little foggy. In the actual business we usually put it in for two aspects, one is including CPU, memory, including the monitoring of the overall system level, and in the garden a lot of articles have done a demo for your reference, you can search about . On the other hand the relevant statistical data by burying point the way, the back-end database as commonly used InfluxDB and use Grafana or Prometheus to display the data.

This article deals use is described in a second clock, the message queue is performed by means of statistical Buried statistical data includes a queue number, the number (consumption has not consumption) of each current message queue, and generating the message and publication rate.

The final version of the general will look like:

 

Three, InfluxDB 

Before using App.Metrics, we need to be ready database, which is InfluxDB, first a quick look at what is InfluxDB:

InfluxDB is an open source distributed timing written in the Go language, events and metrics database, no external dependencies.

Similar databases have Elasticsearch, Graphite and so on.

Its main features

1) based on the time sequence of correlation functions relating to support of time (e.g., maximum, minimum, sum, etc.)

2) measurability: You can compute large amounts of data in real time

3) based on the event: it supports arbitrary event data

The main features of InfluxDB

1) No structure (modeless): may be any number of columns

2) can expand

3) support min, max, sum, count, mean, median and a series of functions, convenient statistics

4) native HTTP support, built-in HTTP API

5) a powerful class SQL syntax

6) own management interface, easy to use

 

Influx contains the following properties:

Database : database name, in InfluxDB can create multiple databases, data files in different databases are stored separately and stored in different directories on the disk.

retention policy: 存储策略，用于设置数据保留的时间，每个数据库刚开始会自动创建一个默认的存储策略 autogen，数据保留时间为永久，之后用户可以自己设置，例如保留最近2小时的数据。插入和查询数据时如果不指定存储策略，则使用默认存储策略，且默认存储策略可以修改。InfluxDB 会定期清除过期的数据。

measurement: 测量指标名，例如 cpu_usage 表示 cpu 的使用率。

tag sets: tags 在 InfluxDB 中会按照字典序排序，不管是 tagk 还是 tagv，只要不一致就分别属于两个 key，例如 host=server01,region=us-west 和 host=server02,region=us-west 就是两个不同的 tag set。

field name: 例如上面数据中的 value 就是 fieldName，InfluxDB 中支持一条数据中插入多个 fieldName，这其实是一个语法上的优化，在实际的底层存储中，是当作多条数据来存储。

timestamp: 每一条数据都需要指定一个时间戳，在 TSM 存储引擎中会特殊对待，以为了优化后续的查询操作。

2）Point

InfluxDB 中单条插入语句的数据结构，series + timestamp 可以用于区别一个 point，也就是说一个 point 可以有多个 field name 和 field value。

3）Series

series 相当于是 InfluxDB 中一些数据的集合，在同一个 database 中，retention policy、measurement、tag sets 完全相同的数据同属于一个 series，同一个 series 的数据在物理上会按照时间顺序排列存储在一起。

series 的 key 为 measurement + 所有 tags 的序列化字符串，这个 key 在之后会经常用到。

 

四、搭建InfuxDB+Grafana 

OK，这篇是一个DEMO演示篇，所以我们使用Dokcer快速的创建InfluxDB和Grafana:

docker run -d -p 8083:8083 -p 8086:8086 --expose 8090 --expose 8099 tutum/influxdb

docker run -d --name=grafana -p 3000:3000 grafana/grafana

 

运行成功之后我们分别可以访问8083端口的InfluxDB和3000端口的Grafana：

 

 

 

 

 

 

我们首先给InfluxDB添加一个用户：

 

 

 添加完成后配置一下Grafana：

 

 

 

四、.NET CORE使用App.Metrics

这里我们使用.net core控制台项目来演示（API项目例子已经有很多了，但是控制台项目没看到），新建一个控制台项目 AppMetricsPractice：

通过NuGet引用App.Metrics和App.Metrics.Reporting.InfluxDB

 

 

 

 

 

 然后我们就可以愉快的使用了，简单使用可以如下：

static void Main(string[] args)
        {
            var metrics = new MetricsBuilder().Report
                .ToInfluxDb("http://47.99.92.76:8086", "metricsdatabase")
                .Build();


            var counter = new CounterOptions { Name = "my_counter", MeasurementUnit = Unit.Calls };
            metrics.Measure.Counter.Increment(counter,"MqQueueNmae");

            var task = Task.Run(async () =>
            {
                await Task.WhenAll(metrics.ReportRunner.RunAllAsync());
            });

            task.Wait();


            Console.Read();
        }

使用方式在官网有简介，这里介绍一下，ToInfluxDb（influxDB url,InfluxDB databaseName）,这里是InfluxDB的地址和数据库名称，如果没有改数据库，会自动创建。

以上写法是简写，当然我们可以详细的控制InfluxDB的属性，通过以下写法：

            var metrics = new MetricsBuilder()
                .Report.ToInfluxDb(
                    options =>
                    {
                        options.InfluxDb.BaseUri = new Uri("http://47.99.92.76:8086");
                        options.InfluxDb.Database = "metricsdatabase";
                        options.InfluxDb.Consistenency = "consistency";
                        options.InfluxDb.UserName = "admin";
                        options.InfluxDb.Password = "password";
                        options.InfluxDb.RetentionPolicy = "rp";
                        options.InfluxDb.CreateDataBaseIfNotExists = true;
                        options.HttpPolicy.BackoffPeriod = TimeSpan.FromSeconds(30);
                        options.HttpPolicy.FailuresBeforeBackoff = 5;
                        options.HttpPolicy.Timeout = TimeSpan.FromSeconds(10);
                        options.MetricsOutputFormatter = new MetricsInfluxDbLineProtocolOutputFormatter();
                        options.Filter = filter;
                        options.FlushInterval = TimeSpan.FromSeconds(20);
                    })
                .Build();

 

Option	Description
MetricsOutputFormatter	向InfluxDB报告指标时使用的格式化程序。
Filter	该过滤器仅用于为此报告者过滤指标。
FlushInterval	向InfluxDB报告指标之间的延迟。
InfluxDb.Database	报告指标的InfluxDB数据库。
InfluxDb.BaseUri	InfluxDB服务器的URI。
InfluxDb.UserName	使用基本身份验证与InfluxDB进行身份验证时的用户名。
InfluxDb.Password	使用基本身份验证与InfluxDB进行身份验证时使用的密码。
InfluxDb.Consistenency	要使用的InfluxDB数据库一致性级别。
InfluxDb.RetentionPolicy	InfluxDB数据库保留策略以向其写入指标。
InfluxDb.CreateDataBaseIfNotExists	如果指定的influxdb数据库不存在，将尝试创建该数据库。
HttpPolicy.BackoffPeriod	TimeSpan当指标无法向指标入口端点报告时，从后退。
HttpPolicy.FailuresBeforeBackoff	指标未能向指标入口端点报告时，在回退之前的失败次数。
HttpPolicy.Timeout	尝试向度量标准入口端点报告度量标准时的HTTP超时持续时间。

 

然后我们要存储一个Counter类型的数据，App.Metrics里有主要有6种数据类型：Counter、Gauge、Histograms 、Meter 、Timer 、Apdex。我们本篇主要使用Counter和Gauge这两种数据类型，

CounterOptions种的Name是数据表名，MeasurementUnit是测量的内容的描述。

metrics.Measure.Counter.Increment(counter,"MqQueueNmae");  会往把“my_counter”表里的value + 1，实际就是对value加加减减，大致如下：

 

 同时还会创建一张名为my_counter__items的表，同时为一个字段为“MqQueueNmae”的value+1，如下：

 

 

多了几个字段，通过这个我们可以用来对不同的消息度列Queue进行统计，而第一张表则当做一张当前消费消息的统计表。

            var task = Task.Run(async () =>
            {
                await Task.WhenAll(metrics.ReportRunner.RunAllAsync());
            });

            task.Wait();

这句代码是指将当前的统计数据报告到InfluDB，这段代码一定要在最后，它会将数据发送到所有已注册的存储端，比如你同时注册了InfluxDB和Prometheus，那么数据会同时发送到这两个存储端。

执行完成后创建的两张表如下：

 

 

然后我们就可以去Granfan里自定义统计图了，比较简单，大家可以自己研究一下，大致如下：

 

 

下一篇将会把这一套集成到实际项目中，用来监控消息队列系统，这一篇只是了解，等我明天写可以直接用于生产的！