Now that we have added two Panels to a Dashboard, we can obviously see that all node information will be displayed directly in the same panel, but if there are a lot of nodes, the amount of data will be very large, in this case The best way for us is to use the node as a parameter (add this parameter to Promql to filter the time series, such as filtering out only certain nodes), so that the user can choose which node to view for monitoring. To achieve this function, we need to add a variable with a node as a parameter to query the monitoring data.
Step 1: Open the dashboard settings
The panel was added to the dashboard before. But there is a very obvious flaw here. The information of all nodes is displayed on the panel. If the number of nodes is very large, there will be very, very many curves in it, which will look very inconvenient.
In this case, we usually do a drop-down box, select a node, and only look at the relevant information of a certain point.
In this way, you need to use a function in granfa: parameters, you can add some node parameters to do a filter.
There is a dashboard setting in the upper right corner. Click the button at the top right of the Dashboard page Dashboard settings to enter the configuration page:
On this Settings page, you can configure the entire Dashboard, such as names, labels, variables, etc.:
This brings you to the dashboard configuration page, where you can create a variable.
Here we click on the left to Variables add a variable, variables support more interactive and dynamic dashboards, we can use variables in their place, instead of hardcoding in the metric query, variables are displayed as a drop-down list at the top of the Dashboard, these Drop-down lists make it easy to change the data displayed in the dashboard.
Step 2 Configure dashboard variable settings
In order to be able to select node data, here we define a instance variable name called , and the page where you add variables mainly includes the following attributes:
Name: variable name,$变量名the way to call and use in the dashboard, this should be placed in the PromQL statementType: variable type, there are many types of variables, whichquerymeans that this variable is a query statementHide: If it is empty, it will appear as a drop-down box. Selecting label means that the name of the drop-down box will not be displayed. Selecting variable means to hide the variable. The variable will not be displayed above the Dashboard. The default selection is empty.Data source: The data source of the query statementRefresh: When to update the value of the variable, the value of the variable is obtained by querying the data source, but the data source itself will change, so it is necessary to update the value of the variable from time to time, so that the change of the data source will correspond to the variable. displayed in the drop-down box. Refresh has two values to choose from:On Dashboard Load(update when the Dashboard loads),On Time Range Change(update when the time range changes)Query: query expression, different data source query expressions are differentRegex: Regular expression, used to filter the captured data, not filtered by defaultSort: Sort, sort the variable values in the drop-down box, the default is disable, which means how the query result is displayed in the drop-down boxMulti-value: When this function is enabled, multiple values of the variable can be selected. The specific performance is that the combination of multiple values can be selected in the drop-down box corresponding to the variable.Include All option: Enable this function, there is an option to select all in the variable drop-down box- The corresponding value of lable is:
- The Multi-value drop-down box can be multi-selected, then the value of the variable can be multiple, you need to use =~ regular matching in PromQL
- Include All option drop-down box with all selection or selection
The value of the node comes from the instance tag
The relevant indicators of the monitoring node are derived from node-exporter the task named, and we can up get all the monitoring instances by querying:
You can see that all the values are displayed, the green part is the preview value, and the preview value needs to be processed.
Method 1: Regular Expression
But we only need the value of instance, which is matched by regular expressions.
To get instance the value in the tag, we can use a regular expression .*instance="(.*?)".* here to get the instance data, so that a variable is successfully defined. In addition to using the regular expression to get the required value, we can also use label_values() a function to directly get the value of a label label in the query result:
Then you can do a sorting according to the result value, mainly because the above regular expression should be able to write.
Method 2: lable_values function
In addition to the above method, there is a simple method. We only need the value of the instance tag. We can directly use the lable_values function, which will get the value of a tag in the time series.
label_values(up{job="kubernetes-node-exporter"},instance)
Go back to the Dashboard page and you can see one more 选择节点drop-down box:
Step 3 Configure the variables in the dashboard in the query statement PromQL
But at this time, the panel will not change with the selection of our drop-down box. We need to pass instance this variable into the query statement, such as the re-modified CPU使用率query statement:
Finally, modify the query statement. The query statement is written to death, and there is no filtering according to the parameters of the drop-down box.
(1 - sum(rate(node_cpu_seconds_total{mode="idle",instance=~"$instance"}[1m])) by (instance) / sum(rate(node_cpu_seconds_total{instance=~"$instance"}[1m])) by (instance) ) * 100
When you can see which node is selected, this variable will be replaced with node1 node2.
Returning to the Dashboard page, you can select the node data to be monitored according to our drop-down box. When defining the parameters, if you select all, you can also view the data of all nodes:
