Dameng Database DM8 Tutorial: Operation and Maintenance Monitoring Tools (DEM, DMLOG Log Tool, NMON, Prometheus)

Operation and maintenance monitoring tools

I. Introduction

The database operation and maintenance monitoring system can provide important guarantee for the stable operation of the database. If a reliable operation and maintenance monitoring system can be established to give early warning of possible disasters, the emergency response mechanism can be quickly activated to eliminate system failures, which is of great practical value for ensuring the stable operation of the system.
This article mainly introduces four commonly used operation and maintenance monitoring tools for Dameng database:

• Dameng Enterprise Manager DEM
• Dameng SQL log analysis tool DMLOG
• Open source performance monitoring tool NMON (Linux)
• Open source operation and maintenance monitoring tool Prometheus

2. Dameng Enterprise Manager DEM

2.1 Tool introduction

DM Enterprise Manager (DEM for short) provides a centralized management platform to monitor, manage and maintain the DM database through a Web interface. Database administrators can log in to DEM through any web application to manage and monitor the DM database. DEM mainly has functions such as cluster deployment, automatic inspection, monitoring and alarming. DEM consists of the following parts:

• DEM server: refers to the DEM application server, which is responsible for processing the function logic of the client tool and storing the data collected by dmagent to the DEM storage database, and displaying the monitoring data to the client.
• DEM storage database: stores DEM metadata and monitoring data collected by dmagent.
• Database instance: The database instance that needs to be managed and monitored.
• Database proxy service (dmagent): The proxy deployed on the remote machine, DEM accesses the remote host through dmagent, and dmagent collects monitoring information and sends it to DEM.

The DEM system architecture diagram is as follows:

DEM tools mainly provide the following functions:

• Client tools. Users can use the DEM tool to perform object management, status monitoring, SQL query and debugging of the DM database.
• Monitoring and alerting. This function is the core function of DEM tools. By deploying the agent on the remote host, you can monitor the status of the remote host and the status of the DM database instance on the remote host. The DEM monitoring function is not limited to a single database instance, but can also monitor and manage database clusters (MPP, RAC, data guardian, etc.).
• System Management. The DEM tool provides system configuration and permission management of the tool itself, making it convenient for different users to use the tool at the same time, and can also limit the permissions of non-admin users.

2.2 Scope of application

This tool is suitable for monitoring DM7 database and DM8 database.

2.3 Tool download

The DEM trial version can be downloaded from the Dameng database official website or Dameng online service platform .

2.4 DEM deployment

2.4.1 Environment preparation

1. Hardware environment

(1) Server: physical machine or virtual machine;

(2) Operating system: Windows or Linux operating system is acceptable;

(3) Memory: It is recommended to provide more than 2G memory;

(4) Others: Since DEM will obtain information from the monitored database to the backend database, it is recommended that the storage directory of DEM data in the production environment should be no less than 100GB; the remaining hardware requirements are consistent with the stand-alone deployment of the database.

2. Software environment

(1) DEM WAR package: located in the Dameng installation directory “…/web/dem.war”;

(2) Tomcat: self-prepared, required to match the deployed system environment;

(3) JAVA 1.8: The machine where DEM and dmagent are located needs to be configured with a JAVA environment, and the JAVA version must be JAVA 1.8. If there is no jdk environment in the system, you can specify the jdk directory in the database in the environment variable, as shown below:

[root@localhost ~]# cat /etc/profile
export JAVA_HOME=/home/dmdba/dm/dmdbms/jdk
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

(4) Dameng database: It is the backend database of DEM. It is necessary to ensure that the machine where DEM is located can access the Dameng database.

Before deploying dmagent, you need to set the time synchronization between DEM and the server host where dmagent is located. For formal environments, it is recommended to use ntp to configure time synchronization. The following takes "10.10.10.10" as the ntp server as an example to set up scheduled tasks:

[root@localhost ~]# crontab -l
# Time sync
0,10,20,30,40,50 * * * * /root/sync_time.sh
[root@localhost ~]# cat /root/sync_time.sh 
#!/bin/bash
. /etc/profile
. /root/.bash_profile
ntpdate 10.10.10.10 ; hwclock –w

2.4.2 Initialize the backend database

1. Create a backend database. Create a DM database as the DEM background database. The initialization parameters are not required, and the default is sufficient, and the dm.ini parameter configuration of the database is optimized. The recommended configuration is as follows:

MEMORY_POOL         	=  200
BUFFER                  		=  1000
KEEP                    		=  64
SORT_BUF_SIZE          	=  50

1. Execute the initialization script. Use SYSDBA (or administrator user) to execute the SQL script dem_init.sql in the background database created. The script is located in the Dameng installation directory ".../web/dem_init.sql" (this SQL script is encoded as UTF-8, if you use disql To execute SQL script, please set it up first set CHAR_CODE UTF8). After executing the script, a DEM schema will be generated in the background database to store the tables and views required for DEM operation.
2. Backup strategy. You can determine whether it is necessary to set up a backup of the backend database based on actual project needs, and determine the backup strategy.

2.4.3 Configuring Tomcat

1. Unzip the tomcat package. After installing tomcat, the following directory will be generated:

[dmdba@localhost apache-tomcat-7.0.75]# ll
总用量 144
drwxr-x--- 2 dmdba dinstall  4096  8月 27 09:28 bin
drwx------ 3 dmdba dinstall  4096  8月 31 09:32 conf
drwxr-x--- 2 dmdba dinstall  4096  8月 27 09:15 lib
-rw-r----- 1 dmdba dinstall 57092  9月  5  2018 LICENSE
drwxr-x--- 2 dmdba dinstall  4096  9月  3 09:46 logs
-rw-r----- 1 dmdba dinstall  1726  9月  5  2018 NOTICE
-rw-r----- 1 dmdba dinstall  7142  9月  5  2018 RELEASE-NOTES
-rw-r----- 1 dmdba dinstall 16262  9月  5  2018 RUNNING.txt
drwxr-x--- 2 dmdba dinstall  4096  8月 27 09:15 temp
drwxr-x--- 8 dmdba dinstall  4096  8月 27 09:32 webapps
drwxr-x--- 3 dmdba dinstall  4096  8月 27 09:32 work

1. Check the JAVA environment. Before starting, you need to make sure to configure the running environment of JAVA 1.8 and above, and run java –versionto check the JAVA version.

[dmdba@localhost ~/apache-tomcat-7.0.75]$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)

1. Modify tomcat configuration.

<Connector port="8080" protocol="HTTP/1.1"...(1) Add the attribute field at the position in the conf/server.xml configuration file maxPostSize="-1".

Copy<Connector port="8080" protocol="HTTP/1.1" maxPostSize="-1"
               connectionTimeout="20000"
              redirectPort="8443" />

(2) In the bin/catalina.sh configuration file, modify the jvm startup parameters according to the installation path of the database.

CopyLinux：bin/catalina.sh -> 	JAVA_OPTS="-server -Xms256m -Xmx1024m -Djava.library.path=/home/dmdba/dmdbms/bin"
Windows：bin/catalina.bat -> set java_opts= -server -Xms40m -Xmx1024m -Djava.library.path=c:\dmdbms\bin

2.4.4 DEM connection configuration

1. Unzip the dem.war package.

Place the dem.war package in Tomcat's webapps directory, start Tomcat, and the war package will be automatically decompressed to generate a DEM directory. Execute the script in the bin directory to start Tomcat.

[dmdba@localhost ~/apache-tomcat-7.0.75/bin]$ ./startup.sh
Using CATALINA_BASE:   /home/dmdba/apache-tomcat-7.0.75
Using CATALINA_HOME:   /home/dmdba/apache-tomcat-7.0.75
Using CATALINA_TMPDIR: /home/dmdba/apache-tomcat-7.0.75/temp
Using JRE_HOME:        /usr
Using CLASSPATH:       /home/dmdba/apache-tomcat-7.0.75/bin/bootstrap.jar:/home/
dmdba/apache-tomcat-7.0.75/bin/tomcat-juli.jar
Tomcat started.

1. Configure the background database connection.

Configure the connection information of the backend database: ip, port, user name, password, connection pool size, SSL login information, etc., configure it in the file dem/WEB-INF/db.xml;

[root@localhost WEB-INF]# cat db.xml
<?xml version="1.0" encoding="UTF-8"?>
<ConnectPool>
        <Server>10.9.13.11</Server>
        <Port>5236</Port>
        <User>SYSDBA</User>
        <Password>SYSDBA</Password>
        <InitPoolSize>5</InitPoolSize>
        <CorePoolSize>10</CorePoolSize>
        <MaxPoolSize>50</MaxPoolSize>
        <KeepAliveTime>60</KeepAliveTime>
        <DbDriver></DbDriver>
        <DbTestStatement>select 1</DbTestStatement>
        <SSLDir>../sslDir/client_ssl/SYSDBA</SSLDir>
        <SSLPassword></SSLPassword>
</ConnectPool>

If SSL security is required to connect to the backend database, SSLDir and SSLPassword must be configured: By default, the key pair is stored in the WEB-INF/sslDir directory. WEB-INF/db.xml configures the key file (SSLDir) used for client connections as WEB-INF/sslDir/client_ssl/SYSDBA, the password (SSLPassword) is empty, corresponding to the login user SYSDBA configured in WEB-INF/db.xml. At this time, you only need to copy WEB-INF/sslDir/server_sslto the background database execution code directory.

Password and SSLPassword in db.xml support setting a custom encryption and decryption engine to encrypt and decrypt passwords. The encryption and decryption engine class path is specified through the attribute engine. Custom encryption and decryption engines need to implement the com.dameng.dem.server.util.IEncryptEngine interface provided in demsdk.jar. Package the custom encryption and decryption engine into a jar file and place it in the WEB-INF\lib directory.

demsdk.jar provides a default encryption and decryption engine: com.dameng.dem.server.util.DefaultEncryptEngine. The default encryption and decryption engine configuration is as follows:

<Password engine="com.dameng.dem.server.util.DefaultEncryptEngine" 通过 DefaultEncryptEngine 加密后的密码</Password>

1. Configure DEM log level.

In tomcat's webapps/dem/WEB-INF directory, the log4j.xml file controls the log level. The LOG_LEVEL parameter controls the display information of the log. The dynamic combination of these three parameters, LOG_MAX_SIZE, LOG_MAX_COUNT, and LOG_PRESERVE_DURATION, controls the log volume and the maximum size of the log storage. Logs exceeding this maximum size will be deleted.

日志最大大小= LOG_MAX_SIZE × LOG_MAX_COUNT × LOG_PRESERVE_DURATION；

When debugging and finding problems in DEM, you can use the DEBUG or ALL parameters for LOG_LEVEL. It is recommended to use the ERROR level in the production environment. According to the actual needs of the project and the available size of the disk, adjust the size of the three parameters LOG_MAX_SIZE, LOG_MAX_COUNT, and LOG_PRESERVE_DURATION to avoid disk bursting. This configuration file will not take effect until Tomcat is restarted.

1. Restart takes effect. Restart Tomcat for the configuration to take effect.

2.4.5 Deploy dmagent

Before deploying dmagent, you need to make sure that the server is configured with a running environment of JAVA 1.8 and above, run to java –versioncheck the JAVA version, and check that the time of the dmagent machine and the dem running machine are consistent.

1. Get dmagent. There are two ways to obtain dmagent:
   (1) There is dmagent under tool in the Dameng database installation directory;
   (2) Log in to the deployed DEM, click the host under the monitoring and alarm module, open the host panel, and select in the toolbar: More -> Download agent, download dmagent.
2. Copy dmagent to the machine to be deployed.
3. Modify dmagent configuration. Configure agent.ini in the dmagent directory:

##dem 所在机器的地址
center_url  =  http://192.168.104.72:8080/dem  

1. Configure dmagent log level. Also modify the log4j.xml file in the dmagent directory. The modification method is the same as that on the dem side.
2. Start dmagent.

Linux：bin/start.sh –d  agent.ini
Windows：start.bat –d  agent.ini

2.4.6 Others

Client tools display. If it is required not to display the manager client tool, add an attribute field to the tomcat/bin/catalina.sh file JAVA_OPTS=”…… -Drun_mode=DMA”. After the configuration is successful, the instance connection will not be displayed.

2.5 Tool usage

2.5.1 System resource monitoring

1. Open the DEM tool

Enter the DEM IP in the browser: port/DEM, the default user and password are "admin/888888", log in to the DEM page, as shown in the figure below:

2. Function introduction

Click [Intelligent Operation and Maintenance] -> [Resource Monitoring] -> Host Options. In addition to displaying the usage of the host's CPU, memory, etc., you can also see the information panel, load statistics, disk analysis, and custom monitoring of the host system under monitoring. Process monitoring and Core monitoring. In addition, there are settings such as network configuration, upgrade agent, subscription, deletion, viewing properties, etc.

3. Load statistics

Click [Load Statistics], and the page will display various load conditions of the current system in detail. Users can find load statistics from 30 minutes to the past week (the time range can also be customized), displayed in a line chart, including memory and virtual memory usage, CPU usage, disk read and write rates, network send and receive s speed.

Users can intuitively monitor system load statistics through the trend of the line chart, as shown in the following figure:

4. Disk analysis

The analysis page displays the usage, space size and total size of each disk path. The disk usage is displayed in the line chart below with the system time point as the horizontal axis.

5. Customized monitoring

You can add custom monitoring to monitor the execution time, execution status, latest execution results and execution information of a program or script to meet specialized monitoring needs.

6. Process monitoring

You can add process monitoring by setting name and process filtering conditions to monitor process running status in real time.

7. Core monitoring

Set the Core monitoring of the corresponding process by setting the process path, Core file location and process name, and timely grasp the downtime of related processes.

2.5.2 Database monitoring

1. Function introduction

Click [Intelligent Operation and Maintenance]–>[Resource Monitoring]–>[Database] option to add, edit, and delete monitored databases. In addition, click the [Operation] option in the monitored database to view basic database information, AWR report view, inspection report view, table space analysis, SQL analysis, login lock, deadlock analysis, session analysis, event analysis, and customization SQL analysis, table data monitoring, running log monitoring, transaction log monitoring, DM.INI modification and viewing and other configurations.

2. AWR Report

AWR reports can be customized for any time period and include database and system overview, resources, performance, session, SQL, deadlock and abnormal login information. The detailed information of each part is also displayed in a line chart.

3. Table space analysis

You can select table space usage and configuration for any time period, including table space usage, free size, total size, maximum size, number of files, automatic expansion, complete path and other information.

4. SQL Analysis

SQL analysis can monitor the occurrence of slow SQL, high-frequency SQL and error SQL in the database within a specified time period.

2.5.3 Alarm configuration

Click [Alarm Configuration] -> [Add] to set additional monitoring alarm options and set an alarm mailbox. When an indicator exceeds the threshold point, the alarm information can be sent to the designated mailbox to facilitate timely acquisition of system risk points. and take emergency measures.

To enable email notification, you need to log in to the system as an administrator and complete the relevant configuration of the system mailbox in the system configuration. If you need to enable SMS notification, you need to use the WEB-INF/lib/demsdk.jar file to implement the com.dameng.dem.server.util.IPhoneNotify interface, and package the dependent packages and implementation classes into WEB-INF/lib. , restart the web container, and complete the relevant configuration of SMS notification in the system configuration.

This module can add alarms and set alarm policies according to specific system requirements.

Email and SMS reminders can be set in the system management system configuration.

2.5.4 Precautions

1. The versions of DEM and dmagent need to be consistent, and the system time at both ends should also be consistent. Otherwise, host information will not be displayed.
2. In a production environment, it is recommended not to use the SYSDBA user connection. It is recommended to create an independent DEM user, grant query permissions to the corresponding tables and views, and limit the maximum number of sessions and session idle periods. Monitoring users should not configure excessive permissions such as any. Take the test user as an example:

--授予DEM监控数据库最小连接权限：
create user "TEST" identified by "******" limit session_per_user 50, connect_idle_time 10;
grant "PUBLIC" to "TEST";
grant SELECT TABLE,SELECT VIEW,SELECT SEQUENCE to "TEST";

1. In a production environment, it is recommended to configure monitoring items based on actual conditions. For database deadlocks, database events, high-frequency SQL, error SQL, slow SQL, table spaces, user locks and other information, please monitor and set up as needed.
2. Historical data clearing. DEM will generate historical data after running for a period of time, which will occupy disk space. The following two methods can be used to clean historical data regularly to free up space.

Method 1: Set scheduled cleaning through the Web page.

Method 2: Set up database scheduled tasks to clean historical data regularly.

The relevant scheduled task SQL statements are as follows:

call SP_CREATE_JOB('DEL_DEM_HIS',1,0,'',0,0,'',0,'');
call SP_JOB_CONFIG_START('DEL_DEM_HIS');
call SP_JOB_SET_EP_SEQNO('DEL_DEM_HIS', 0);
call SP_ADD_JOB_STEP('DEL_DEM_HIS', 'DEL_DEM_HIS', 0, 'truncate table "DEM"."DMA_DATABASE_SQL_COUNT";
truncate table "DEM"."DMA_DATABASE_SQL_COUNT_SQL_TEXT";
truncate table "DEM"."DMA_DATABASE_SESSION";
truncate table "DEM"."DMA_DATABASE_SQL_ERR";
truncate table "DEM"."DMA_DATABASE_SQL_SLOW";
truncate table "DEM"."DMA_DATABASE_USER";
truncate table "DEM"."DMA_MAINFRAME_DISK";', 0, 0, 0, 0, NULL, 0);
call SP_ADD_JOB_SCHEDULE('DEL_DEM_HIS', 'DEL_DEM_HIS', 1, 2, 1, 1, 0, '01:00:00', NULL, '2022-03-04 15:56:01', NULL, '');
call SP_JOB_CONFIG_COMMIT('DEL_DEM_HIS');

3. Dameng SQL log analysis tool DMLOG

3.1 Tool introduction

Instance-level SQL log analysis tool, DMLOG tool is a simple and easy-to-use Dameng database SQL log analysis tool. By analyzing the SQL log file of the database (Daemon database can capture all SQL running in the database by turning on SVR_LOG and generate SQL log file) to count the longest execution time and the most frequently executed SQL statement in the log, intuitively reflecting the SQL execution status, and providing great convenience for SQL optimization work.

The DMLOG tool can achieve the following functions:

• Dameng Database SQL log analysis function can statistically analyze logs and sort them according to execution time and execution frequency, and generate excel documents.
• Supports the generation of echart scatter plots, and the SQL at each point can be displayed and copied.
• Supports the generation of SQL statistical graphs and statistics based on execution times and execution time.

3.2 Scope of application

• The DMLOG tool is not restricted by the operating system platform, and the size does not exceed 10M. After the Java environment is installed, it can run on all platforms.
• The DMLOG tool works with various versions of DM7 databases and DM8 databases.

3.3 Tool download

The DMLOG tool can be downloaded from the VIP area of ​​the online service platform or obtained by contacting Dameng technical service personnel.

3.4 Tool usage

3.4.1 Conditions and restrictions of use

1. The operating environment requires the Java environment to be installed in advance. DMLOG supports running on Linux and Windows systems. It is recommended to use the java1.8 version. Linux minimum installation requires at least the printing service component to be installed. The java1.6 version is not supported under Windows.
2. The format of the SQL log requires that each statement be followed by the time of the sql statement. Therefore, it is necessary to confirm the sql trace parameter of the database where the log is generated. It is recommended to use the default parameter.
3. Since the program needs to create the log_commit table in the background database, the table with the same name will be deleted before the table is created. Therefore, if there is a table with the same name, please make a backup first.
4. Please ensure that there are only SQL logs in the SQL folder to be analyzed and no other files.
5. The log path in the dmlog.properties configuration file. Note that under windows, use "\" instead of "\".
6. Since the maximum number of rows on a page of Excel is 65536, when the amount of logs to be analyzed is large, a warning of exceeding the limit will be prompted, and only the first 65535 records will be taken. At this point, the logs can be split or analyzed in batches, or the SQL with relatively long execution time and high execution time can be analyzed, or the parameter replacement function can not be used to reduce the output results.

warn

This tool can only be used for analysis in the test environment. Do not connect to the production environment! ! !

3.4.2 Preparation before use

Please check Java installation before use.

[root@localhost DMLOG8.3]# su - dmdba
Last login: Tue Oct 12 11:20:08 CST 2021 on pts/1
[dmdba@localhost ~]$ java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

3.4.3 Usage methods and steps

1. Unzip the tool package and configure tool parameters.

[root@localhost /]# unzip DMLOG8.*.zip
[root@localhost /]# cd DMLOG8.*/ 
[root@localhost DMLOG8.10]# vi dmlog.properties

Notice

In the Windows environment, the log path sqlpath parameter in the dmlog.properties configuration file needs to use "\" instead of "\", and fill in the normal path under liunx.

1. Execute the jar package command.

[root@localhost DMLOG8.10]# java -jar Dmlog_DM_8.10.jar

The running process is shown in the figure below. After the execution is completed, the corresponding RESULT_$DATE folder will be generated according to the current time, and all statistical information will be stored in this directory.

3.4.4 Interpretation of results

Under the generated RESULT_$DATE result folder, there are excel files (xls) named according to the configured execution time and upper limit of execution times, SQL files with errors (txt), SQL files with a length exceeding 30,000 (txt), echart scattered points Figure and 90% average times and average time-consuming SQL statistics chart (html).

1. The more_than_0_ms_log_result.xls worksheet is sorted in descending order by maximum execution time, as shown below.

1. The more_than_0_times_log_result.xls worksheet is sorted in descending order by the number of executions, as shown below.

1. echarts_scatter_loading10.html The SQL information of each point of the echart scatter plot can be displayed and copied, as shown below.

Scatter plots support the following features:

(1) Filter SEL/DDL/INS/CAL and other categories, and just click to remove or select the corresponding category to display the corresponding SQL distribution points on the graph;

(2) Supports the scatter plot area magnification function. You can specify the position with the mouse and use the scroll wheel to zoom in to enlarge the statement.

(3) Click each point to get the detailed SQL information of each point.

(4) Click on the upper right corner of the scatter plot to save the image.

1. echarts_scatter_Statistics.html is a SQL statistical chart generated based on the scatter chart based on the 90% average number of executions and the average time taken.

1. echarts_qps.html is a QPS (Queries-per-second, query rate per second, in the database refers to the number of times SQL queries are executed per second) line chart.

4. Open source performance monitoring tool NMON (Linux)

4.1 Tool introduction

nmon is a monitoring and analysis tool widely used on AIX and various Linux operating systems. Compared with other system resource monitoring tools, nmon records more comprehensive information. It can capture system resource usage in real time during system operation, output result files, and generate data files and graphical results through the nmon_analyzer tool. General nmon monitoring system resources include cpu usage, memory usage, disk I/O speed, transmission and read-write ratio, file system usage, network I/O speed, transmission and read-write ratio, error statistics and transmission packets Information such as size, processes consuming the most resources, machine details and resources.

4.2 Scope of application

This tool is suitable for monitoring Linux system resource usage.

4.3 Tool download

NMON and nmon_analyser tools can be downloaded from the NMON official website .

4.4 NMON deployment

1. Upload the nmon file to the /home/dmdba/dm/dmdbms/tool ​​directory, and then execute it ./nmon. If the following interface appears, it means that the current environment can run normally.

1. Set up scheduled schedules to monitor system resources in real time.

##使用 root 用户执行
crontab -e

##添加以下脚本内容，实时监控系统资源并生成 NMON 日志文件。-s20 表示 20s 采集一次，-c4320 表示一天 24 小时采集 4320 次
0 0 * * * /home/dmdba/dm/dmdbms/tool/nmon -s20 -c4320 -fT -m /home/dmdba/dm/nmon_log > /dev/null 2>&1

##添加以下脚本内容，设定定时删除计划，保留某时间段内的日志，防止空间占满。+365 为保留时间，日志具保存时长，需根据系统实际情况而定
0 0 * * * find /home/dmdba/dm/nmon_log  -type f -mtime +365 -exec rm -f {
    
    } \;

4.5 Tool usage

4.5.1 NMON log analysis

After the NMON tool is deployed, you can monitor the system status and generate monitoring log files. The NMON_ANALYZER tool analyzes and loads excel macro commands to generate excel charts to display various information on resource occupation. During use, you can visually view system resource usage through chart information.

Usage: Open nmon analyzer v66.xlsm, use analyze nmon data to open the nmon log file that needs to be analyzed. After the analysis is completed, save it as an excel file, and you can query the chart information of the system resources.

4.5.2 Interpretation of results

The following provides an Excel report generated by using the NMON_ANALYZER tool to analyze NMON after monitoring NMON in a system for a period of time, and details the meaning of multiple sub-tables in the report for reference.

1. Summary of system resource usage

The SYS_SUMM subtable shows the overall usage of system resources. The main information on the page is as follows:

The left vertical axis is the system CPU (user%+sys%) usage, the horizontal axis is the running time, the right vertical axis is the system disk transfer (Disk xfers), the blue line represents the system CPU usage, and the pink line represents the system I/O situation. . The lower left side of the coordinates is statistical information, involving system I/O conditions (average value, maximum value, and time of maximum value within a collection interval), system CPU usage, etc.

2. System information

The AAA subtable displays basic system information, mainly including: executed commands, host CPU number, operating system kernel version information, host name, etc.

3. Disk information summary

The DISK_SUMM subtable summarizes system disk information. Mainly includes the following contents:

(1) Disk total KB/s represents the execution interval list;

(2) Disk Read KB/s represents the read rate of the disk device during the collection interval;

(3) Disk Write KB/s represents the write rate of the disk device during the collection interval;

(4) IO/sec represents the overall average number of IOs per second on the disk during the collection interval.

4. Overall CPU condition

The CPU_ALL subtable provides statistics on the overall status of the CPU. Mainly includes the following contents:

(1) CPU Total: execution interval list;

(2) User%: Time proportion (Avg, Max) of all CPUs in User Mode during the collection interval;

(3) Sys%: Time proportion (Avg, Max) of all CPUs in System Mode during the collection interval;

(4) Wait%: the proportion of time that all CPUs are idle and waiting for I/O completion during the collection interval (Avg, Max);

(5) Idel%: The proportion of all CPUs in idle time during the collection interval (Avg, Max). The sum of this value and User%, Sys%, Wait% and Steal% is equal to 1;

(6) CPU%: Overall CPU usage, this value is usually equal to User%+Sys%;

(7)CPUs: The number of CPU cores of the operating system.

5. Memory status

The MEM subtable counts the memory usage of the system. Mainly includes the following contents:

(1) memtotal: total size of physical memory, unit MB;

(2) swaptotal: the total size of virtual memory (i.e., swap space);

(3) memfree: remaining physical memory size;

(4) swapfree: remaining virtual memory size;

(5) cached: the cache size occupied by cache in physical memory;

(6) active: memory size in active use;

(7) buffers: the size of the file system buffer;

(8) swapcached: the cache size occupied by cache in virtual memory;

(9) inactive: Memory size that is not commonly used.

6. System network conditions

The NET/NETPACKET sub-table is a summary of system network conditions. Reflects the network operation of the system and the number of data packets read and written by each network adapter in the system; the NET page displays the data transfer rate (kilobytes/second) of each network adapter in the system.

7. Disk busyness

The DISKBUSY subtable displays system disk busyness information.

(1) Normal situation, as shown in the figure below:

In the figure, dm2 corresponds to disk dbdata, dm1 corresponds to dbarch, and dm0 corresponds to dbbak. We can see that during off-duty hours, disk IO fluctuates greatly, but it is still idle most of the time. During off-duty hours, disk IO remains almost 100% busy. status, indicating that the business itself puts greater pressure on the database to read and write.

(2) Abnormal situations, as shown in the figure below:

In the figure, starting from 10:25, the IO busyness continues to remain at 100%. A long period of full busyness will cause the database to read and write slowly, block the application's read and write operations, and make the application's read and write slow down, resulting in Related issues; This situation requires analysis of application business and database transactions, and also requires analysis based on disk read and write speeds. It may be that the disk read and write performance has declined, causing such problems, or it may be abnormal transactions or new services. This leads to excessive read and write pressure on the database.

5. Open source operation and maintenance monitoring tool Prometheus

5.1 Tool introduction

Prometheus is an open source monitoring and alarm system and time series database (TSDB) developed by SoundCloud. Prometheus is developed using the Go language and is an open source version of Google's BorgMon monitoring system. The basic principle of Prometheus is to periodically capture the status of monitored components through the HTTP protocol. Any component can be accessed for monitoring as long as it provides the corresponding HTTP interface.

Prometheus is suitable for recording time series in text format. It is suitable for both machine-centric monitoring and monitoring of highly dynamic service-oriented architectures. Prometheus is designed to improve system reliability and can quickly diagnose problems during power outages. Each Prometheus Server is independent of each other and does not rely on network storage or other remote services.

DEM provides data collection function for Prometheus. By configuring monitoring resources in dem, metrics data can be generated for use by the Prometheus platform.

5.2 Scope of application

Prometheus tool is suitable for monitoring DM7 database and DM8 database.

5.3 Tool download and installation

For details on Prometheus software download and installation, see Prometheus official website .

5.4 Prometheus connects to DEM

DEM provides data collection function for Prometheus. By configuring monitoring resources in dem, metrics data can be generated for use by the Prometheus platform. How to use it:

1. Install DEM. For detailed steps of DEM deployment, see Section 2.4 DEM Deployment.
2. From DEM->System Management->Other Functions->prometheus_metric_nodes, set the database instance monitoring information that needs to be pushed.
3. Configure the prometheus.yml file and go to dem/metrics to get monitoring data, as follows:

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
    - targets: ['192.168.142.217:9100']
      labels:
         group: 'client-node-exporter'
    - targets: ['192.168.142.217:9161']
      labels:
         group: 'dmdb-node-exporter'
  - job_name: 'dem'
    metrics_path: '/dem/metrics'
    static_configs:
    - targets: ['192.168.142.241:8080']

1. Prometheus obtains the corresponding information.

1. To view the data obtained by Prometheus, you can obtain the PromQL query statement through the console.

---

links:
https://eco.dameng.com/document/dm/zh-cn/ops/tool-monitor