Introduction to Linux system performance monitoring tools-tsar

Performance monitoring tools
In the process of using Linux, the more troublesome thing is that the system provides many Linux system monitoring tools. How to use these tools fully and reasonably to find out the performance bottlenecks of system operation, including CPU, memory, disk, and network bottlenecks. Similar to the Internet, there are 20 basic tools that many administrators have to learn. I will not introduce them one by one here. Here are some additional monitoring tools that are used and what you see is what you get to help you find the problem faster.

System resource breakdown When
it comes to system performance monitoring and analysis tools, I have to mention Brendan Gregg’s system analysis. His diagram systematically shows the relationship between applications, system calls, kernels, protocol stacks, and hardware. Interactive. Interested friends can monitor, analyze, and locate system problems by combining the commands in this picture.

Monitoring tool:
Introduction to
tsar Tsar is a collection tool developed by Taobao itself. It is mainly used to collect server system information (such as cpu, io, mem, tcp, etc.) and application data (such as squid haproxy nginx, etc.). The collected data is stored on the disk, and historical information can be queried at any time. The output method is flexible and diverse. In addition, it supports storing the data in mysql or sending data to the nagios alarm server. When tsar displays data, you can specify the module, and can merge the data of multiple pieces of information. With the -live parameter, you can output real-time information in seconds.

The overall architecture
Tsar is a program based on modular design. The program consists of two parts: a framework and a module.
The source code of the framework program is mainly in the src directory, and the source code of the module is mainly in the modules directory.
The framework provides parsing of configuration files, loading of modules, parsing of command line parameters, and parsing and output of module raw data through the interface of application modules. The module provides an interface for the framework to call.
Tsar relies on cron to collect data every minute, so it needs the system to install and enable crond. After installation, tsar will execute tsar -cron every minute to collect information regularly and record it to the original log file.

tsar environment installation guide:

]# wget -O tsar.zip https://github.com/alibaba/tsar/archive/master.zip --no-check-certificate
]# unzip tsar.zip
]# cd tsar
]# make && make install

Error report question 1: As shown in the figure, the command is not found, just use yum to install an unzip.

Error Issue 2: Make compilation error is reported because gcc is not installed. After installing the gcc compiler through yum, execute the following make.

Timed tasks during deployment, which can be used directly later

]# crontab -e * * * * * root /usr/bin/tsar --cron > /dev/null 2>&1

Tsar configuration introduction

The introduction of tsar was introduced earlier, now let's take a look at the configuration of tsar.

Timing task configuration: /etc/cron.d/tsar

cat  /etc/cron.d/tsar

As shown above, /etc/cron.d/tsar is responsible for calling tsar commands in the role of root user every minute to perform data collection.

Log file rotation: /etc/logrotate.d/tsar

cat /etc/logrotate.d/tsar

In the log file rotation configuration, tsar's local storage is rotated every month, and the data is also set here under /var/log/tsar.data

Configuration file: /etc/tsar/tsar.conf

cat /etc/tsar/tsar.conf

/etc/tsar/tsar.conf is responsible for the specific configuration of tsar's acquisition module and output; configure which modules are enabled, output and other content here.

tsar module library

Module path: /usr/local/tsar/modules, dynamic library so files of each module;

Introduction to the use of tsar

In the use of tsar, you can refer to the following help information to complete the corresponding monitoring.

tsar -h

Usage: tsar [options]
Options:
-check View the last collected data
--check/-C View the reminder information of the last tsar, such as: tsar --check / tsar --check --cpu --io
--cron /-c Use crond mode for tsar monitoring
--interval/-i Specify the tsar interval time, the default unit is minutes, with the --live parameter, the unit is seconds
--list/-L List the enabled modules
--live /-l View real-time data
--file/-f specify the input file
--ndays/-n specify the number of data days in the past, the default is 1 day
--date/-d specify the date, YYYYMMDD or n represents n days ago
--detail/ -D can specify to view the main field or all the fields of the module
--spec/-s specify the field, tsar --cpu -s sys,util

Modules Enabled:
--cpu List CPU-related monitoring counts
--mem physical memory usage
--swap virtual memory usage
--tcp TCP protocol IPV4 usage
--udp UDP protocol IPV4 usage
--traffic Network outgoing usage--
io Linux IO situation--
pcsw process and context switch-- partition
disk usage--
tcpx TCP connection-related data parameters--
load system load situation

tsar cpu monitoring:

As shown below, using the parameter -cpu can monitor the cpu of the system, the parameter user represents the user space cpu, sys kernel space cpu usage, wait is the cpu usage corresponding to IO, hirq and sirq are hardware interrupts and software interrupts respectively. , Util is the total amount of cpu used by the system. The data in the following table shows that the current system has used about 30% of the cpu.