Zabbix monitoring system ultra-detailed operation configuration

1. Overview of Zabbix

1. Reasons for using zabbix

As an operation and maintenance officer, you need to be able to use the monitoring system to check server status and website traffic indicators, and use the data from the monitoring system to understand the results of online publishing and the health status of the website.

Using an excellent monitoring software, we can:
●Browse all server status of the entire website through a friendly interface
●Conveniently view monitoring data on the Web front-end
●Can look back to find system problems and alarm conditions when an accident occurred

2. Concept and composition of zabbix

●zabbix is ​​an enterprise-level open source solution based on Web interface that provides distributed system monitoring and network monitoring functions.

●zabbix can monitor various network parameters to ensure the safe operation of the server system; and provides a flexible notification mechanism to allow system administrators to quickly locate/solve various existing problems.

●Zabbix consists of 2 parts, zabbix server and optional component zabbix agent. Data is collected through C/S mode and displayed and configured on the Web through B/S mode.

●zabbix server can provide remote server/network status monitoring, data collection and other functions through SNMP, zabbix agent, ping, port monitoring and other methods. It can run on platforms such as Linux.

●zabbix agent needs to be installed on the monitored target server. It mainly collects hardware information or operating system-related memory, CPU and other information.

Simple Network Management Protocol SNMP (Simple Network Management Protocol) is used for the management of network devices

3. Zabbix monitoring principle:

The zabbix agent is installed on the monitored host. The zabbix agent is responsible for regularly collecting various local data on the client and sending it to the zabbix server. After the zabbix server receives the data, it stores the data in the database. The user can see it based on the Zabbix WEB The data is presented as an image on the front end.

When zabbix monitors a specific project, the project will set a trigger threshold. When the monitored indicator exceeds the threshold set by the trigger, some necessary actions will be taken. The actions include: sending information (email, WeChat, SMS ), send commands (shell commands, reboot, restart, install, etc.).

4. Zabbix program components

Zabbix monitoring is deployed in the system and includes five common programs: zabbix_server, zabbix_agent, zabbix_proxy, zabbix_get, zabbix_sender, etc.

●zabbix server: zabbix server daemon, in which the data of zabbix_agent, zabbix_get, zabbix_sender, and zabbix_proxy are finally submitted to zabbix server;

●zabbix agent: client daemon, responsible for collecting client data, such as collecting CPU load, memory, hard disk usage, etc.;

●zabbix proxy: zabbix distributed proxy daemon, usually more than 500 hosts, requires distributed monitoring architecture deployment;

●zabbix get: zabbix data receiving tool, a command used alone, usually executed on the server or proxy side to obtain remote client information;

●zabbix sender: Zabbix data sending tool. Users send data to the server or proxy. Usually it takes a long time for users to check.

2. Install zabbix 5.0

Environment preparation:

zabbix-server has at least 2G of memory, and 4G is recommended;

192.168.2.66 zabbix server

192.168.2.200 zabbix client

1. Steps to deploy zabbix server

1.1 Turn off the firewall and set the host name

systemctl disable --now firewalld
setenforce 0
hostnamectl set-hostname zbx-agent01 

1.2 Obtain the download source of zabbix

rpm -ivh https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/zabbix-release-5.0-1.el7.noarch.rpm  

1.3 Replace zabbix.repo with Alibaba source

cd /etc/yum.repos.d
sed -i 's#http://repo.zabbix.com#https://mirrors.aliyun.com/zabbix#' zabbix.repo
 
yum clean all && yum makecache
 
yum install -y zabbix-server-mysql zabbix-agent

1.4 Install SCL (Software Collections)

To facilitate the subsequent installation of higher versions of PHP, the default PHP version installed by yum is 5.4, which is too low. Zabbix 5.0 requires a minimum PHP version of 7.2.0.

SCL enables the use of multiple versions of software on the same machine without affecting the dependency environment of the entire system. The software package will be installed in the /opt/rh directory.

yum install -y centos-release-scl 

1.5 Modify the zabbix-front front-end source and install the zabbix front-end environment to the scl environment

vim zabbix.repo
......
[zabbix-frontend]
......
enabled=1 #开启安装源
......
 
yum install -y zabbix-web-mysql-scl zabbix-apache-conf-scl 

1.6 Install the database required by zabbix

yum install -y mariadb-server mariadb
 
systemctl enable --now mariadb
 
mysql_secure_installation #初始化数据库,并设置密码,如 123

1.7 Add database user and database information required by zabbix

mysql -u root -p123
 
CREATE DATABASE zabbix character set utf8 collate utf8_bin;
GRANT all ON zabbix.* TO 'zabbix'@'%' IDENTIFIED BY 'zabbix';
flush privileges;

1.8 Import database information

rpm -ql zabbix-server-mysql #查询 sql 文件的位置
 
zcat /usr/share/doc/zabbix-server-mysql-5.0.16/create.sql.gz | mysql -uroot -p123 zabbix 

1.9 Modify the zabbix server configuration file and change the database password

vim /etc/zabbix/zabbix_server.conf
......
DBPassword=zabbix   #124行,指定 zabbix 数据库的密码 

1.10 Modify the php configuration file of zabbix

vim /etc/opt/rh/rh-php72/php-fpm.d/zabbix.conf
......
php_value[date.timezone] = Asia/Shanghai    #24行,取消注释,修改时区  

1.11 Start zabbix related services

systemctl restart zabbix-server zabbix-agent httpd rh-php72-php-fpm
systemctl enable zabbix-server zabbix-agent httpd rh-php72-php-fpm 

1.12 Browser access: http://192.168.2.66/zabbix

Click Next to set the database password zabbix

After the installation is complete, the default login account and password are: Admin/zabbix

Setting file interface: Click [User settings] on the left menu bar, select Chinese(zh_CN) for [Language], and then click Update to update.

1.13 Solve the problem of Chinese garbled characters on the zabbix-server Web page

yum install -y wqy-microhei-fonts
 
\cp -f /usr/share/fonts/wqy-microhei/wqy-microhei.ttc /usr/share/fonts/dejavu/DejaVuSans.ttf

2. Example operation: deploy zabbix server

1.1 Turn off the firewall and set the host name

1.2 Obtain the download source of zabbix and replace zabbix.repo with the Alibaba source

1.3 Install SCL (Software Collections)

1.4 Modify the zabbix-front front-end source and install the zabbix front-end environment to the scl environment

1.5 Install the database required by zabbix

1.6 Add database user and database information required by zabbix

1.7 Import database information

1.8 Modify the zabbix server configuration file and change the database password

vim /etc/zabbix/zabbix_server.conf

1.9 Modify the php configuration file of zabbix

1.10 Start zabbix related services

1.11 Browser access: http://192.168.2.66/zabbix

Access using the host’s Firefox browser

1.12 Solve the problem of Chinese garbled characters on the zabbix-server Web page

Refresh the browser to see if the garbled code problem is resolved.

3. Deploy zabbix client

Zabbix 5.0 version uses a new version of client agent2 developed in golang language.

The zabbix server zabbix_server uses port 10051 by default, and the client zabbix_agent2 uses port 10050 by default.

3.1 Turn off the firewall and set the host name

systemctl disable --now firewalld
setenforce 0
hostnamectl set-hostname zbx-client  

Add mapping relationship (added on both server and client)

vim /etc/hosts
192.168.2.200 zbx-client
192.168.2.66 zbx-server

3.2 Configure time synchronization on both the server and the client

yum install -y ntpdate
ntpdate -u ntp.aliyun.com

3.3 The client configures the time zone to be consistent with the server

mv /etc/localtime{,.bak}
ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
 
date

3.4 Set the download source of zabbix and install zabbix-agent2

rpm -ivh https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/zabbix-release-5.0-1.el7.noarch.rpm
 
cd /etc/yum.repos.d
sed -i 's#http://repo.zabbix.com#https://mirrors.aliyun.com/zabbix#' /etc/yum.repos.d/zabbix.repo
 
yum install -y zabbix-agent2

3.5 Modify agent2 configuration file

vim /etc/zabbix/zabbix_agent2.conf
......
Server=192.168.2.66   #80行,指定 zabbix 服务端的 IP 地址
ServerActive=192.168.2.66 #120行,指定 zabbix 服务端的 IP 地址
Hostname=zbx-client #131行,指定当前 zabbix 客户端的主机名

3.6 Start zabbix-agent2

systemctl start zabbix-agent2
systemctl enable zabbix-agent2
 
netstat -natp | grep zabbix
tcp6 0 0 :::10050 :::* LISTEN 43654/zabbix_agent2

3.7 Verify the connectivity of zabbix-agent2 on the server side

yum install -y zabbix-get   #安装 zabbix 主动获取数据的命令
 
zabbix_get -s '192.168.2.200' -p 10050 -k 'agent.ping'
1
 
zabbix_get -s '192.168..200' -p 10050 -k 'system.hostname'
zbx-client

The zabbix_get command is a command used on the server side to check the agent side. It is often used to check whether the configuration of the agent is correct. It can easily know whether the key can obtain data normally. It is especially useful when testing customized monitoring.

zabbix_get parameter description:
-s --host: Specify the client host name or IP
-p --port: Client port, the default is 10050, the default is 10050 port, you can not write
-I --source-address: Specify the source IP, write Just enter the IP address of the Zabbix server. Generally, leave it blank. If the server has multiple IP addresses, you can specify one.
-k --key: the key you want to obtain

3.8 Add agent host to the Web page

Click [Host] in [Configuration] on the left menu bar, click [Create Host]
[Host Name] set to zbx-client
[Visible Name] set to zbx-client-192.168.2.200
[Group] Select Linux server
[Interfaces] 】The [IP address] is set to 192.168.2.200


Then click [Template] [Link new tamplates] on the upper menu bar, search for Linux, select Template OS Linux by Zabbix agent
and click [Add]

4. Example operation: deploy zabbix client

4.1 Turn off the firewall and set the host name

Add mapping relationships on both the server and client

4.2 Configure time synchronization on both the server and the client

4.3 The client configures the time zone to be consistent with the server (both the server and the client are set)

4.4 Set the download source of zabbix and install zabbix-agent2

4.5 Modify agent2 configuration file

4.6 Start zabbix-agent2

4.7 Verify the connectivity of zabbix-agent2 on the server side

4.8 Add agent host to Web page

3. Case operation: Customized monitoring content

Case: Customized monitoring of the number of people logged in to the client server

Requirement: Limit the number of logins to no more than 3. If more than 3, an alarm message will be issued.

1. Operation steps for customizing monitoring content

1.1 Create a custom key on the client

Clearly identify the linux commands that need to be executed

who | wc -l  

1.2. Create zabbix monitoring item configuration file for custom key

vim /etc/zabbix/zabbix_agent2.conf
#可以将自定义的监控项配置文件创建在 zabbix_agent2.d 目录中
268 Include=/etc/zabbix/zabbix_agent2.d/*.conf
#自定义监控项的格式如下
291 #   Format: UserParameter=<key>,<shell command>
 
cd /etc/zabbix/zabbix_agent2.d/
 
vim UserParameter_login.conf
UserParameter=login.user,who|wc -l
 
systemctl restart zabbix-agent2 

1.3. Verify the newly created monitoring items on the server side

zabbix_get -s '192.168.2.200' -p 10050 -k 'login.user' 

1.4 Create a custom monitoring item template on the web page

1. Create a template

Click [Template] in [Configuration] on the left menu bar, click [Create Template]
[Template Name] Set to Template Login User
[Visible Name] Set to Template Login User
[Group] Select Template
[Description] to customize
the click [Add], now you can search for Template Login User in [Linked Templates]

2. Create an application set (used to manage monitoring items)

Click [Application Set] on the upper menu bar, click [Create Application Set],
set [Name] to Login User,
and click [Add]

3. Create monitoring items

Click [Monitoring Items] on the upper menu bar, click [Create Monitoring Item]
[Name] set to Number of login users
[Key Value] set to login.user #The key value must be consistent with the setting in the custom monitoring item configuration file Consistent
[Update Interval] is set to 10s
[Historical data retention time] Storage period 30d #The retention time can be customized.
Click [Add]

4. Create a trigger (when the monitoring item obtains the monitored value, compare it with the preset value of the trigger to determine whether to alarm)

Click [Trigger] on the upper menu bar, click [Create Trigger]
[Name] set to Number of login users is greater than 3
[Severity] set to normal severity
[Expression] click Add, [Monitoring Item] click to select Number of login users, [Function] select last(), [Result] select > 3, click [Insert] and
click [Add]

5. Create graphics

Click [Graph] on the menu bar above, click [Create Graph]
[Name] and set it to Number of login users
[Width] and [Height] can directly use the default values.
[Monitoring Item] Click Add to check the relevant monitoring item Number of login users. [Function] Select the maximum value, and keep the default values ​​for other values.
Click [Add]

6. Associate the host with the template (one host can be associated with multiple templates)

Click [Host] in [Configuration] on the left menu bar, click the host you want to associate, click
[Template] on the upper menu bar, [Link new tamplates], search for login, select Template Login User, and click [Update]

At this time, click [Host] in [Monitoring] and click [Graph] of the host you associated to view the relevant monitoring item indicators.

7.Set email alarm

Click [Alarm Media Type] in [Management] on the left menu bar, click [Create Media Type],
set [Name] to qq_Email,
[SMTP Server], set it to smtp.qq.com
, [SMTP Server Port], set it to 25
[SMTP HELO] Set it to qq.com
[SMTP email] Set it to your own email address
[Authentication] Select the user name and password
[User name] Set it to your own email address
[Password] You can log in to the QQ mailbox page, click [Settings] --> [Account] 】【Generate Authorization Code】, get the authorization code through SMS.
【Description】can be customized.
Click [Message templates] on the upper menu bar, click [Add], [Message type] select the question, click [Update],
click [Add], and test functionality

Click [User settings]-->[Alarm Media] on the left menu bar, click [Add] [
Type] select qq_Email
[Recipient] set to
[When enabled] set to 1-7,00:00-24:00
Click [Add]
and then [Update]

Click [Action] in [Configuration] on the left menu bar, select the corresponding action name, and click [Enable]

1.5 Test email alarm

If the number of user logins to the test client exceeds the preset value of the trigger, check [Monitoring] --> [Dashboard] to confirm the alarm.

2. Example operation: Customize monitoring content

2.1 Create a custom key on the client

2.2. Create zabbix monitoring item configuration file for custom key

2.3. Verify the newly created monitoring items on the server side

2.4 Create custom monitoring item templates on the web page

2.5.Set email alarm

2.6 Test email alarm

4. Zabbix automatic discovery and automatic registration

1. Zabbix automatic discovery (passive mode for agent2)

Zabbix server actively discovers all clients and then registers the client information on the server.

The disadvantage is that if there are many hosts in the defined network segment, zabbix server registration will take a long time and the pressure will be greater.

1.1. Ensure that the zabbix-agent2 service status on the client is normal

systemctl is-active zabbix-agent2.service
active 

1.2. Delete the original client host on the Web page

Click [Host] in [Configuration] on the left menu bar, check the original client host, and click Delete

1.3. Configure hosts resolution on the server and client

vim /etc/hosts
192.168.2.66 zbx-server
192.168.2.200 zbx-client  

1.4. Configure automatic discovery on the Web page

Click [Auto Discovery] in [Configuration] on the left menu bar, click [Create Discovery Rule]
[Name] set to mynetwork
[IP Range] set to 192.168.229.1-254
[Update Interval] set to 30s
[Check] click [Add] ], [Check type] select Zabbix client, [Port range] is set to 10500, [Key value] is set to system.uname [Device
uniqueness criterion] Select IP address
[Host name] Select DNS name
[Visible name] select Host name
Check [Enabled] and click [Update]

Click [Actions] in [Configuration] on the left menu bar, select [Discovery actions] from the upper menu,
check [Auto discovery. Linux servers.], and click [Enable]

Click [Host] in [Configuration] on the left menu bar to refresh. After a period of time, the automatically discovered client hosts will be refreshed.

1.5 Zabbix logs can be viewed on the server

tail -f /var/log/zabbix/zabbix_server.log
......
6601:20210922:225044.115 enabling Zabbix agent checks on host "zbx-client": host became available

2. Example operation: zabbix automatic discovery (passive mode for agent2)

2.1. Ensure that the zabbix-agent2 service status on the client is normal

2.2. Delete the original client host on the Web page

2.3. Configure hosts resolution on the server and client

It has been added in the previous experiment. If it has not been added, it can be added in this step.

2.4. Configure automatic discovery on the Web page

2.5 Zabbix logs can be viewed on the server

3. Zabbix automatically registers (active mode for agent2)

zabbix agent2 will actively report its own information and send it to zabbix server.

The disadvantage is that zabbix agent2 may not be able to find the zabbix server due to incorrect configuration of the configuration file or network failure.

3.1.Environment preparation

Click [Auto Discovery] in [Configuration] on the left menu bar, check the discovery rule, and click Disable

Click [Host] in [Configuration] on the left menu bar, check the original client host, and click Delete

vim /etc/hosts
192.168.2.66 zbx-server
192.168.2.200 zbx-client 

3.2. Modify zabbix-agent2 configuration file

vim /etc/zabbix/zabbix_agent2.conf
......
HostnameItem=system.hostname    #139行,取消注释
 
egrep -v "^#|^$" /etc/zabbix/zabbix_agent2.conf
PidFile=/var/run/zabbix/zabbix_agent2.pid
LogFile=/var/log/zabbix/zabbix_agent2.log
LogFileSize=0
Server=192.168.2.66
ServerActive=192.168.2.66
Hostname=zbx-client
HostnameItem=system.hostname
Include=/etc/zabbix/zabbix_agent2.d/*.conf
ControlSocket=/tmp/agent.sock

3.3. Configure automatic registration on the Web page

Click [Action] in [Configuration] on the left menu bar, select [Autoregistration actions] in the upper menu, click [Create Action] [
Name], set it to Auto registration
, click [Add], [Type] select the host name, [Operator] select Contains, [Value] is set to zbx-agent01
, click [Operation] on the upper menu bar, click [Add], [Operation Type] selects to add a host, click [add]
and then click [Add], [Operation Type] selects to add to the host group Group, [Host Group] select Linux servers, click [add]
and then click [Add], [Operation Type] select associate with template, [Template] search Linux, select Template OS Linux by Zabbix agent, click [add] and click
below [Add]

Click [Host] in [Configuration] on the left menu bar to refresh. After a period of time, the automatically discovered client hosts will be refreshed.

3.4 View zabbix logs on the server

tail -f /var/log/zabbix/zabbix_server.log

4. Example operation: zabbix automatic registration (active mode for agent2)

4.1.Environment preparation

4.2. Modify zabbix-agent2 configuration file

4.3. Configure automatic registration on the Web page

4.4 View zabbix logs on the server

5. Deploy zabbix proxy server

The role of distributed monitoring:

●Share the centralized pressure of the server

●Solve the network delay problem between multiple computer rooms

agent --> proxy --> server

1. Steps to deploy zabbix proxy server

Environment preparation:

192.168.2.66 zabbix server (deployed)

192.168.2.200 zabbix client (deployed)

192.168.229.33 zabbix proxy server

1.1 Turn off the firewall and set the host name

systemctl disable --now firewalld
setenforce 0
hostnamectl set-hostname zbx-proxy 

1.2 Set the download source of zabbix according to zabbix-proxy

rpm -ivh https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/zabbix-release-5.0-1.el7.noarch.rpm
 
cd /etc/yum.repos.d
sed -i 's#http://repo.zabbix.com#https://mirrors.aliyun.com/zabbix#' /etc/yum.repos.d/zabbix.repo
 
yum install -y zabbix-proxy-mysql zabbix-get 

1.3 Install the database required by zabbix

yum install -y mariadb-server mariadb
 
systemctl enable --now mariadb
 
mysql_secure_installation #初始化数据库,并设置密码,如 123

1.4 Add database user and database information required by zabbix

mysql -u root -p123
 
CREATE DATABASE zabbix_proxy character set utf8 collate utf8_bin;
GRANT all ON zabbix_proxy.* TO 'zabbix'@'%' IDENTIFIED BY 'zabbix';
flush privileges;

1.5 Import database information

rpm -ql zabbix-proxy-mysql #查询 sql 文件的位置
 
zcat /usr/share/doc/zabbix-proxy-mysql-5.0.16/schema.sql.gz | mysql -uroot -pabc123 zabbix_proxy

1.6 Modify zabbix-proxy configuration file

vim /etc/zabbix/zabbix_proxy.conf
Server=192.168.2.66   #30行,指定 zabbix 服务端的 IP 地址
Hostname=zbx-proxy  #49行,指定当前 zabbix 代理服务器的主机名
DBPassword=zabbix   #196行,指定当前数据库 zabbix 用户的密码 

1.7 Start zabbix-proxy

systemctl start zabbix-proxy
systemctl enable zabbix-proxy 

1.8 Configure hosts resolution on all hosts

vim /etc/hosts
192.168.2.66 zbx-server
192.168.2.200 zbx-client
192.168.22.77 zbx-proxy 

1.9 Configure the agent on the Web page

Click [Action] in [Configuration] on the left menu bar, check the automatic registration rule, and click Disable. Click
[Host] in [Configuration] on the left menu bar, check the original client host, and click Delete.

Click [Agent Agent] in [Management] on the left menu bar, click [Create Agent]
[Agent Agent Name] set to zbx-proxy
[System Agent Mode] select active
[Agent Address] set to 192.168.2.77
Click 【Add to】

1.10 Configure agent to use proxy

Modify the agent2 configuration file on the client side

vim /etc/zabbix/zabbix_agent2.conf
......
Server=192.168.2.77   #80行,指定 zabbix 代理服务器的 IP 地址
ServerActive=192.168.2.77 #120行,指定 zabbix 代理服务器的 IP 地址

1.11 Configure on the Web page

Click [Host] in [Configuration] on the left menu bar, click [Create Host]
[Host Name], set it to zbx-client
[Visible Name], set it to zbx-client [Group], select the [IP] of
the Linux server [Interfaces]
Address] Set to 192.168.2.200
[Monitored by agent] Select zbx-proxy


Then click [Template] [Link new tamplates] on the upper menu bar, search for Linux, select Template OS Linux by Zabbix agent
and click [Add]

1.12 Restart the service on the client and proxy server respectively

systemctl restart zabbix-agent2
systemctl restart zabbix-proxy

Click [Host] in [Configuration] on the left menu bar to refresh and check that the client host monitoring status is normal.

1.13 View logs on the server

tail -f /var/log/zabbix/zabbix_proxy.log 

2. Example operation: deploy zabbix proxy server

2.1 Turn off the firewall and set the host name

2.2 Set the download source of zabbix

2.3 Install the database required by zabbix

2.4 Add database user and database information required by zabbix

2.5 Import database information

2.6 Modify zabbix-proxy configuration file

2.7 Start zabbix-proxy

2.8 Configure hosts resolution on all hosts

2.9 Configure the agent on the Web page

2.10 Configure agent to use proxy

Modify the agent2 configuration file on the client side

2.11 Configure on the Web page

2.12 Restart the service on the client and proxy server respectively

2.13 View logs on the server

6. Set up zabbix-snmp monitoring

SNMP, Simple Network Management Protocol

1. Steps to set up zabbix-snmp monitoring

1.1. Install the snmp monitoring program on the server

yum install -y net-snmp net-snmp-utils  

1.2. Modify the snmp configuration file and start the service

vim /etc/snmp/snmpd.conf
......
view systemview included .1 #57行,添加此配置
 
systemctl start snmpd 

1.3. Use snmp command to test

snmpwalk -v 2c -c public 127.0.0.1 sysname
SNMPv2-MIB::sysName.0 = STRING: zbx-server
-------------------------------------------------------
–v 1|2c|3:指定SNMP协议版本
–c:指定共同体字符串
sysname:为 snmp 的 key
----------------------------------------- 

1.4. Configure snmp mode monitoring on the Web page

Click [Host] in [Configuration] on the left menu bar, and click Zabbix server

[Interfaces] Click [Add] to select SNMP and set the port to 161

Click [Template] on the upper menu bar, select all agent templates in [Linked Templates] and select Unlink and Clean

Then search for Linux SNMP in [Link new templates] and select Template OS Linux SNMP

Click [Update], and after waiting for a period of time, the availability of Zabbix server will change to SNMP monitoring mode.

2. Example operation: Set up zabbix-snmp monitoring

2.1. Install the snmp monitoring program on the server

2.2. Modify the snmp configuration file and start the service

2.3. Use snmp command to test

2.4. Configure snmp mode monitoring on the Web page

Guess you like

Origin blog.csdn.net/weixin_69148277/article/details/130922989