The original text of this article was first published on my personal blog www.ipengtao.com in 2017. Today I thought that I might mention it in follow-up articles. I will share it with you in advance so that you will have an impression when you mention it.
I gave a brief technical sharing to the R&D team before, mainly talking about why we need to monitor ourselves, the indicator dimensions of the monitoring, the basic monitoring process and the introduction of common monitoring tools, and finally how to quickly build our own monitoring tools from scratch. Please reply " PPT " in the background of "Tao Ge Talking Python ".
Tool introduction
1. Statsd is a network daemon that uses Node to develop. Its characteristic is to monitor various data information through UDP (good performance, timely hang up does not affect the main service) or TCP, and then send aggregated data to the back-end service for processing . Commonly supported "Graphite", "ElasticaSearch", "InfluxDB" and so on. It integrates client APIs in various languages. Here we use jsocol/pystatsd: A Python client for statsd for data collection.
2. Graphite is an open source programming interface written in Python. It is mainly used to collect the real-time status of the server. It is mainly used as the data backend of statsd. Divided into three sub-projects
-The carbon daemon, receiving raw statistics sent by StatsD.
-Whisper is a time series database used to store statistical data .
-graphite webapp is a web project used to graphically display statistical data
3. Grafana uses Go to develop, you can design and adjust your own statistical charts on the interface, support multiple alarms, and can be customized.
installation
Here I used [synthesize(https://github.com/obfuscurity/synthesize) to quickly install the Graphite and Statsd packages. Note that the installation data is under the **/opt/graphite** directory. Here we installed and started
service carbon-cache start # statsd数据处理后会进入中转
service memcached start # 缓存
service collectd start # 收集服务负载可选
service apache2 start # 这可以通过使用nginx替换
service statsite start # statsd的后端服务
Here you need to start the graphite-web application separately, the port is started: 0.0.0.0:8080, and the data source needs to be used later.
cd /opt/graphite/
sudo ./run-graphite-devel-server.py /opt/graphite/
Grafana data package can be downloaded and installed manually or installed via apt-get
sudo apt-get install grafana
After startup, you can see such a page by visiting port 3000. The default account password: admin, admin can log in by configuring github or google.
login
data collection
On the Python side, we use statsd for unified data management to the monitoring server collection
>>> import statsd
>>> c = statsd.StatsClient('localhost', 8125)
>>> c.incr('foo') # Increment the 'foo' counter.
>>> c.timing('stats.timed', 320) # Record a 320ms 'stats.timed'.
Here we collect the data to statsite through Python and display it through Graphite Web service.
Configure Graphite
Enter the Grafana backend, configure and click "Data Sources" to configure Graphite data (collected from front-end statsd)
data_source
Next, configure the corresponding data display:
add-graph
At this point, the basic data configuration is complete. Congratulations on having a complete monitoring system. Please "Taoge in chat P YT Hon " backstage reply " PPT " .
In order to facilitate understanding, let me add that the data is added from the application, and the time performance data of the application is collected and sent to the server through statsd . Through carbon statistics, whisper storage is finally displayed in graphite, where we use grafana instead. Graphite's native UI display is the effect of our title map.