Ganglia is a very good open source monitoring framework

Ganglia is a very good open source monitoring framework, the following is an explanation from Wikipedia:
Ganglia is an open source cluster monitoring project initiated by UC Berkeley, designed to measure thousands of nodes. The core of Ganglia consists of gmond, gmetad and a web front end. It is mainly used to monitor system performance, such as: cpu, mem, hard disk utilization, I/O load, network traffic, etc. It is easy to see the working status of each node through the curve, adjust and allocate system resources reasonably, improve The overall performance of the system plays an important role.
Each computer runs a daemon named gmond that collects and sends metrics data. A host that receives all metric data can display this data and can pass a reduced form of this data into the hierarchy. It is because of this hierarchy pattern that Ganglia can scale well. There is very little system load from gmond, which makes it a piece of code that runs on each machine in the cluster without affecting user performance. Collecting all this data multiple times can affect node performance. "Jitter" in the network occurs when a large number of small messages occur at the same time, and this problem can be avoided by keeping the node clocks consistent.
gmetad can be deployed on any node in the cluster or an independent host connected to the cluster through the network. It communicates with gmond through unicast routing, collects the status information of nodes in the area, and saves it in the database in the form of XML data .
The data is processed by the RRDTool tool, and the corresponding graphic display is generated, which is intuitively provided to the client in the form of Web.

1. Ganglia's components
Ganglia includes the following programs, which transmit monitoring data in XML format to achieve monitoring effects.
Gmetad This program is responsible for collecting the data of each cluster and updating it to the rrd database.
Gmond collects monitoring data from this machine and sends it to other machines. Collect monitoring data from other machines for Gmetad to read. Default listening port 8649
web front-end A web-based monitoring interface that needs to be installed on the same node as Gmetad, fetches data from Gmetad, reads the rrd database, generates pictures, and displays them.
Second, Ganglia's working mode
Ganglia's data collection work can work in unicast (unicast) or multicast (multicast) mode, the default is multicast mode.
Unicast: Send the monitoring data collected by yourself to a specific machine or machines, which can cross network
segments Multicast: Send the monitoring data collected by yourself to all machines in the same network segment, and collect the same network segment at the same time Monitoring data sent from all machines in the system. Because it is sent in the form of broadcast packets, it needs to be in the same network segment. However, within the same network segment, different sending channels can be defined.

The topology of a single ganglia is as follows:



The topology of a clustered ganglia is as follows:

Java code  

1. The data between gmond nodes and between gmond and gmetad are all transmitted in xml format. The data transmission mode between gmond nodes not only supports unicast point-to-point transmission, but also supports multicast. If hundreds of servers are stacked in a local area network , sending data every 15 seconds, this feature can still save some network overhead  
2. In order to understand the ganglia report faster, there are several terms in the report that need to be clarified, which are also the embodiment of the structure level  
3. node: the smallest unit, that is, a single server  
4. cluster: server cluster, consisting of multiple servers  
5. grid: grid, consisting of multiple server clusters  
6. The relationship between the above nouns and entities is:  
7. 1. One grid corresponds to one gmetad, and multiple lines of data_source can be specified in gmetad.conf  
8. 2. A cluster is a line of data_source in gmetad.conf, and a line can specify one or more gmonds for gmetad to collect  
9. 3. A node is a gmond, gmond is responsible for collecting the data of the machine where it is located, and gmond can also temporarily store data from other gmonds  
10. From the correspondence between nouns and entities and the two data flow diagrams just now, it can be seen that the same requirement can have multiple configuration schemes, which is different from most software, so it is easy to get confused when you first come into contact with ganglia ,for example:  
11. A cluster has multiple gmonds. You can list all gmonds in the data_source of gmetad, and gmetad polls all gmonds to obtain data. You can also aggregate the data of multiple gmonds into one gmond, and gmetad polls the aggregated gmonds to obtain the entire gmond. cluster data  
12. Several features of gmond:  
13. 1. gmond can wait for gmetad to collect the data in person, or send its own data to other gmonds to deliver gmetad  
14. 2. The communication between gmond nodes is based on udp, and the poll request received by gmetad is based on tcp (port: 8649 ), and gmond can send data to multiple gmonds at the same time  
15. 3. Also, the communication between gmonds mentioned above supports broadcast as well as unicast.