If your purpose is clear is directed to the title, I do not like nagging, please go directly to the content after the second dividing line.
Before actually going to do is to have Swift monitoring platform, but because it needs no hard and fast, it does not matter, it has been shelved. Recent laboratory came a little MM sophomore junior year experience life with her boss let us do this thing together, the main benefit is that we can learn some practical techniques to understand the basic architecture of cloud storage, while not need to go into detail, it is difficult to understand.
Well, I was a love that crap and lay the girl, so that it reflects the "original" is not? O (∩_∩) O haha ~ to business. . .
================================ the roast out of the parting line ======== 38 ° Hangzhou ========================
On the monitoring of the distributed file system, divided into two parts: monitor machine performance monitoring and related operational status of the system.
Machine performance-related monitoring: This comparison generalization, more mature, mainly to monitor the CPU, memory, network, disk I / O and other information, we do not need to repeat create the wheel, you can choose more convenient powerful open source tools to build here we choose Ganglia;
Monitor the operation of the system: The associated with different file systems, although different DFS also have some common content, such as accounts, logs, request amount, but specific to each DFS implementation but it is still very different, requiring special customization. Taking into account the concept of swift in the ring, ReST the interface, account creation and other information, that a part of our intention to develop their own, and can provide ring to create, add accounts and other basic admin tools;
Integration: to provide a unified entrance, uniform style, which requires us to Ganglia need to do some secondary development, so as to integrate with the swift-related monitoring section, taking into account the ganglia web front using the PHP language development, the operating environment for Apache2. And we intend to develop for the swift-related part with JSP, running in Tomcat, and therefore need to integrate PHP + JSP environment, and then again after this introduction.
Well, that's the whole idea, step by step, This part describes the Ganglia.
=====================================Ganglia=====================================
Ganglia cluster monitoring is a tool to create and open source by the UC Berkeley. Ganglia Chinese meaning is the nerve center, now supports multi-part operating systems (including linux, unix, windows), to support network monitoring 2000 nodes (of course this is not the limit, just a great example of the use of cluster).
basic structure
Ganglia underlying data obtained using RRDTool, Ganglia process is divided into two components:
- gmond（ganglia monitor deamon）
- gmetad（ganglia metadata deamon）
Wherein data, gmond runs on each node of the cluster, collected RRDTool generated; The gmetad running on the monitoring server, to collect the data for each gmond. Ganglia also provides a PHP web front end to achieve, generally using Apache2 as its operating environment, we can see a variety of intuitive cluster data chart from the Web Front.
Ganglia hierarchical structure to do very well, from small to large can be divided into node -> cluster -> grid, these three levels.
- A node is a need to monitor nodes, usually hosts, represented by IP. On each node running a gmond process used to collect the data and submit it to gmetad.
- A cluster of a plurality of node composed, is a cluster, we can give the cluster definition name. A cluster node can choose a running gmetad process, summary / pull data gmond submitted, and deploy web front, data will be collected gmetad show it graphically.
- A grid composed of a plurality cluster composed, is a higher level concept, we can define the name to the grid. grid can be defined in a top gmetad process, summary / pull more gmond, sub gmetad data submitted to deploy web front, top gmetad the data collected show it graphically.
Obviously, this approach is very flexible and can achieve data monitoring various structures. In the figure below, we can clearly see that this hierarchical structure, and a different deployment.
Cluster deployment
Here, we introduce only deployed on ubuntu, because it is simple! Only you need to install two packages: ganglia-monitor and ganglia-webfrontend. Ganglia-webfrontend which only needs to be installed on the server to show the page, and if you do not ubuntu Apache2 and PHP environment, then it will automatically help you install.
Here I introduce direct multicast (ganglia default mode, in addition to support unicast) cluster deployment, as to stand-alone mode, in fact, as long as the ganglia-monitor and ganglia-webfrontend installed on a machine on it.
1) cluster environment
gmond: 4 units (192.168.1.101,192.168.1.102,192.168.1.103,192.168.1.104)
gmetad + Ganglia Web: 1 Taiwan (192.168.1.104)
operating system: Ubuntu 11.04
2) the deployment process
installation 192.168.1.101,192.168.1.102,192.168.1.103 ganglia-monitor in this run only three nodes gmond:
sudo apt-get install ganglia-monitor
Installation ganglia-monitor and ganglia-webfrontend on this 192.168.1.104 node contains all the ganglia services, this process will be installed by default Apache2 and PHP for the Apache and PHP system does not have the environment:
sudo apt-get install ganglia-webfrontend ganglia-monitor
After ganglia-webfrontend installed, ganglia-webfrontend this package will by default install Web-related code in the "/ usr / share / ganglia-webfrontend /" path, so apache not visit. Therefore, you can use a soft link, or directly to the next directory "/ var / www /" directory.
sudo ln -s /usr/share/ganglia-webfrontend/ /var/www/ganglia
or:
sudo mv /usr/share/ganglia-webfrontend/ /var/www/ganglia
3) Configuration
In our monitoring system, the cluster named swift, grid named MyGrid, and using multicasting.
Configuration gmond: Open /etc/ganglia/gmond.conf modify cluster name:
cluster { name = "swift" owner = "unspecified" latlong = "unspecified" url = "unspecified" }
The gmetad Configuration: Open /etc/ganglia/gmetad.conf grid and add a data source name:
data_source "swift" localhost gridname "MyGrid"
4) Start
to restart 192.168.1.104 of gmated:
service gmated restart
Restart each node gmond (note the service name-Monitor Ganglia , not gmond!):
service ganglia-monitor restart
5) page
After completion of the above work, ganglia of the deployment is complete, and now you can open a browser to http directly on 192.168.1.104: // localhost / ganglia, the graphical interface will be able to see it! Or you can directly access via IP, if you change the port be sure to add a new port access.
Here is a standalone deployment of web front end capture, schematic schematic ^ _ ^ ~ this stand-alone service, I will be their cluster named "yuki-cluster", ha ha
So far, the work of the deployment is complete it! = D
important point
- Intended use unicast approach to deployment, the IP address of each gmond gmetad specified in the profile, but failed.
- In using multicast in a timely manner, if the node where gmetad ganglia-monitor does not run, can not collect data from other pure ganglia-monitor, I do not know why.
- If the data inconsistency in the process of starting, such as 4-core CPU obviously, but only see one, there may be ganglia-monitor start-up time gmetad there is a conflict, you can restart it.
Reference article
Ganglia Architecture and Features
Ubuntu10.04 Ganglia installation configuration
Ganglia monitoring and Hbase Hadoop cluster performance (installation configuration)
If after the successful implementation of the unicast deployment, and run only gmetad, and then come back to add Kazakhstan. For more details, refer to the article you can see I have given :)
---------------------
作者：YUKI小糖
来源：CNBLOGS
原文：https://www.cnblogs.com/yuki-lau/p/3201110.html
版权声明：本文为作者原创文章，转载请附上博文链接！