Flume- data flow monitoring Ganglia

Ganglia from the gmond, gmetad and gweb three parts.

gmond (Ganglia Monitoring Daemon) is a lightweight service installed on each host node need to collect indicator data. Use gmond, you can easily collect a lot of data system metrics, such as CPU, memory, disk, and network data and other active processes.

gmetad (Ganglia Meta Daemon) to integrate all the information and store it to disk service at RRD format.

gweb (Ganglia Web) Ganglia visualization tools, gweb using a browser to display the stored data gmetad PHP front end. A variety of different metrics data collected under the operational status graphically show the cluster in the Web interface.

 

First, install

# Replace Ali yum source
 mv / etc / yum .repos.d / CentOS-Base.repo / etc / yum .repos.d / CentOS- Base.repo.backup 
curl -o / etc / yum .repos.d / CentOS- HTTP Base.repo: // mirrors.aliyun.com/repo/Centos-7.repo 
sed -i -e ' /mirrors.cloud.aliyuncs.com/d ' -e ' /mirrors.aliyuncs.com/d ' / etc / yum .repos.d / CentOS- Base.repo 

# httpd service and installation PHP 
the sudo  yum -Y the install httpd PHP 

# install additional dependencies 
the sudo  yum -Yinstall rrdtool perl-rrdtool rrdtool-devel
sudo yum -y install apr-devel

# ganglia 在 epel 源中
mv /etc/yum.repos.d/epel.repo /etc/yum.repos.d/epel.repo.backup
mv /etc/yum.repos.d/epel-testing.repo /etc/yum.repos.d/epel-testing.repo.backup
curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo

# 安装 ganglia
sudo yum -y install ganglia-gmetad
sudo yum -y install ganglia-web
sudo yum -y install ganglia-gmond

 

Second, the configuration

1.httpd Configuration

https://blog.csdn.net/mxl00z/article/details/83041266

sudo vim /etc/httpd/conf.d/ganglia.conf

ganglia.conf

#
# Ganglia monitoring system php web frontend
#

Alias /ganglia /usr/share/ganglia

<Location /ganglia>
  # Options Indexes FollowSymLinks
  # AllowOverride None
  Require all granted
  # Order deny,allow
  # Deny from all
  # Allow from all
  # Require local
  # Require ip 10.1.2.3
  # Require host example.org
</Location>

2.gmetad Configuration

sudo vim /etc/ganglia/gmetad.conf

gmetad.conf

。。。。。。
# A list of machines which service the data source follows, in the 
# format ip:port, or name:port. If a port is not specified then 8649
# (the default gmond port) is assumed.
# default: There is no default value
#
# data_source "my cluster" 10 localhost  my.machine.edu:8649  1.2.3.5:8655
# data_source "my grid" 50 1.3.4.7:8655 grid.org:8651 grid-backup.org:8651
# data_source "another source" 1.3.4.7:8655  1.3.4.8

data_source "h136" 192.168.8.136

。。。。。。

3.gmond Configuration

sudo vim /etc/ganglia/gmond.conf

gmond.conf

。。。。。。
/*
 * The cluster attributes specified will be used as part of the <CLUSTER>
 * tag that will wrap all hosts collected by this instance.
 */
cluster {
  name = "h136"
  owner = "unspecified"
  latlong = "unspecified"
  url = "unspecified"
}

/* The host section describes attributes of the host, like the location */
host {
  location = "unspecified"
}

/* Feel free to specify as many udp_send_channels as you like.  Gmond
   used to only support having a single channel */
udp_send_channel {
  #bind_hostname = yes # Highly recommended, soon to be default.
                       # This option tells gmond to use a source address
                       # that resolves to the machine's hostname.  Without
                       # this, the metrics may appear to come from any
                       # interface and the DNS names associated with
                       # those IPs will be used to create the RRDs.
  # mcast_join = 239.2.11.71
  host = 192.168.8.136
  port = 8649
  ttl = 1
}

/* You can specify as many udp_recv_channels as you like as well. */
udp_recv_channel {
  # mcast_join = 239.2.11.71
  port = 8649
  bind = 192.168.8.136
  retry_bind = true
  # Size of the UDP buffer. If you are handling lots of metrics you really
  # should bump it up to e.g. 10MB or even higher.
  # buffer = 10485760
}

。。。。。。

4.selinux Configuration

setenforce 0

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

 

Third, start

systemctl restart httpd
systemctl restart gmetad
systemctl restart gmond

Browser to access: http: // IP / ganglia /

If there You do not have permission to access / ganglia / on this server, you can modify the / var / lib authority / ganglia directory.

sudo  chmod -R 0777 / var / lib / ganglia

Fourth, the operation Flume test monitoring

Modify flume-env.sh, adding JAVA_OPTS

vim /opt/apache-flume-1.9.0-bin/conf/flume-env.sh


JAVA_OPTS="-Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=192.168.8.136:8649 -Xms100m -Xmx200m"

Testing Task File

flume-netcat-logger.conf

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = 127.0.0.1
a1.sources.r1.port = 4444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

View Code

Start Flume task

cd /opt/apache-flume-1.9.0-bin

bin/flume-ng agent --conf conf/ --name a1 --conf-file /tmp/flume-job/flume-netcat-logger.conf -Dflume.root.logger==INFO,console -Dflume.monitoring.type=ganglia -Dflume.monitoring.hosts=192.168.8.136:8649

nc 127.0.0.1 4444

123

View Ganglia

illustration:

Field (chart name)	Field Meaning
EventPutAttemptCount	The total number of events written to try to source the channel
EventPutSuccessCount	The total number of events successfully written and submitted by channel
EventTakeAttemptCount	The total number of sink trying to pull events from channel
EventTakeSuccessCount	The total number of events sink successfully read
StartTime	channel start time (ms)
StopTime	channel stop time (ms)
ChannelSize	At present the total number of channel in the event of
ChannelFillPercentage	channel occupancy percentage
ChannelCapacity	channel capacity