继 (Ambari 操作 (Ambari Operations) 之四 : https://blog.csdn.net/devalone/article/details/80826036 )
9. 使用 Ambari 核心服务 (Using Ambari Core Services)
Ambari 核心服务可用于监控，分析，以及搜索集群主机的操作状态。
9.1 理解 Ambari 度量器 (Understanding Ambari Metrics)
-----------------------------------------------------------------------------------------------------------------------------------------
Ambari Metrics System (AMS) 在 Ambari 管理的集群上收集，聚集，并服务于 Hadoop 和系统度量
9.1.1 AMS 体系结构 (AMS Architecture)
-----------------------------------------------------------------------------------------------------------------------------------------
AMS 有四个组件：Metrics Monitors, Hadoop Sinks, Metrics Collector, 以及 Grafana.
• Metrics Monitors ：在集群的每部主机上收集系统级别的度量并发布到 Metrics Collector 上
• Hadoop Sinks ：插入到 Hadoop 组件中用于发布 Hadoop 度量到 Metrics Collector 上
• Metrics Collector ：是一个运行在集群上特定主机中的 daemon 并从注册的发布者接收数据，Monitors 和 Sinks
• Grafana ：是一个运行在集群上特定主机中的 daemon，并为在 Metrics Collector 中收集到的 metrics 的可视化提供预构建表盘
9.1.2 使用 Grafana (Using Grafana)
-----------------------------------------------------------------------------------------------------------------------------------------
Ambari Metrics System 包括 Grafana 用于为高级可视化集群度量提供预构建表盘。
9.1.2.1 访问 Grafana (Accessing Grafana)
-----------------------------------------------------------------------------------------------------------------------------------------
① Ambari Web 中，浏览到 Services > Ambari Metrics > Summary
② 选择 Quick Links 然后选取 Grafana
一个只读版本的 Grafana 页面在浏览器的一个新 tab 页面打开
9.1.2.2 查看 Grafana 表盘(Viewing Grafana Dashboards)
-----------------------------------------------------------------------------------------------------------------------------------------
在 Grafana 主页上，Dashboards 提供了一个 AMS 链接列表，Ambari server, Druid and HBase metrics.
查看包含在列表中的特定 metric:
① 在 Grafana 中，浏览到 Dashboards
② 单击 Dashboards 名称
③ 查看更多表盘，单击 Home 列表
④ 滚动查看这个列表
例如，System - Servers
9.1.2.3 在 Grafana 表盘上查看选择的 Metrics (Viewing Selected Metrics on Grafana Dashboards)
-----------------------------------------------------------------------------------------------------------------------------------------
在表盘上，展开一个或多个行以查看详细的度量。 例如：
在 System - Servers 表盘上，单击行名称，例如单击 System Load Average - 1 Minute
这个行展开以显示一个图表显示度量信息。
9.1.2.4 查看选定主机的 Metrics (Viewing Metrics for Selected Hosts)
-----------------------------------------------------------------------------------------------------------------------------------------
默认情况下，Grafana 显示集群上所有主机 metric. 通过从 Hosts 菜单上选择，可以限制显示一个或几个主机的 metric
① 展开 Hosts
② 选择一个或多个主机名
9.1.3 Grafana 表盘参考 (Grafana Dashboards Reference)
-----------------------------------------------------------------------------------------------------------------------------------------
Ambari Metrics System 包含的 Grafana 为集群 metrics 的高级可视化带有预构建的表盘。
• AMS HBase Dashboards
• Ambari Dashboards
• HDFS Dashboards
• YARN Dashboards
• Hive Dashboards
• Hive LLAP Dashboards
• HBase Dashboards
• Kafka Dashboards
• Storm Dashboards
• System Dashboards
• NiFi Dashboard
9.1.3.1 AMS HBase 表盘 (AMS HBase Dashboards)
-----------------------------------------------------------------------------------------------------------------------------------------
AMS HBase 指的是由 Ambari Metrics Service 独立管理的 HBase 实例。它与集群的 HBase 服务没有任何连接。AMS HBase 表盘跟踪与常规 HBase 表盘
相同的度量，只是 AMS 自身的实例。
如下的 Grafana 表盘适用于 AMS HBase
• AMS HBase - Home
• AMS HBase - RegionServers
• AMS HBase - Misc
9.1.3.1.1 AMS HBase 表盘 (AMS HBase - Home)
-----------------------------------------------------------------------------------------------------------------------------------------
AMS HBase - Home 表盘显示 HBase 集群基本的统计信息，这些仪表提供了 HBase 集群整体状态的观察。
REGIONSERVERS / REGIONS
-------------------------------------------------------------------------------------------------------------------------------------
Num RegionServers : Total number of RegionServers in the cluster.
Num Dead RegionServers : Total number of RegionServers that are dead in the cluster.
Num Regions : Total number of regions in the cluster.
Avg Num Regions per RegionServer : Average number of regions per RegionServer.
NUM REGIONS/STORES
-------------------------------------------------------------------------------------------------------------------------------------
Num Regions /Stores - Total : Total number of regions and stores (column families) in the cluster.
Store File Size /Count - Total : Total data file size and number of store files.
NUM REQUESTS
-------------------------------------------------------------------------------------------------------------------------------------
Num Requests - Total : Total number of requests (read, write and RPCs) in the cluster.
Num Request - Breakdown - Total : Total number of get,put,mutate,etc requests in the cluster.
REGIONSERVER MEMORY
-------------------------------------------------------------------------------------------------------------------------------------
RegionServer Memory - Average : Average used, max or committed on-heap and offheap memory for RegionServers.
RegionServer Offheap Memory - Average : Average used, free or committed on-heap and offheap memory for RegionServers.
MEMORY - MEMSTORE BLOCKCACHE
-------------------------------------------------------------------------------------------------------------------------------------
Memstore - BlockCache - Average : Average blockcache and memstore sizes for RegionServers.
Num Blocks in BlockCache - Total : Total number of (hfile) blocks in the blockcaches across all RegionServers.
BLOCKCACHE
-------------------------------------------------------------------------------------------------------------------------------------
BlockCache Hit/Miss/s Total : Total number of blockcache hits misses and evictions across all RegionServers.
BlockCache Hit Percent - Average : Average blockcache hit percentage across all RegionServers.
OPERATION LATENCIES - GET/MUTATE
-------------------------------------------------------------------------------------------------------------------------------------
Get Latencies - Average : Average min, median, max, 75th, 95th, 99th percentile latencies for Get operation across
all RegionServers.
Mutate Latencies - Average : Average min, median, max, 75th, 95th, 99th percentile latencies for Mutate operation across
all RegionServers.
OPERATION LATENCIES - DELETE/INCREMENT
-------------------------------------------------------------------------------------------------------------------------------------
Delete Latencies - Average : Average min, median, max, 75th, 95th, 99th percentile latencies for Delete operation across
all RegionServers.
Increment Latencies - Average : Average min, median, max, 75th, 95th, 99th percentile latencies for Increment operation across
all RegionServers.
OPERATION LATENCIES - APPEND/REPLAY
-------------------------------------------------------------------------------------------------------------------------------------
Append Latencies - Average : Average min, median, max, 75th, 95th, 99th percentile latencies for Append operation across
all RegionServers.
Replay Latencies - Average : Average min, median, max, 75th, 95th, 99th percentile latencies for Replay operation across
all RegionServers.
REGIONSERVER RPC
-------------------------------------------------------------------------------------------------------------------------------------
RegionServer RPC - Average : Average number of RPCs, active handler threads and open connections across all RegionServers.
RegionServer RPC Queues - Average : Average number of calls in different RPC scheduling queues and the size of all requests in the
RPC queue across all RegionServers.
REGIONSERVER RPC
-------------------------------------------------------------------------------------------------------------------------------------
RegionServer RPC Throughput - Average : Average sent and received bytes from the RPC across all RegionServers.
9.1.3.1.2 AMS HBase 表盘 (AMS HBase - RegionServers)
-----------------------------------------------------------------------------------------------------------------------------------------
AMS HBase - RegionServers 仪表显示在监控的 HBase 集群中的 RegionServers 度量，包括一些性能相关的数据。这些仪表帮助查看基本 I/O 数据，以及
RegionServers 中进行负载比较。
......
9.1.3.1.3 AMS HBase 表盘 (AMS HBase - Misc)
-----------------------------------------------------------------------------------------------------------------------------------------
AMS HBase - Misc 仪表显示 HBase 集群相关的多方面的度量信息。可以在某些任务中利用这些度量信息，例如，调试身份认证，授权问题，以及由
RegionServers 产生的异常问题等。
......
9.1.3.2 Ambari 表盘 (Ambari Dashboards)
-----------------------------------------------------------------------------------------------------------------------------------------
下面的仪表可用于 Ambari ：
• Ambari Server Database
• Ambari Server JVM
• Ambari Server Top N
9.1.3.2.1 Ambari server 数据库 (Ambari Server Database)
-----------------------------------------------------------------------------------------------------------------------------------------
显示 Ambari server 数据库的操作状态。
TOTAL READ ALL QUERY
-------------------------------------------------------------------------------------------------------------------------------------
Total Read All Query Counter (Rate) : Total ReadAllQuery operations performed.
Total Read All Query Timer (Rate) : Total time spent on ReadAllQuery.
TOTAL CACHE HITS & MISSES
-------------------------------------------------------------------------------------------------------------------------------------
Total Cache Hits (Rate) : Total cache hits on Ambari Server with respect to EclipseLink cache.
Total Cache Misses (Rate) : Total cache misses on Ambari Server with respect to EclipseLink cache.
QUERY
-------------------------------------------------------------------------------------------------------------------------------------
Query Stages Timings : Average time spent on every query sub stage by Ambari Server
Query Types Avg. Timings : Average time spent on every query type by Ambari Server.
HOST ROLE COMMAND ENTITY
-------------------------------------------------------------------------------------------------------------------------------------
Counter.ReadAllQuery.HostRoleCommandEntity (Rate) : Rate (num operations per second) in which ReadAllQuery operation on
HostRoleCommandEntity is performed.
Timer.ReadAllQuery.HostRoleCommandEntity (Rate) : Rate in which ReadAllQuery operation on HostRoleCommandEntity is performed.
ReadAllQuery.HostRoleCommandEntity : Average time taken for a ReadAllQuery operation on HostRoleCommandEntity (
Timer / Counter).
9.1.3.2.2 Ambari server JVM (Ambari Server JVM)
-----------------------------------------------------------------------------------------------------------------------------------------
JVM - MEMORY PRESSURE
-------------------------------------------------------------------------------------------------------------------------------------
Heap Usage : Used, max or committed on-heap memory for Ambari Server.
Off-Heap Usage : Used, max or committed off-heap memory for Ambari Server.
JVM GC COUNT
-------------------------------------------------------------------------------------------------------------------------------------
GC Count Par new /s : Number of Java ParNew (YoungGen) Garbage Collections per second.
GC Time Par new /s : Total time spend in Java ParNew(YoungGen) Garbage Collections per second.
GC Count CMS /s : Number of Java Garbage Collections per second.
GC Time Par CMS /s : Total time spend in Java CMS Garbage Collections per second.
JVM THREAD COUNT
-------------------------------------------------------------------------------------------------------------------------------------
Thread Count : Number of active, daemon, deadlock, blocked and runnable threads.
9.1.3.2.3 Ambari Server Top N (Ambari Server Top N)
-----------------------------------------------------------------------------------------------------------------------------------------
READ ALL QUERY
-------------------------------------------------------------------------------------------------------------------------------------
Top ReadAllQuery Counters : Top N Ambari Server entities by number of ReadAllQuery operations performed.
Top ReadAllQuery Timers : Top N Ambari Server entities by time spent on ReadAllQuery operations.
Cache Misses
-------------------------------------------------------------------------------------------------------------------------------------
Cache Misses : Top N Ambari Server entities by number of Cache Misses.
9.1.3.3 Druid Dashboards
-----------------------------------------------------------------------------------------------------------------------------------------
......
9.1.3.4 HDFS Dashboards
-----------------------------------------------------------------------------------------------------------------------------------------
如下 Grafana 仪表适用于 Hadoop Distributed File System (HDFS) 组件
• HDFS - Home
• HDFS - NameNodes
• HDFS - DataNodes
• HDFS - Top-N
• HDFS - Users
......
9.1.3.5 YARN Dashboards
-----------------------------------------------------------------------------------------------------------------------------------------
如下 Grafana 仪表适用于 YARN:
• YARN - Home
• YARN - Applications
• YARN - MR JobHistory Server
• YARN - MR JobHistory Server
• YARN - NodeManagers
• YARN - Queues
• YARN - ResourceManager
......
9.1.3.6 Hive Dashboards
-----------------------------------------------------------------------------------------------------------------------------------------
如下 Grafana 仪表适用于 Hive:
• Hive - Home
• Hive - HiveMetaStore
• Hive - HiveServer2
......
9.1.3.7 Hive LLAP Dashboards
-----------------------------------------------------------------------------------------------------------------------------------------
如下 Grafana 仪表适用于 Hive LLAP:
• Hive LLAP - Heatmap
• Hive LLAP - Overview
• Hive LLAP - Daemon
......
9.1.3.8 HBase Dashboards
-----------------------------------------------------------------------------------------------------------------------------------------
如下 Grafana 仪表适用于 Hive HBase:
• HBase - Home
• HBase - RegionServers
• HBase - Misc
• HBase - Tables
• HBase - Users
......
9.1.3.9 Kafka Dashboards
-----------------------------------------------------------------------------------------------------------------------------------------
如下 Grafana 仪表适用于 Hive Kafka:
• Kafka - Home
• Kafka - Hosts
• Kafka - Topics
......
9.1.3.10 Storm Dashboards
-----------------------------------------------------------------------------------------------------------------------------------------
如下 Grafana 仪表适用于 Hive Storm:
• Storm - Home
• Storm - Topology
• Storm - Components
......
9.1.3.11 System Dashboards
-----------------------------------------------------------------------------------------------------------------------------------------
如下 Grafana 仪表适用于 Hive System:
• System - Home
• System - Servers
......
9.1.3.12 NiFi Dashboards
-----------------------------------------------------------------------------------------------------------------------------------------
如下 Grafana 仪表适用于 Hive NiFi:
• NiFi-Home
......
9.1.4 AMS 性能调优 (AMS Performance Tuning)
-----------------------------------------------------------------------------------------------------------------------------------------
要在环境中设置 Ambari Metrics System, 查看并自定义如下 Metrics Collector 配置选项：
• Customizing the Metrics Collector Mode
• Customizing TTL Settings
• Customizing Memory Settings
• Customizing Cluster-Environment-Specific Settings
• Moving the Metrics Collector
• (Optional) Enabling Individual Region, Table, and User Metrics for HBase
9.1.4.1 自定义 Metrics Collector 模式 (Customizing the Metrics Collector Mode)
-----------------------------------------------------------------------------------------------------------------------------------------
Metrics Collector 利用 Hadoop 技术构建，例如 Apache HBase, Apache Phoenix, and Apache Traffic Server (ATS). Collector 可存储度量数据到本地
文件系统，成为 embedded mode, 或使用外部 HDFS, 成为 distributed mode. 默认情况下，Collector 运行于嵌入模式。在嵌入模式下，Collector 获取
数据并把度量数据写入到运行 Collector 主机的本地文件系统。
重要提示：
运行嵌入模式时，应该确认 hbase.rootdir 和 hbase.tmp.dir 有足够的大小容纳数据，并且负载要轻。目录配置在
Ambari Metrics > Configs > Advanced > ams-hbasesite
所在分区要有足够的大小，并且负载不要繁重，例如：
file:///grid/0/var/lib/ambari-metrics-collector/hbase.
也要确认 TTL 设置合适。
Collector 配置为分布式模式，它将度量数据写入到 HDFS, 并且组件运行于分布式进程上，有助于管理 CPU 和内存消耗。
切换 Metrics Collector 从嵌入模式到分布式模式：
① 在 Ambari Web 中, 浏览到 Services > Ambari Metrics > Configs
② 修改列于如下表格中的属性值：
+-----------------------+-------------------------------------------+-------------------------------+-------------------------------+
| Configuration Section | Property | Description | Value |
+-----------------------+-------------------------------------------+-------------------------------+-------------------------------+
| General |Metrics Service operation mode | Designates whether to run in |distributed |
| |(timeline.metrics.service.operation.mode) | distributed or embedded mode. | |
+-----------------------+-------------------------------------------+-------------------------------+-------------------------------+
| Advanced amshbase-site| hbase.cluster.distributed | Indicates AMS will run in |true |
| | | distributed mode. | |
+-----------------------+-------------------------------------------+-------------------------------+-------------------------------+
| Advanced amshbase-site| hbase.rootdir 1 |The HDFS directory location |hdfs://$NAMENODE_FQDN:8020/ |
| | |where metrics will be stored |apps/ams/metrics |
+-----------------------+-------------------------------------------+-------------------------------+-------------------------------+
③ Ambari Web > Hosts > Components 重启 Metrics Collector
如果集群配置为 NameNode 高可用性，设置 hbase.rootdir 值为 HDFS 名称服务替代 NameNode 主机名称：
hdfs://hdfsnameservice/apps/ams/metrics
可选地，可以在切换到分布式模式之前，将本地存储的现有数据迁移到 HDFS。
步骤：
① 为 ams 用户创建目录
su - hdfs -c 'hdfs dfs -mkdir -p /apps/ams/metrics'
② 停止 Metrics Collector
③ 将度量数据从 AMS 本地目录复制到 HDFS 目录。这是 hbase.rootdir 值，如：
su - hdfs -c 'hdfs dfs -copyFromLocal /var/lib/ambari-metrics-collector/hbase/* /apps/ams/metrics'
su - hdfs -c 'hdfs dfs -chown -R ams:hadoop /apps/ams/metrics'
④ 切换到分布式模式
⑤ 重启 Metrics Collector
9.1.4.2 自定义 TTL 设置 (Customizing TTL Settings)
-----------------------------------------------------------------------------------------------------------------------------------------
AMS 可以为聚集的度量设置 Time To Live (TTL), 通过 Ambari Metrics > Configs > Advanced ams-siteEach 自解释的属性名，以及控制度量值在其被
清除之前保持的时间数量(单位，秒)。TTL 设置的时间值单位为秒。
例如，假设正在运行一个单节点的沙箱(a single-node sandbox), 并且要确保不保存超过七天的数据，以降低磁盘空间消耗。可以设置任何以 .ttl 结尾的
属性值为 604800(七天的秒数)。
可能要为 timeline.metrics.cluster.aggregator.daily.ttl 属性设置这个值，控制每日聚集 TTL, 默认设置为 2 年。
另外两个消耗大量磁盘空间的属性为：
• timeline.metrics.cluster.aggregator.minute.ttl : 控制分钟级聚集度量 TTL
• timeline.metrics.host.aggregator.ttl : 控制基于主机精度的度量 TTL
9.1.4.3 自定义 Memory 设置 (Customizing Memory Settings)
-----------------------------------------------------------------------------------------------------------------------------------------
因为 AMS 使用多个组件(例如 Apache HBase 和 Apache Phoenix) 来存储度量和查询，因此多个可调控的属性可用于调优内存使用：
+---------------------------+-------------------------------+-------------------------------------------------------------------+
| 配置 | 属性 | 描述 |
+---------------------------+-------------------------------+-------------------------------------------------------------------+
| Advanced ams-env | metrics_collector_heapsize | Heap size configuration for the Collector. |
+---------------------------+-------------------------------+-------------------------------------------------------------------+
| Advanced ams-hbase-env | hbase_regionserver_heapsize | Heap size configuration for the single AMS HBase Region Server. |
+---------------------------+-------------------------------+-------------------------------------------------------------------+
| Advanced ams-hbase-env | hbase_master_heapsize | Heap size configuration for the single AMS HBase Master. |
+---------------------------+-------------------------------+-------------------------------------------------------------------+
| Advanced ams-hbase-env | regionserver_xmn_size | Maximum value for the young generation heap size for the single |
| | | AMS HBase RegionServer. |
+---------------------------+-------------------------------+-------------------------------------------------------------------+
| Advanced ams-hbase-env | hbase_master_xmn_size | Maximum value for the young generation heap size for the single |
| | | AMS HBase Master. |
+---------------------------+-------------------------------+-------------------------------------------------------------------+
9.1.4.4 自定义集群环境特定的设置 (Customizing Cluster-Environment-Specific Settings)
-----------------------------------------------------------------------------------------------------------------------------------------
对 AMS 的 Metrics Collector 模式，TTL 设置，内存设置，以及磁盘空间要求取决于集群的节点数量。下面表格列出对每种配置的建议和调优原则：
+---------------+-----------+-----------+---------------+---------------+-----------------------------------+
|集群环境 | 主机数量 | 磁盘空间 | Collector | TTL | Memory 设置 |
| | | | 模式 | | |
+---------------+-----------+-----------+---------------+---------------+-----------------------------------+
|Single-Node | 1 | 2GB | embedded |Reduce TTLs |metrics_collector_heap_size=1024 |
|Sandbox | | | |to 7 Days |hbase_regionserver_heapsize=512 |
| | | | | |hbase_master_heapsize=512 |
| | | | | |hbase_master_xmn_size=128 |
+---------------+-----------+-----------+---------------+---------------+-----------------------------------+
|PoC | 1-5 | 5GB | embedded |Reduce TTLs |metrics_collector_heap_size=1024 |
| | | | |to 30 Days |hbase_regionserver_heapsize=512 |
| | | | | |hbase_master_heapsize=512 |
| | | | | |hbase_master_xmn_size=128 |
+---------------+-----------+-----------+---------------+---------------+-----------------------------------+
|Pre-Production | 5-20 | 20GB | embedded |Reduce TTLs |metrics_collector_heap_size=1024 |
| | | | |to 3 Months |hbase_regionserver_heapsize=1024 |
| | | | | |hbase_master_heapsize=512 |
| | | | | |hbase_master_xmn_size=128 |
+---------------+-----------+-----------+---------------+---------------+-----------------------------------+
|Production | 20-50 | 50GB | embedded | n.a. |metrics_collector_heap_size=1024 |
| | | | | |hbase_regionserver_heapsize=1024 |
| | | | | |hbase_master_heapsize=512 |
| | | | | |hbase_master_xmn_size=128 |
+---------------+-----------+-----------+---------------+---------------+-----------------------------------+
|Production | 50-200 | 100GB | embedded | n.a. |metrics_collector_heap_size=2048 |
| | | | | |hbase_regionserver_heapsize=2048 |
| | | | | |hbase_master_heapsize=2048 |
| | | | | |hbase_master_xmn_size=256 |
+---------------+-----------+-----------+---------------+---------------+-----------------------------------+
|Production | 200-400 | 200GB | embedded | n.a. |metrics_collector_heap_size=2048 |
| | | | | |hbase_regionserver_heapsize=2048 |
| | | | | |hbase_master_heapsize=2048 |
| | | | | |hbase_master_xmn_size=512 |
+---------------+-----------+-----------+---------------+---------------+-----------------------------------+
|Production | 400-800 | 200GB | distributed | n.a. |metrics_collector_heap_size=8192 |
| | | | | |hbase_regionserver_heapsize=122288 |
| | | | | |hbase_master_heapsize=1024 |
| | | | | |hbase_master_xmn_size=1024 |
| | | | | |regionserver_xmn_size=1024 |
+---------------+-----------+-----------+---------------+---------------+-----------------------------------+
|Production | 800+ | 500GB | distributed | n.a. |metrics_collector_heap_size=12288 |
| | | | | |hbase_regionserver_heapsize=16384 |
| | | | | |hbase_master_heapsize=16384 |
| | | | | |hbase_master_xmn_size=2048 |
| | | | | |regionserver_xmn_size=1024 |
+---------------+-----------+-----------+---------------+---------------+-----------------------------------+
9.1.4.5 移动 Metrics Collector (Moving the Metrics Collector)
-----------------------------------------------------------------------------------------------------------------------------------------
使用如下过程将 Ambari Metrics Collector 移动到一个新的主机上：
① 在 Ambari Web , 停止 Ambari Metrics 服务
② 执行下列 API 调用来删除当前的 Metric Collector 组件：
curl -u admin:admin -H "X-Requested-By:ambari" - i -X \
DELETE http://ambari.server:8080/api/v1/clusters/cluster.name/hosts/metrics.collector.hostname/host_components/METRICS_COLLECTOR
③ 执行下列 API 调用在新主机上添加 Metric Collector：
curl -u admin:admin -H "X-Requested-By:ambari" - i -X \
POST http://ambari.server:8080/api/v1/clusters/cluster.name/hosts/metrics.collector.hostname/host_components/METRICS_COLLECTOR
④ 在 Ambari Web, 导航到安装了新 Metrics Collector 的主机上并单击 Install the Metrics Collector
⑤ 在 Ambari Web, 启动 Ambari Metrics 服务
9.1.4.6 (可选)为 HBase 启动单独的 Region, Table, and User Metrics (Enabling Individual Region, Table, and User Metrics for HBase)
-----------------------------------------------------------------------------------------------------------------------------------------
不像 HBase RegionServer metrics, Ambari 默认禁用 per region, per table, and per user metrics, 因为这些 metrics 非常多因而会导致性能问题。
如果要 Ambari 收集这些 metrics, 可以重新启用它们。然而，要首先测试这个选项并确认 AMS 性能可接受。
① 在 Ambari Server 上，浏览到如下位置：
/var/lib/ambari-server/resources/common-services/HBASE/0.96.0.2.0/package/templates
② 编辑如下模板文件：
hadoop-metrics2-hbase.properties-GANGLIA-MASTER.j2
hadoop-metrics2-hbase.properties-GANGLIA-RS.j2
③ 注释掉或者删除下面的行
*.source.filter.class=org.apache.hadoop.metrics2.filter.RegexFilter
hbase.*.source.filter.exclude=.*(Regions|Users|Tables).*
④ 保存模板文件并重启 Ambari Server 使修改生效。
重要提示：
如果 Ambari 升级到一个新的版本，必须要重新对模板文件进行上述修改
9.1.5 AMS 高可用性 (AMS High Availability)
-----------------------------------------------------------------------------------------------------------------------------------------
Ambari 默认安装 Ambari Metrics System (AMS) 到集群中一个 Metrics Collector 组件。Collector 是运行在集群的一个特定主机上的守护进程，从注册
的发布者接收数据，Monitors 和 Sinks .
取决于需要，可以要求 AMS 有两个 Collector 来形成高可用性情形。
前提：
必须部署 AMS 为分布式模式(not embedded)
步骤：
① 在 Ambari Web 中，浏览到打算安装另一个收集器的主机
② 在 Hosts 页面，选取 +Add
③ 从列表上选取 Metrics Collector
Ambari 安装新的 Metrics Collector 并配置 HA 的 Ambari Metrics
新安装的收集器处于 “stopped” 状态
④ 在 Ambari Web 中，启动新的 Collector 组件
Note：
如果在安装第二个 Collector 到集群中之前没有将 AMS 切换为分布式模式，第二个收集器会被安装，但不会启动。
9.1.6 AMS 安全性 (AMS Security)
-----------------------------------------------------------------------------------------------------------------------------------------
9.1.6.1 修改 Grafana 管理员密码 (Changing the Grafana Admin Password)
-----------------------------------------------------------------------------------------------------------------------------------------
如果需要在初始安装 Ambari 之后修改 Grafana 管理员密码，可以直接在 Grafana 中修改密码，然后在 Ambari Metrics 配置中做同样的修改。
(1) 在 Ambari Web 中, 浏览到 Services > Ambari Metrics, 选择 Quick Links, 然后选取 Grafana
Grafana UI 以只读方式打开
(2) 单击 Sign In
(3) 以管理员登录，使用未更改的密码 admin/admin
(4) 单击 admin 标签以查看管理员信息，单击 Change password
(5) 输入未改变的密码，输入并确认新密码，然后单击 Change password 按钮
(6) 回到 Ambari Web > Services > Ambari Metrics, 然后浏览 Configs tab
(7) 在 General 部分，使用新密码更新并确认 Grafana Admin Password
(8) 保存配置并重启服务，如果提示。
9.1.6.2 为 AMS 设置 HTTPS (Set Up HTTPS for AMS)
-----------------------------------------------------------------------------------------------------------------------------------------
如果要限制访问 AMS 通过 HTTPS 连接，必须提供一个证书。起初测试的时候可以使用自签名的证书，但不适用于生产环境。在获得了一个证书之后，必须
运行特定的安装命令(setup command)。
步骤：
(1) 创建自己的 CA 证书(CA certificate)
openssl req -new -x509 -keyout ca.key -out ca.crt -days 365
(2) 导入 CA 证书到信任站 (truststore)
# keytool -keystore /<path>/truststore.jks -alias CARoot -import -file ca.crt -storepass bigdata
(3) 检查 truststore
# keytool -keystore /<path>/truststore.jks -list
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 2 entries
caroot, Feb 22, 2016, trustedCertEntry,
Certificate fingerprint (SHA1):
AD:EE:A5:BC:A8:FA:61:2F:4D:B3:53:3D:29:23:58:AB:2E:B1:82:AF
(4) 为 AMS Collector 生成证书并存储私钥到 keystore.
# keytool -genkey -alias c6401.ambari.apache.org -keyalg RSA -keysize 1024
-dname "CN=c6401.ambari.apache.org,OU=IT,O=Apache,L=US,ST=US,C=US" -keypass
bigdata -keystore /<path>/keystore.jks -storepass bigdata
(5) 为 AMS collector 证书创建证书请求(certificate request)
keytool -keystore /<path>/keystore.jks -alias c6401.ambari.apache.org -certreq -file c6401.ambari.apache.org.csr -storepass bigdata
(6) 利用 CA 证书为证书请求签名
openssl x509 -req -CA ca.crt -CAkey ca.key -in c6401.ambari.apache.org.csr
-out c6401.ambari.apache.org_signed.crt -days 365 -CAcreateserial -passin
pass:bigdata
(7) 把 CA 证书导入到 keystore.
keytool -keystore /<path>/keystore.jks -alias CARoot -import -file ca.crt -storepass bigdata
(8) 导入签名的证书到 keystore.
keytool -keystore /<path>/keystore.jks -alias c6401.ambari.apache.org -
import -file c6401.ambari.apache.org_signed.crt -storepass bigdata
(9) 检查 keystore.
caroot2, Feb 22, 2016, trustedCertEntry,
Certificate fingerprint (SHA1):
7C:B7:0C:27:8E:0D:31:E7:BE:F8:BE:A1:A4:1E:81:22:FC:E5:37:D7
[root@c6401 tmp]# keytool -keystore /tmp/keystore.jks -list
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 2 entries
caroot, Feb 22, 2016, trustedCertEntry,
Certificate fingerprint (SHA1):
AD:EE:A5:BC:A8:FA:61:2F:4D:B3:53:3D:29:23:58:AB:2E:B1:82:AF
c6401.ambari.apache.org, Feb 22, 2016, PrivateKeyEntry,
Certificate fingerprint (SHA1):
A2:F9:BE:56:7A:7A:8B:4C:5E:A6:63:60:B7:70:50:43:34:14:EE:AF
(10) 复制 /<path>/truststore.jks 文件到所有节点的 /<path>/truststore.jks 并设置合适的访问权限
(11) 复制 /<path>/keystore.jks 文件到 AMS 收集器节点只到 /<path>/keystore.jks 路径，并设置合适的访问权限。建议设置 ams 用户为文件 owner, 并设置
访问权限为 400
(12) 在 Ambari Web 中，更新 AMS 配置，setams-site/timeline.metrics.service.http.policy=HTTPS_ONLY
•ams-ssl-server/ssl.server.keystore.keypassword=bigdata
•ams-ssl-server/ssl.server.keystore.location=/<path>/keystore.jks
•ams-ssl-server/ssl.server.keystore.password=bigdata
•ams-ssl-server/ssl.server.keystore.type=jks
•ams-ssl-server/ssl.server.truststore.location=/<path>/truststore.jks
•ams-ssl-server/ssl.server.truststore.password=bigdata
•ams-ssl-server/ssl.server.truststore.reload.interval=10000
•ams-ssl-server/ssl.server.truststore.type=jks
•ams-ssl-client/ssl.client.truststore.location=/<path>/truststore.jks
•ams-ssl-client/ssl.client.truststore.password=bigdata
•ams-ssl-client/ssl.client.truststore.type=jks
•ssl.client.truststore.alias=<Alias used to create certificate for AMS. (Default is hostname)>
(13) 重启服务
(14) 配置 Ambari server 使用 truststore
# ambari-server setup-security
Using python /usr/bin/python
Security setup options...
===========================================================================
Choose one of the following options:
[1] Enable HTTPS for Ambari server.
[2] Encrypt passwords stored in ambari.properties file.
[3] Setup Ambari kerberos JAAS configuration.
[4] Setup truststore.
[5] Import certificate to truststore.
===========================================================================
Enter choice, (1-5): 4
Do you want to configure a truststore [y/n] (y)?
TrustStore type [jks/jceks/pkcs12] (jks):jks
Path to TrustStore file :/<path>/keystore.jks
Password for TrustStore:
Re-enter password:
Ambari Server 'setup-security' completed successfully.
(15) 配置 ambari server 在请求 AMS Collector 时使用 https 替代 http：
# echo "server.timeline.metrics.https.enabled=true" >> /etc/ambari-server/conf/ambari.properties
(16) 重启 ambari server
9.1.6.3 为 Grafana 设置 HTTPS (Set Up HTTPS for Grafana)
-----------------------------------------------------------------------------------------------------------------------------------------
如果要限制访问 Grafana 通过 HTTPS 连接，必须提供一个证书。起初测试的时候可以使用自签名的证书，但不适用于生产环境。在获得了一个证书之后，
必须运行特定的安装命令(setup command)。
步骤：
(1) 登录到 Grafana 主机上
(2) 浏览到 Grafana 配置目录
cd /etc/ambari-metrics-grafana/conf/
(3) 定位到证书
如果要创建一个临时的自签名证书，可以运行：
openssl genrsa -out ams-grafana.key 2048
openssl req -new -key ams-grafana.key -out ams-grafana.csr
openssl x509 -req -days 365 -in ams-grafana.csr -signkey ams-grafana.key -
out ams-grafana.crt
(4) 设置证书和秘钥文件的所有者和权限，让 Grafana 可以访问
chown ams:hadoop ams-grafana.crt
chown ams:hadoop ams-grafana.key
chmod 400 ams-grafana.crt
chmod 400 ams-grafana.key
对于 non-root Ambari user, 使用：
chmod 444 ams-grafana.crt
让 agent user 可以读取文件
(5) 在 Ambari Web, 浏览到 Services > Ambari Metrics > Configs
(6) 在 Advanced ams-grafana-ini 部分更新如下属性：
protocol https
cert_file /etc/ambari-metrics-grafana/conf/ams-grafana.crt
cert-Key /etc/ambari-metrics-grafana/conf/ams-grafana.key
(7) 保存配置并重启服务，如果提示。
参考：
Ambari 操作 (Ambari Operations) 之一
Ambari 操作 (Ambari Operations) 之二
Ambari 操作 (Ambari Operations) 之三
Ambari 操作 (Ambari Operations) 之四
Ambari 操作 (Ambari Operations) 之五
扫描二维码关注公众号，回复：
1926303 查看本文章
Ambari 操作 (Ambari Operations) 之六