【Ambari2.7.3源码分析】Agent组件状态收集器ComponentStatusExecutor

1、InitializerModule.py中

在InitializerModule.py中，初始化了众多属性后，进行线程的初始化
这些线程在agent端非常重要

  def init_threads(self):
    """
    Initialize thread objects
    """
    self.component_status_executor = ComponentStatusExecutor(self)
    self.action_queue = ActionQueue(self)
    self.alert_scheduler_handler = AlertSchedulerHandler(self)
    self.command_status_reporter = CommandStatusReporter(self)
    self.host_status_reporter = HostStatusReporter(self)
    self.alert_status_reporter = AlertStatusReporter(self)
    self.heartbeat_thread = HeartbeatThread.HeartbeatThread(self)

其中ComponentStatusExecutor(self)就是ambari-agent对组件状态执行器进行初始化
本文也主要讲此线程

2、ComponentStatusExecutor.py中

在ComponentStatusExecutor.py中
核心方法为下面的run()方法

  def run(self):
    """
    Run an endless loop which executes all status commands every 'status_commands_run_interval' seconds.
    """
    if self.status_commands_run_interval == 0:
      self.logger.warn("ComponentStatusExecutor is turned off. Some functionality might not work correctly.")
      return

    while not self.stop_event.is_set():
      try:
        # 为了删除已删除的信息需要这样子做
        self.clean_not_existing_clusters_info()
        cluster_reports = defaultdict(lambda:[])

        with self.reports_to_discard_lock:
          self.reports_to_discard = []

        for cluster_id in self.topology_cache.get_cluster_ids():
          # TODO: check if we can make clusters immutable too
          try:
            topology_cache = self.topology_cache[cluster_id]
            metadata_cache = self.metadata_cache[cluster_id]
          except KeyError:
            # multithreading: if cluster was deleted during iteration
            continue

          if 'status_commands_to_run' not in metadata_cache:
            continue

          status_commands_to_run = metadata_cache.status_commands_to_run

          if 'components' not in topology_cache:
            continue

          current_host_id = self.topology_cache.get_current_host_id(cluster_id)

          if current_host_id is None:
            continue

          cluster_components = topology_cache.components
          for component_dict in cluster_components:
            # 这里command_name只有status
            for command_name in status_commands_to_run:

              if self.stop_event.is_set():
                break

              # cluster was already removed
              if cluster_id not in self.topology_cache.get_cluster_ids():
                break

              # check if component is installed on current host
              if current_host_id not in component_dict.hostIds:
                break

              service_name = component_dict.serviceName
              component_name = component_dict.componentName

              # do not run status commands for the component which is starting/stopping or doing other action
              if self.customServiceOrchestrator.commandsRunningForComponent(cluster_id, component_name):
                self.logger.info("Skipping status command for {0}. Since command for it is running".format(component_name))
                continue
              # cluster_id=topology_cache.get_cluster_ids()
              # component_dict in cluster_components
              # service_name=component_dict.serviceName
              # component_name = component_dict.componentName
              # metadata_cache = self.metadata_cache[cluster_id]
              # status_commands_to_run = metadata_cache.status_commands_to_run
              # command_name in status_commands_to_run
              result = self.check_component_status(cluster_id, service_name, component_name, command_name)

              if result:
                cluster_reports[cluster_id].append(result)


        cluster_reports = self.discard_stale_reports(cluster_reports)
        self.send_updates_to_server(cluster_reports)
      except ConnectionIsAlreadyClosed: # server and agent disconnected during sending data. Not an issue
        pass
      except:
        self.logger.exception("Exception in ComponentStatusExecutor. Re-running it")
      # 每status_commands_run_interval秒内执行所有状态命令（默认20s)
      self.stop_event.wait(self.status_commands_run_interval)
    self.logger.info("ComponentStatusExecutor has successfully finished")

3、对run()归纳如下

run()方法中
运行一个无限循环，在每status_commands_run_interval秒内执行所有状态命令（默认20s）

self.stop_event.wait(self.status_commands_run_interval)

3.1、循环所有cluster_id

for cluster_id in self.topology_cache.get_cluster_ids():

topology_cache为InitializerModule.init()中初始化的属性

self.topology_cache = ClusterTopologyCache(self.config.cluster_cache_dir, self.config)

3.2、获取topology_cache、metadata_cache

topology_cach与metadata_cache实际上都是从
/var/lib/ambari-agent/cache/cluster_cache
目录中进行查询的

 topology_cache = self.topology_cache[cluster_id]

 metadata_cache = self.metadata_cache[cluster_id]

如果cluster_id不存在，则continue
修改下面的文件
vim /etc/ambari-agent/conf/ambari-agent.ini
loglevel=DEBUG
DEBUG后，日志中topology_cache、metadata_cache我各截取了部分
topology_cache：

metadata_cache：

3.3、获取status_commands_to_run、current_host_id

if 'status_commands_to_run' not in metadata_cache:
  continue

status_commands_to_run = metadata_cache.status_commands_to_run

如果metadata_cache中不存在status_commands_to_run，则continue

   if 'components' not in topology_cache:
       continue

   current_host_id = self.topology_cache.get_current_host_id(cluster_id)

   if current_host_id is None:
     continue

topology_cache中没有components则continue
current_host_id 从topology_cache取不到的话也continue

3.4、获取cluster_components并循环所有component

 cluster_components = topology_cache.components
 for component_dict in cluster_components:
 	# 这里command_name只有status
 	for command_name in status_commands_to_run:
		 if self.stop_event.is_set():
	       break
	
	     # cluster was already removed
	     if cluster_id not in self.topology_cache.get_cluster_ids():
	       break
	
	     # check if component is installed on current host
	     if current_host_id not in component_dict.hostIds:
	       break

在每一个集群cluster_id的循环里

• 要对获取到的所有component进行循环

  • 并对该component进行所有要执行的status_command指令循环（# 这里command_name只有status）
    （这里注意下，此循环逻辑说白了就是收集所有组件信息，并非执行server端传来的命令，server端的命令是由另外一个线程CommandStatusReporter执行的）

  • 如果有stop_event事件，则break跳出，等待下一个20s执行下一轮while循环

  • 在这时检查cluster_id是否还在topology_cache.get_cluster_ids()中，current_host_id主机id是否还在component_dict.hostIds中
    （因为循环很多次，需要在每次执行前进行cluster、host存在的确认）

  • 在当前要执行的status_command命令执行前获取component_dict组件名称以及对应的服务名称（例如NAMENODE，HDFS；HMASTER，HBASE等）

    service_name = component_dict.serviceName
    component_name = component_dict.componentName
    

  • 不为正在启动/停止或正在执行其他操作的组件运行status commands（过滤掉这些处于操作状态的组件）

     if self.customServiceOrchestrator.commandsRunningForComponent(cluster_id, component_name):
         self.logger.info("Skipping status command for {0}. Since command for it is running".format(component_name))
         continue
    

  • 对当前组件执行status检查命令，并返回结果集（check_component_status逻辑为：如果组件状态已更改，则返回组件状态，否则为None）

    result = self.check_component_status(cluster_id, service_name, component_name, command_name)
    

    check_component_status()方法的解析请看第4小节

4、check_component_status()

  def check_component_status(self, cluster_id, service_name, component_name, command_name, report=False):
    """
    Returns components status if it has changed, otherwise None.
    """

    # if not a component
    if self.topology_cache.get_component_info_by_key(cluster_id, service_name, component_name) is None:
      return None

    command_dict = {
      'serviceName': service_name,
      'role': component_name,
      'clusterId': cluster_id,
      'commandType': AgentCommand.status,
    }

    component_status_result = self.customServiceOrchestrator.requestComponentStatus(command_dict)
    status = LiveStatus.LIVE_STATUS if component_status_result['exitcode'] == 0 else LiveStatus.DEAD_STATUS

    # log if status command failed
    if status == LiveStatus.DEAD_STATUS:
      stderr = component_status_result['stderr']
      if "ComponentIsNotRunning" not in stderr and "ClientComponentHasNoStatus" not in stderr:
        self.logger.info("Status command for {0} failed:\n{1}".format(component_name, stderr))

    result = {
      'serviceName': service_name,
      'componentName': component_name,
      'command': command_name,
      'status': status,
      'clusterId': cluster_id,
    }

    if status != self.reported_component_status[cluster_id]["{0}/{1}".format(service_name, component_name)][command_name]:
      logging.info("Status for {0} has changed to {1}".format(component_name, status))
      self.recovery_manager.handle_status_change(component_name, status)

      if report:
        with self.reports_to_discard_lock:
          self.reports_to_discard.append(result)

        self.send_updates_to_server({cluster_id: [result]})

      return result
    return None

其中
requestComponentStatus()做组件状态请求，然后返回component_status_result结果集

 component_status_result = self.customServiceOrchestrator.requestComponentStatus(command_dict)

DEBUG后将这个component_status_result 取了出来，如下

{
	'structuredOut': {
		u'version': u'3.1.0.0-78',
		u'repository_version_id': 1
	},
	'stdout': "2020-07-08 11:10:24,938 - Action afix 'pre_get_version' not present\n2020-07-08 11:10:24,938 - Action afix 'post_get_version' not present\nPackages:\n  accumulo-client\n  accumulo-gc\n  accumulo-master\n  accumulo-monitor\n  accumulo-tablet\n  accumulo-tracer\n  atlas-client\n  atlas-server\n  beacon\n  beacon-client\n  beacon-server\n  druid-broker\n  druid-coordinator\n  druid-historical\n  druid-middlemanager\n  druid-overlord\n  druid-router\n  druid-superset\n  falcon-client\n  falcon-server\n  flume-server\n  hadoop-client\n  hadoop-hdfs-client\n  hadoop-hdfs-datanode\n  hadoop-hdfs-journalnode\n  hadoop-hdfs-namenode\n  hadoop-hdfs-nfs3\n  hadoop-hdfs-portmap\n  hadoop-hdfs-secondarynamenode\n  hadoop-hdfs-zkfc\n  hadoop-httpfs\n  hadoop-mapreduce-client\n  hadoop-mapreduce-historyserver\n  hadoop-yarn-client\n  hadoop-yarn-nodemanager\n  hadoop-yarn-registrydns\n  hadoop-yarn-resourcemanager\n  hadoop-yarn-timelinereader\n  hadoop-yarn-timelineserver\n  hbase-client\n  hbase-master\n  hbase-regionserver\n  hive-client\n  hive-metastore\n  hive-server2\n  hive-server2-hive\n  hive-server2-hive2\n  hive-webhcat\n  hive_warehouse_connector\n  kafka-broker\n  knox-server\n  livy-client\n  livy-server\n  livy2-client\n  livy2-server\n  mahout-client\n  oozie-client\n  oozie-server\n  phoenix-client\n  phoenix-server\n  pig-client\n  ranger-admin\n  ranger-kms\n  ranger-tagsync\n  ranger-usersync\n  shc\n  slider-client\n  spark-atlas-connector\n  spark-client\n  spark-historyserver\n  spark-schema-registry\n  spark-thriftserver\n  spark2-client\n  spark2-historyserver\n  spark2-thriftserver\n  spark_llap\n  sqoop-client\n  sqoop-server\n  storm-client\n  storm-nimbus\n  storm-slider-client\n  storm-supervisor\n  superset\n  tez-client\n  zeppelin-server\n  zookeeper-client\n  zookeeper-server\nAliases:\n  accumulo-server\n  all\n  client\n  hadoop-hdfs-server\n  hadoop-mapreduce-server\n  hadoop-yarn-server\n  hive-server\n2020-07-08 11:10:25,023 - Command: /usr/bin/hdp-select status hadoop-hdfs-datanode > /tmp/tmpTAl577\nOutput: hadoop-hdfs-datanode - 3.1.0.0-78\n\n2020-07-08 11:10:25,024 - Version for component hadoop-hdfs-datanode: 3.1.0.0-78",
	'stderr': '',
	'exitcode': 0
}

requestComponentStatus()如下

  def requestComponentStatus(self, command_header, command_name="STATUS"):
    """
     Component status is determined by exit code, returned by runCommand().
     Exit code 0 means that component is running and any other exit code means that
     component is not running
    """
    override_output_files = True
    if logger.level == logging.DEBUG:
      override_output_files = False

    # make sure status commands that run in parallel don't use the same files
    status_commands_stdout = self.status_commands_stdout.format(uuid.uuid4())
    status_commands_stderr = self.status_commands_stderr.format(uuid.uuid4())
    status_structured_out = self.status_structured_out.format(uuid.uuid4())

    try:
      res = self.runCommand(command_header, status_commands_stdout,
                            status_commands_stderr, command_name,
                            override_output_files=override_output_files, is_status_command=True,
                            tmpstrucoutfile=status_structured_out)
      #add debug log
      logger.debug('=====gaofeng=====requestComponentStatus=====status_commands_stdout=== %s',status_commands_stdout)
      logger.debug('=====gaofeng=====requestComponentStatus=====status_commands_stderr=== %s',status_commands_stderr)
      logger.debug('=====gaofeng=====requestComponentStatus=====status_structured_out=== %s',status_structured_out)
      logger.debug('=====gaofeng=====requestComponentStatus=====res=== %s',res)

    finally:
      try:
        os.unlink(status_commands_stdout)
        os.unlink(status_commands_stderr)
        os.unlink(status_structured_out)
      except OSError:
        pass # Ignore failure

    return res

DEBUG 2020-07-08 11:10:25,050 CustomServiceOrchestrator.py:541 - =====gaofeng=====requestComponentStatus=====status_commands_stdout=== /var/lib/ambari-agent/data/status_command_stdout_ce21f1b2-d80d-46aa-949c-5576a2ef6755.txt
DEBUG 2020-07-08 11:10:25,050 CustomServiceOrchestrator.py:542 - =====gaofeng=====requestComponentStatus=====status_commands_stderr=== /var/lib/ambari-agent/data/status_command_stderr_73644f49-c8d8-4f3f-8841-6e733271ce40.txt
DEBUG 2020-07-08 11:10:25,050 CustomServiceOrchestrator.py:543 - =====gaofeng=====requestComponentStatus=====status_structured_out=== /var/lib/ambari-agent/data/status_structured-out-a9bc749d-c09a-49a5-b722-cd26502b5ea3.json
DEBUG 2020-07-08 11:10:25,050 CustomServiceOrchestrator.py:544 - =====gaofeng=====requestComponentStatus=====res=== {'structuredOut': {u'version': u'3.1.0.0-78', u'repository_version_id': 1}, 'stdout': "2020-07-08 11:10:24,938 - Action afix 'pre_get_version' not present\n2020-07-08 11:10:24,938 - Action afix 'post_get_version' not present\nPackages:\n  accumulo-client\n  accumulo-gc\n  accumulo-master\n  accumulo-monitor\n  accumulo-tablet\n  accumulo-tracer\n  atlas-client\n  atlas-server\n  beacon\n  beacon-client\n  beacon-server\n  druid-broker\n  druid-coordinator\n  druid-historical\n  druid-middlemanager\n  druid-overlord\n  druid-router\n  druid-superset\n  falcon-client\n  falcon-server\n  flume-server\n  hadoop-client\n  hadoop-hdfs-client\n  hadoop-hdfs-datanode\n  hadoop-hdfs-journalnode\n  hadoop-hdfs-namenode\n  hadoop-hdfs-nfs3\n  hadoop-hdfs-portmap\n  hadoop-hdfs-secondarynamenode\n  hadoop-hdfs-zkfc\n  hadoop-httpfs\n  hadoop-mapreduce-client\n  hadoop-mapreduce-historyserver\n  hadoop-yarn-client\n  hadoop-yarn-nodemanager\n  hadoop-yarn-registrydns\n  hadoop-yarn-resourcemanager\n  hadoop-yarn-timelinereader\n  hadoop-yarn-timelineserver\n  hbase-client\n  hbase-master\n  hbase-regionserver\n  hive-client\n  hive-metastore\n  hive-server2\n  hive-server2-hive\n  hive-server2-hive2\n  hive-webhcat\n  hive_warehouse_connector\n  kafka-broker\n  knox-server\n  livy-client\n  livy-server\n  livy2-client\n  livy2-server\n  mahout-client\n  oozie-client\n  oozie-server\n  phoenix-client\n  phoenix-server\n  pig-client\n  ranger-admin\n  ranger-kms\n  ranger-tagsync\n  ranger-usersync\n  shc\n  slider-client\n  spark-atlas-connector\n  spark-client\n  spark-historyserver\n  spark-schema-registry\n  spark-thriftserver\n  spark2-client\n  spark2-historyserver\n  spark2-thriftserver\n  spark_llap\n  sqoop-client\n  sqoop-server\n  storm-client\n  storm-nimbus\n  storm-slider-client\n  storm-supervisor\n  superset\n  tez-client\n  zeppelin-server\n  zookeeper-client\n  zookeeper-server\nAliases:\n  accumulo-server\n  all\n  client\n  hadoop-hdfs-server\n  hadoop-mapreduce-server\n  hadoop-yarn-server\n  hive-server\n2020-07-08 11:10:25,023 - Command: /usr/bin/hdp-select status hadoop-hdfs-datanode > /tmp/tmpTAl577\nOutput: hadoop-hdfs-datanode - 3.1.0.0-78\n\n2020-07-08 11:10:25,024 - Version for component hadoop-hdfs-datanode: 3.1.0.0-78", 'stderr': '', 'exitcode': 0}

再回到check_component_status()中

对比之前汇报给server端的
[cluster_id][service_name/component_name][command_name]维度的状态信息
如果当前的状态与服务端不一致，则记录日志，’状态发生改变’，
并调用recovery_manager.handle_status_change修改状态

5、总结

前提准备
InitializerModule.py中初始化metadata_cache、topology_cache等属性
（/var/lib/ambari-agent/cache/cluster_cache获取的）
在ComponentStatusExecutor.run()中
1、循环topology_cache中的所有cluster_ids
2、循环topology_cache中当前cluster_id的所有components
3、循环metadata_cache中status_commands_to_run所有的command_name（其实就一个status）
4、check_component_status()检查组件结果状态，如果组件状态已更改,
   则给server返回组件状态，否则等待20s执行下一个ComponentStatusExecutor.run()。