flume源码分析3--组件的启动

接上面，当获取到最新的配置文件后，触发监听方法重启组件：

@Subscribe
public synchronized void handleConfigurationEvent(MaterializedConfiguration conf) {
  stopAllComponents();
  startAllComponents(conf);
}

下面看startAllComponents方法：

private void startAllComponents(MaterializedConfiguration materializedConfiguration) {
  logger.info("Starting new configuration:{}", materializedConfiguration);

  this.materializedConfiguration = materializedConfiguration;

  for (Entry<String, Channel> entry :
      materializedConfiguration.getChannels().entrySet()) {
    try {
      logger.info("Starting Channel " + entry.getKey());
      supervisor.supervise(entry.getValue(),
          new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
    } catch (Exception e) {
      logger.error("Error while starting {}", entry.getValue(), e);
    }
  }

  /*
   * Wait for all channels to start.
   */
  for (Channel ch : materializedConfiguration.getChannels().values()) {
    while (ch.getLifecycleState() != LifecycleState.START
        && !supervisor.isComponentInErrorState(ch)) {
      try {
        logger.info("Waiting for channel: " + ch.getName() +
            " to start. Sleeping for 500 ms");
        Thread.sleep(500);
      } catch (InterruptedException e) {
        logger.error("Interrupted while waiting for channel to start.", e);
        Throwables.propagate(e);
      }
    }
  }

  for (Entry<String, SinkRunner> entry : materializedConfiguration.getSinkRunners().entrySet()) {
    try {
      logger.info("Starting Sink " + entry.getKey());
      supervisor.supervise(entry.getValue(),
          new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
    } catch (Exception e) {
      logger.error("Error while starting {}", entry.getValue(), e);
    }
  }

  for (Entry<String, SourceRunner> entry :
       materializedConfiguration.getSourceRunners().entrySet()) {
    try {
      logger.info("Starting Source " + entry.getKey());
      supervisor.supervise(entry.getValue(),
          new SupervisorPolicy.AlwaysRestartPolicy(), LifecycleState.START);
    } catch (Exception e) {
      logger.error("Error while starting {}", entry.getValue(), e);
    }
  }

  this.loadMonitoring();
}

启动顺序：启动channel、等待channel启动成功、启动sinks、启动source。

分析sink的启动：

前面提过，在加载配置文件的时候，SinkRunner可能对应一个sink也可能对应一个sinkgroup。因为如果配置文件中有sinkgroup则这个sinkgroup对应的sink会组成一个group然后封装为一个sinkRunner，然后不在sinkgroup中的sink会自己成为一个sinkRunner。每个SinkRunner的构造方法的参数是一个SinkProcessor是用来处理多个sink的。

看下SinkRunner的start方法，：

@Override
public void start() {
  SinkProcessor policy = getPolicy();

  policy.start();

  runner = new PollingRunner();

  runner.policy = policy;
  runner.counterGroup = counterGroup;
  runner.shouldStop = new AtomicBoolean();

  runnerThread = new Thread(runner);
  runnerThread.setName("SinkRunner-PollingRunner-" +
      policy.getClass().getSimpleName());
  runnerThread.start();

  lifecycleState = LifecycleState.START;
}

首先会获取SinkProcessor ，如果没有使用group，那么就是默认的DefaultSinkProcessor，默认只处理一个sink。

PollingRunner是一个线程，启动代码如下：

@Override
public void run() {
  logger.debug("Polling sink runner starting");

  while (!shouldStop.get()) {
    try {
      if (policy.process().equals(Sink.Status.BACKOFF)) {
        counterGroup.incrementAndGet("runner.backoffs");

        Thread.sleep(Math.min(
            counterGroup.incrementAndGet("runner.backoffs.consecutive")
            * backoffSleepIncrement, maxBackoffSleep));
      } else {
        counterGroup.set("runner.backoffs.consecutive", 0L);
      }
    } catch (InterruptedException e) {
      logger.debug("Interrupted while processing an event. Exiting.");
      counterGroup.incrementAndGet("runner.interruptions");
    } catch (Exception e) {
      logger.error("Unable to deliver event. Exception follows.", e);
      if (e instanceof EventDeliveryException) {
        counterGroup.incrementAndGet("runner.deliveryErrors");
      } else {
        counterGroup.incrementAndGet("runner.errors");
      }
      try {
        Thread.sleep(maxBackoffSleep);
      } catch (InterruptedException ex) {
        Thread.currentThread().interrupt();
      }
    }
  }
  logger.debug("Polling runner exiting. Metrics:{}", counterGroup);
}

主要就是循环调用policy.process()，policy前面说了，默认实现是DefaultSinkProcessor：

@Override
public Status process() throws EventDeliveryException {
  return sink.process();
}

接着随便找一个sink，比如org.apache.flume.sink.LoggerSink#process

@Override
public Status process() throws EventDeliveryException {
  Status result = Status.READY;
  Channel channel = getChannel();
  Transaction transaction = channel.getTransaction();
  Event event = null;

  try {
    transaction.begin();
    event = channel.take();

    if (event != null) {
      if (logger.isInfoEnabled()) {
        logger.info("Event: " + EventHelper.dumpEvent(event, maxBytesToLog));
      }
    } else {
      // No event found, request back-off semantics from the sink runner
      result = Status.BACKOFF;
    }
    transaction.commit();
  } catch (Exception ex) {
    transaction.rollback();
    throw new EventDeliveryException("Failed to log event: " + event, ex);
  } finally {
    transaction.close();
  }

  return result;
}

就是从channel获取event，然后事物提交。