Source code analysis of Flume-ng 1.6 startup process (1)

 The startup part is mainly divided into four parts

 

1. Loading of command line parameters, this part is implemented by common cli

2. For optional zk configuration loading

3. For the loading of flume-ng configuration, here we use EventBus to dynamically load configuration files

4. Startup of components

Briefly introduce EventBus , which is a message publish - subscribe class library provided by Guava , and the mechanism is similar to the observer mode.

 

The following mainly introduces Parts 3 and 4 mentioned above .

First of all , the startup class of flume-ng is org.apache.flume.node.Application located in the project of flume-ng-node

First introduce the ConfigurationProvider interface, which only provides one method

public MaterializedConfiguration getConfiguration();

 

 

Let's start with loading the flume-ng configuration and introduce the startup process

In general there are three parts :

1. Initialize the factory object of Source Channel Sink

2. Create a specific Source Channel Sink instance

3. Component startup

Details :

1. Create a factory object for Source Channel Sink

Enter the main method of the startup class org.apache.flume.node.Application

 

Take the else process as an example to see what the PropertiesFileConfigurationProvider did when initializing

 

The construction of the parent class is called. As can be seen from the previous inheritance diagram, its parent class AbstractConfigurationProvider ,

Continue to look at the construction of the parent class, initialize the factory class and channelCache

 

 

2. Create a specific Source Channel Sink instance

After creating the instance configurationProvider of PropertiesFileConfigurationProvider , configurationProvider.getConfiguration() is called in the code . Due to the inheritance relationship, the getConfiguration of AbstractConfigurationProvider is actually called to see what it does

 

After reading the configuration of Flume , loading the components, the loaded components will be placed in three Maps : channelComponentMap , sourceRunnerMap and sinkRunnerMap .

Take loadSources as an example, investigate how to load Sources

loadSources mainly does four things,

 

1. Use the previously instantiated DefaultSourceFactory object to instantiate the source , where the create method will first determine whether the type is built-in or custom based on the type, and then create an instance by reflection

2. Create a ChannelProcessor object and load the interceptor configuration

3. Return the SourceRunner object, which is divided into two categories: PollableSourceRunner and EventDrivenSourceRunner . The judgment here is based on whether the source implements the PollableSource interface or the EventDrivenSource interface. Of course, it is judged by instanceof in the code.
   

Expand here, Source is divided into PollableSource and EventDrivenSource , EventDrivenSource can understand time-driven, such sources include HttpSource , ExecSource , etc. PollableSource is a Source that requires external drivers to poll , such as KafkaSource .