Flume architecture and operating mechanism
1.Flume's three-tier architecture
Flume uses a three-tier architecture, namely agent, collector, and storage, each of which can be expanded horizontally. Among them, all agents and collectors are managed uniformly by the master, which makes the system easy to monitor and maintain, and multiple masters (using ZooKeeper for management and load balancing) are allowed, which avoids single point of failure.
1) Agent layer: This layer contains the Agent components of Flume and is connected to the data source that needs to transmit data.
2) Collector: This layer collects data from the Agent layer through multiple collectors, and then forwards these to the next Layer
3) storage: This layer receives data from the collector layer and stores it
2.flume operating mechanism
1. The core role in the Flume distributed system is the agent, and the flume collection system is formed by connecting
each agent. 2. Each agent is equivalent to a data transmitter, and there are three internal components:
a. Source: the source of collection, Used to
connect with the data source to obtain data b. Sink: sink to the ground, the purpose of collecting data transmission, used to transfer data to the next level of agent to transfer data to the final storage system
c. Channel: the data transmission channel inside the angent , Used to transfer data from source to sink