SinkProcessor： ============================ FailOver： Load balancing: // load balancing processor // round_robin polling 1-2-3-1-2-3-... // random random 1-3-2-3-1-... 1. round_robin polling 1-2-3-1-2-3 -... 2 , random random: Custom Sink && InterCeptor =============================================== 1、pom <dependency> <groupId>org.apache.flume</groupId> <artifactId>flume-ng-core</artifactId> <version>1.8.0</version> </dependency> 2. Write sink: public class MySink extends AbstractSink { public Status process() throws EventDeliveryException { // Initialize status Status result = Status.READY; // Get channel object Channel channel = getChannel(); Transaction transaction = channel.getTransaction(); Event event = null ; try { // Open the transaction transaction.begin(); // Get the event from the channel event = channel.take(); if (event != null ) { // Manually add the header Map to the event <String,String> map = new HashMap<String, String> (); map.put("TimeStamp",System.currentTimeMillis() + ""); event.setHeaders(map); // Get body byte [] body = event.getBody(); // Get head value String headVal = event.getHeaders().get("TimeStamp" ); System.out.println("head: "+ headVal + "\tbody: " + new String(body)); } else { // There is no event, that is, backoff result = Status.BACKOFF; } // Commit the transaction transaction.commit(); } catch (Exception ex) { //回滚事务 transaction.rollback(); throw new EventDeliveryException("Failed to log event: " + event, ex); } finally { // Close the transaction transaction.close(); } return result; } } 3. Package and put it under /soft/flume/ lib 4. Use custom sink a1.sources = r1 a1.sinks = k1 a1.channels = c1 # configure source a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 8888 # configure sink a1.sinks.k1.type = com.oldboy.flume.MySink # configure channel a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # 绑定channel-source, channel-sink a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1 Custom interceptor: InterCeptor event size interceptor ================================ Custom speed limit interceptor: 1. Set the parameters 2. Set the construction with parameters 3. Create the Builder inner class, and construct the object through the Builder 1 ) Get the parameter value or default value through configure 2 ) Through the build method, build the InterCeptor object 4. Set the constant value in the Constants inner class public class SpeedInterceptor implements Interceptor { private int speed; public SpeedInterceptor(int speed) { this.speed = speed; } public void initialize() { } /** * Modify the event * Speed limit blocker, speed limit range, single event * The time range needs to pay attention to the first time * speed = bodySize / time * * Calculate the speed of the previous event, if the speed is too fast, sleep * lastTime * lastBodySize */ private long lastTime = -1 ; private long lastBodySize = 0; public Event intercept(Event event) { Map <String, String> headers = event.getHeaders(); // Get the length of the body long bodySize = event.getBody().length; // Get the current time long current = System.currentTimeMillis(); // The first event if (lastTime == -1 ){ lastTime = current; lastBodySize = bodySize; } // Not the first event else { long duration = current - lastTime; int currSpeed = ( int ) (( double )lastBodySize / duration * 1000 ); // The speed is not over if ( speed >= currSpeed){ return event; } // The speed exceeds the else { try { Thread.sleep(lastBodySize/speed * 1000 - duration); } catch (Exception e) { e.printStackTrace (); } } lastBodySize = bodySize; lastTime = System.currentTimeMillis(); } return event; } public List<Event> intercept(List<Event> events) { for (Event event : events) { intercept(event); } return events; } public void close() { } public static class Builder implements Interceptor.Builder { private int speed; public void configure(Context context) { //相当于 context.getInteger("speed",1024); speed = context.getInteger(Constants.SPEED, Constants.SPEED_DEFAULT); } public Interceptor build() { return new SpeedInterceptor(this.speed); } } public static class Constants { public static final String SPEED = "speed"; public static final int SPEED_DEFAULT = 1024; } } How to use a custom interceptor: ============================== 1. Write the code, package it and put it under /soft/flume/ lib 2. Write the configuration file i_speed.conf # Name the agent component a1.sources = r1 a1.sinks = k1 a1.channels = c1 # configure source a1.sources.r1.type = seq # Name the interceptor a1.sources.r1.interceptors = i1 # Specify the interceptor type a1.sources.r1.interceptors.i1.type = com.oldboy.flume.SpeedInterceptor$Builder a1.sources.r1.interceptors.i1.speed = 1 a1.sources.r1.interceptors.i1.speed2 = 10 # configure sink a1.sinks.k1.type = logger # configure channel a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # 绑定channel-source, channel-sink a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1 3、flume-ng agent -n a1 -f i_speed.conf Note: SinkProcessor and ChannelSelector ChannelSelector: select channel ----> sink SinkProcessor: pick sink When configuring SinkProcessor, note that ChannelSelector should be set to default (not configured) Flume configuration management using ZK: ============================================ 1. Create a node on the zk client (/flume/a1) // Note: node a1 is the agent name zkCli.sh -server s102:2181 2. Add data to the /flume/a1 node and use zooInspector // Chinese characters will appear garbled , you can use the idea plugin (zookeeper) a1.sources = r1 a1.sinks = k1 a1.channels = c1 a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 8888 a1.sinks.k1.type = logger a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1 3. Try to start flume flume-ng agent -n a1 -z s102:2181 -p /flume