Presto Event Listener development

Brief introduction

Like Hive Hook, Presto also supports custom implementations Event Listener, for the event that occurs when the query is executed listening Presto engine, and make the appropriate treatment. We can use this feature to achieve, such as custom logging, debugging and performance analysis plug-ins to help us better operation and maintenance Presto cluster. But unlike Hive Hook is in Presto cluster, only one Event Listener is active.

Event Listener listens as Plugin the following events:

• Query Creation (inquiry to establish the relevant information)
• Query completion (success or failure) (query execution-related information, including details successful query information, error codes failed queries)
• Split completion (success or failure) (split execution information, empathy contain details of successes and failures)

Hook and friends understand Listener mode for the steps should be very clear, we just need to:

1. Achieve Presto Event Listener and EventListenerFactory interface.
2. Properly packed our jar.
3. Deployment into Presto specified directory, modify the configuration file.

interface

1. Implement EventListener, the class is where our core logic for containing the above mentioned three events:

public interface EventListener
{
    //query创建的详细信息
    default void queryCreated(QueryCreatedEvent queryCreatedEvent)
    {
    }
    //query执行的详细信息
    default void queryCompleted(QueryCompletedEvent queryCompletedEvent)
    {
    }
    //split执行的详细信息
    default void splitCompleted(SplitCompletedEvent splitCompletedEvent)
    {
    }
}

2. Achieve EventListenerFactory create our own implementation EventListener
3. Achieve Plugin interface, getEventListenerFactories () method to get ourselves to achieve EventListenerFactory
4. Add configuration information for the etc / event-listener.properties. Event-listener.name which is an essential attribute, other attributes to the information we needed plugin.

Examples

The need to cluster operation and maintenance, first you need to check the history of users, time spent on information query statistics, for grading and scoring priority in subsequent queries for each service, facilitate subsequent ease of maintenance Presto cluster stability . Here we will be given a simple sample Mysql information stored in the database.

Maven Pom

<dependency>
      <groupId>com.facebook.presto</groupId>
      <artifactId>presto-spi</artifactId>
      <version>0.220</version>
      <scope>compile</scope>
    </dependency>

QueryEventListenerFactory

public class QueryEventListenerFactory implements EventListenerFactory {

  @Override
  public String getName() {
    return "query-event-listener";
  }

  @Override
  public EventListener create(Map<String, String> config) {
    if (!config.containsKey("jdbc.uri")) {
      throw new RuntimeException("/etc/event-listener.properties file missing jdbc.uri");
    }
    if (!config.containsKey("jdbc.user")) {
      throw new RuntimeException("/etc/event-listener.properties file missing jdbc.user");
    }
    if (!config.containsKey("jdbc.pwd")) {
      throw new RuntimeException("/etc/event-listener.properties file missing jdbc.pwd");
    }

    return new QueryEventListener(config);
  }
}

QueryEventPlugin

public class QueryEventPlugin implements Plugin {

  @Override
  public Iterable<EventListenerFactory> getEventListenerFactories() {
    EventListenerFactory listenerFactory = new QueryEventListenerFactory();
    return Arrays.asList(listenerFactory);
  }
}

QueryEventListener

public class QueryEventListener implements EventListener {

  private Map<String, String> config;
  private Connection connection;

  public QueryEventListener(Map<String, String> config) {
    this.config = new HashMap<>();
    this.config.putAll(config);
    init();
  }

  private void init() {
    try {
      if (connection == null || !connection.isValid(10)) {
        Class.forName("com.mysql.jdbc.Driver");
        connection = DriverManager
            .getConnection(config.get("jdbc.uri"), config.get("jdbc.user"), config.get("jdbc.pwd"));
      }
    } catch (SQLException | ClassNotFoundException e) {
      e.printStackTrace();
    }
  }

  @Override
  public void queryCreated(QueryCreatedEvent queryCreatedEvent) {
  }

  @Override
  public void queryCompleted(QueryCompletedEvent queryCompletedEvent) {
    String queryId = queryCompletedEvent.getMetadata().getQueryId();
    String querySql = queryCompletedEvent.getMetadata().getQuery();
    String queryState = queryCompletedEvent.getMetadata().getQueryState();
    String queryUser = queryCompletedEvent.getContext().getUser();
    long createTime = queryCompletedEvent.getCreateTime().toEpochMilli();
    long endTime = queryCompletedEvent.getEndTime().toEpochMilli();
    long startTime = queryCompletedEvent.getExecutionStartTime().toEpochMilli();
    //insert into query execution table

    long analysisTime = queryCompletedEvent.getStatistics().getAnalysisTime().orElse(Duration.ZERO)
        .toMillis();
    long cpuTime = queryCompletedEvent.getStatistics().getCpuTime().toMillis();
    long queuedTime = queryCompletedEvent.getStatistics().getQueuedTime().toMillis();
    long wallTime = queryCompletedEvent.getStatistics().getWallTime().toMillis();
    int completedSplits = queryCompletedEvent.getStatistics().getCompletedSplits();
    double cumulativeMemory = queryCompletedEvent.getStatistics().getCumulativeMemory();
    long outputBytes = queryCompletedEvent.getStatistics().getOutputBytes();
    long outputRows = queryCompletedEvent.getStatistics().getOutputRows();
    long totalBytes = queryCompletedEvent.getStatistics().getTotalBytes();
    long totalRows = queryCompletedEvent.getStatistics().getTotalRows();
    long writtenBytes = queryCompletedEvent.getStatistics().getWrittenBytes();
    long writtenRows = queryCompletedEvent.getStatistics().getWrittenRows();
    //insert into query info table
    
    queryCompletedEvent.getFailureInfo().ifPresent(queryFailureInfo -> {
      int code = queryFailureInfo.getErrorCode().getCode();
      String name = queryFailureInfo.getErrorCode().getName();
      String failureType = queryFailureInfo.getFailureType().orElse("").toUpperCase();
      String failureHost = queryFailureInfo.getFailureHost().orElse("").toUpperCase();
      String failureMessage = queryFailureInfo.getFailureMessage().orElse("").toUpperCase();
      String failureTask = queryFailureInfo.getFailureTask().orElse("").toUpperCase();
      String failuresJson = queryFailureInfo.getFailuresJson();
      // insert into failed query table
    });
  }


  @Override
  public void splitCompleted(SplitCompletedEvent splitCompletedEvent) {
    long createTime = splitCompletedEvent.getCreateTime().toEpochMilli();
    long endTime = splitCompletedEvent.getEndTime().orElse(Instant.MIN).toEpochMilli();
    String payload = splitCompletedEvent.getPayload();
    String queryId = splitCompletedEvent.getQueryId();
    String stageId = splitCompletedEvent.getStageId();
    long startTime = splitCompletedEvent.getStartTime().orElse(Instant.MIN).toEpochMilli();
    String taskId = splitCompletedEvent.getTaskId();
    long completedDataSizeBytes = splitCompletedEvent.getStatistics().getCompletedDataSizeBytes();
    long completedPositions = splitCompletedEvent.getStatistics().getCompletedPositions();
    long completedReadTime = splitCompletedEvent.getStatistics().getCompletedReadTime().toMillis();
    long cpuTime = splitCompletedEvent.getStatistics().getCpuTime().toMillis();
    long queuedTime = splitCompletedEvent.getStatistics().getQueuedTime().toMillis();
    long wallTime = splitCompletedEvent.getStatistics().getWallTime().toMillis();
    //insert into stage info table
  }

}

Bale

1. Use Presto Service Provider Interface (SPI) to expand Presto. Presto using SPI loading connector , function , type , and system access control . SPI loaded through metadata file. We also need to create src/main/resources/META-INF/services/com.facebook.presto.spi.Plugina metadata file. The file should contain the name of the class our plugins such as:com.ji3jin.presto.listener.QueryEventListener
2. Execution mvn clean installpackaging

deploy

1. Create a configuration file etc / event-listener.properties

event-listener.name=query-event-listener

jdbc.uri=jdbc:mysql://localhost:3306/presto_monitor
jdbc.user=presto
jdbc.pwd=presto123

2. Presto created in the root directory of the query-event-listenerdirectory, the name is consistent with our top event listener's name
3. Our jar package jar package mysql connector and copied to the directory created above
4. Presto service to restart

Well, now you can execute the query, then you can see statistics for your query history and the relevant time in the Mysql. If your current job is also a need for this, so what, fast hands to achieve a bar.

I welcome the attention of the public number: three gold big data