Flink asynchronous IO access external data (mysql papers)

　　Continued article: [translation] Flink asynchronous I / O access to external data

　　Gangster recently read a blog, suddenly remembered Async I / O mode is one of the important functions of Blink push to the community, access to external data can be used in an asynchronous manner, thinking themselves to achieve the following, when used on the project, can not now I went to.

　　Initially I want to achieve a data read hbase demo using the scala, referring to the official website demo:　　

/**
 * An implementation of the 'AsyncFunction' that sends requests and sets the callback.
 */
class AsyncDatabaseRequest extends AsyncFunction[String, (String, String)] {

    /** The database specific client that can issue concurrent requests with callbacks */
    lazy val client: DatabaseClient = new DatabaseClient(host, post, credentials)

    /** The context used for the future callbacks */
    implicit lazy val executor: ExecutionContext = ExecutionContext.fromExecutor(Executors.directExecutor())


    override def asyncInvoke(str: String, resultFuture: ResultFuture[(String, String)]): Unit = {

        // issue the asynchronous request, receive a future for the result
        val resultFutureRequested: Future[String] = client.query(str)

        // set the callback to be executed once the request by the client is complete
        // the callback simply forwards the result to the result future
        resultFutureRequested.onSuccess {
            case result: String => resultFuture.complete(Iterable((str, result)))
        }
    }
}

// create the original stream
val stream: DataStream[String] = ...

// apply the async I/O transformation
val resultStream: DataStream[(String, String)] =
    AsyncDataStream.unorderedWait(stream, new AsyncDatabaseRequest(), 1000, TimeUnit.MILLISECONDS, 100)

Failed, the red icon on the part not realize

1, Future implementation class can not find the

2, unorderedWait been an error

Source example and there was the case of Scala

def main(args: Array[String]) {
    val timeout = 10000L

    val env = StreamExecutionEnvironment.getExecutionEnvironment

    val input = env.addSource(new SimpleSource())

    val asyncMapped = AsyncDataStream.orderedWait(input, timeout, TimeUnit.MILLISECONDS, 10) {
      (input, collector: ResultFuture[Int]) =>
        Future {
          collector.complete(Seq(input))
        } (ExecutionContext.global)
    }

    asyncMapped.print()

    env.execute("Async I/O job")
  }

The main part of this, the chicken dish represents weakness, want to inherit RichAsyncFunction, you can use the open method to initialize the link.

Online blog turned a lot, most of the principles of translation of the official website, there is no case that can be performed, distressed.

Failed.

Into java version, in the group asked yesterday, there is a big brother to give me a Java version: https://github.com/perkinls/flink-local-train/blob/c8b4efe33620352aea0100adef4fae2a068a3b65/src/main/scala/com /lp/test/asyncio/AsyncIoSideTableJoinMysqlJava.java have not seen, because the case Java version of the official website can understand.

Following the start mysql version of the source (HBase No tested, hbase the machine hang):

Business as follows:

Kafka receiving data into user object, call async, user.id query using the corresponding phone, back into user objects, the output

 Main categories:

import com.alibaba.fastjson.JSON;
import com.venn.common.Common;
import org.apache.flink.formats.json.JsonNodeDeserializationSchema;
import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode;
import org.apache.flink.streaming.api.datastream.AsyncDataStream;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
import java.util.concurrent.TimeUnit;


class AsyncMysqlRequest {

    public static void main(String[] args) throws Exception {

        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        FlinkKafkaConsumer<ObjectNode> source = new FlinkKafkaConsumer<>("async", new JsonNodeDeserializationSchema(), Common.getProp());

        // receiving kafka data into User object 
        DataStream <User> INPUT = env.addSource (Source) .map (value -> {
            String id = value.get("id").asText();
            String username = value.get("username").asText();
            String password = value.get("password").asText();

            return new User(id, username, password);
        });
        // asynchronous IO mysql data acquisition, timeout time 1s, capacity 100 (100+ request, the backpressure upstream node) 
        DataStream AsyncDataStream.unorderedWait the async = (INPUT, new new AsyncFunctionForMysqlJava (), 1000, TimeUnit.MICROSECONDS, 100 );

        async.map(user -> {

            return JSON.toJSON(user).toString();
        })
        .print();

        env.execute("asyncForMysql");

    }
}

Functions categories:

import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.functions.async.ResultFuture;
import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.ArrayList;
import java.util.List;

public class AsyncFunctionForMysqlJava extends RichAsyncFunction<User, User> {

    // 链接
    private static String jdbcUrl = "jdbc:mysql://192.168.229.128:3306?useSSL=false";
    private static String username = "root";
    private static String password = "123456";
    private static String driverName = "com.mysql.jdbc.Driver";


    java.sql.Connection conn;
    PreparedStatement ps;
    Logger logger = LoggerFactory.getLogger(AsyncFunctionForMysqlJava.class);

    /**
     * Open method of initializing link
     * @param parameters
     * @throws Exception
     */
    @Override
    public void open(Configuration parameters) throws Exception {
        logger.info("async function for hbase java open ...");
        super.open(parameters);

        Class.forName(driverName);
        conn = DriverManager.getConnection(jdbcUrl, username, password);
        ps = conn.prepareStatement("select phone from async.async_test where id = ?");
    }

    /**
     * use user.getId async get user phone
     *
     * @param user
     * @param resultFuture
     * @throws Exception
     */
    @Override
    public void asyncInvoke(User user, ResultFuture<User> resultFuture) throws Exception {
        // 使用 user id 查询
        ps.setString(1, user.getId());
        ResultSet rs = ps.executeQuery();
        String phone = null;
        if (rs.next()) {
            phone = rs.getString(1);
        }
        user.setPhone(phone);
        List<User> list = new ArrayList();
        list.add(user);
        // back into the result queue 
        resultFuture.complete (list);
    }

    @Override
    public void timeout(User input, ResultFuture<User> resultFuture) throws Exception {
        logger.info("Async function for hbase timeout");
        List<User> list = new ArrayList();
        list.add(input);
        resultFuture.complete(list);
    }

    /**
     * close function
     *
     * @throws Exception
     */
    @Override
    public void close() throws Exception {
        logger.info("async function for hbase java close ...");
        super.close();
        conn.close();
    }
}

Test data are as follows:

{"id" : 1, "username" : "venn", "password" : 1561709530935}
{"id" : 2, "username" : "venn", "password" : 1561709536029}
{"id" : 3, "username" : "venn", "password" : 1561709541033}
{"id" : 4, "username" : "venn", "password" : 1561709546037}
{"id" : 5, "username" : "venn", "password" : 1561709551040}
{"id" : 6, "username" : "venn", "password" : 1561709556044}
{"id" : 7, "username" : "venn", "password" : 1561709561048}

Execution results are as follows:

1> {"password":"1561709536029","phone":"12345678911","id":"2","username":"venn"}
1> {"password":"1561709541033","phone":"12345678912","id":"3","username":"venn"}
1> {"password":"1561709546037","phone":"12345678913","id":"4","username":"venn"}
1> {"password":"1561709551040","id":"5","username":"venn"} # 关联不上，原样返回
1> {"password":"1561709556044","id":"6","username":"venn"}
1> {"password":"1561709561048","id":"7","username":"venn"}

 

hbase, redis or other similar implement, inheritance AsyncStreamFunction, implementation