In the previous blog [Introduction to Flink] Flink reads Kafka data Demo has briefly introduced Flink to read Kafka data and practice it through Demo. This blog briefly introduces Flink to read MySQL data through a custom Source and exercise through Demo. .
First, let's briefly understand the SourceFunction interface, which is the root interface of all stream sources, and it inherits from a marked interface (empty interface) Function.
Open SourceFunction in IDEA, right click the mouse as shown in the figure below, and select Diagrams–>show Diagrams
SourceFunction defines two interface methods:
1. run: Start a source, that is, connect an external data source and emit elements to form a stream (in most cases, the stream is generated by running a while loop in this method).
2. Cancel: Cancel a source, that is, terminate the behavior of the loop emit element in the run.
Under normal circumstances, a SourceFunction can implement these two interface methods. In fact, these two interface methods also fixed an implementation template.
Next, implement
it through Demo. First, add MySQL dependency in pom.xml:
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.27</version>
</dependency>
MySQL database table creation
drop table if exists user_order_count;
create table user_order_count (
user_id varchar(25) NOT NULL,
count int(11),
primary key (user_id)
) engine=innodb default charset=utf8 collate=utf8_bin;
Import simulation data
insert into user_order_count values ('16935394', 6), ('16374609', 4), ('16570065', 4), ('4611433', 3), ('17308713', 3);
Create a new corresponding entity class: UserOrderCount
package com.fuyun.flink.model;
public class UserOrderCount {
public String userId;
public int count;
public UserOrderCount() {
}
public UserOrderCount(String userId, int count){
this.userId = userId;
this.count = count;
}
@Override
public String toString() {
return "UserOrderCount{" +
"userId=" + userId +
", count=" + count +
'}';
}
public String getUserId() {
return userId;
}
public void setUserId(String userId) {
this.userId = userId;
}
public int getCount() {
return count;
}
public void setCount(int count) {
this.count = count;
}
}
Create a new Source class SourceFromMySQL.java, which inherits RichSourceFunction and implements the open, close, run, and cancel methods inside:
package com.fuyun.flink.souce;
import com.fuyun.flink.model.UserOrderCount;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.functions.source.RichSourceFunction;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
public class SourceFromMySQL extends RichSourceFunction<UserOrderCount> {
PreparedStatement ps;
private Connection connection;
/**
* open() 方法中建立连接，这样不用每次 invoke 的时候都要建立连接和释放连接。
*
* @param parameters
* @throws Exception
*/
@Override
public void open(Configuration parameters) throws Exception {
super.open(parameters);
connection = getConnection();
String sql = "select * from user_order_count;"; // 编写具体逻辑代码
ps = this.connection.prepareStatement(sql);
}
/**
* 程序执行完毕就可以进行，关闭连接和释放资源的动作了
*
* @throws Exception
*/
@Override
public void close() throws Exception {
super.close();
if (connection != null) {
//关闭连接和释放资源
connection.close();
}
if (ps != null) {
ps.close();
}
}
@Override
public void run(SourceContext<UserOrderCount> ctx) throws Exception {
ResultSet resultSet = ps.executeQuery(); // 执行SQL语句返回结果集
while (resultSet.next()) {
UserOrderCount userOrderCount = new UserOrderCount(
resultSet.getString("user_id").trim(),
resultSet.getInt("count"));
ctx.collect(userOrderCount);
}
}
@Override
public void cancel() {
}
private static Connection getConnection() {
Connection con = null;
try {
Class.forName("com.mysql.jdbc.Driver");
con = DriverManager.getConnection("jdbc:mysql://bigdata-training.fuyun.com:3306/test?useUnicode=true&characterEncoding=UTF-8", "root", "123456");
} catch (Exception e) {
System.out.println("-----------mysql get connection has exception , msg = "+ e.getMessage());
}
return con;
}
}
Flink main program
package com.fuyun.flink
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment
import com.fuyun.flink.souce.SourceFromMySQL
object SourceMain {
def main(args: Array[String]): Unit = {
// 创建流处理环境
val env = StreamExecutionEnvironment.getExecutionEnvironment()
env.addSource(new SourceFromMySQL).print
env.execute("Flink add data sourc")
}
}
operation result:
