3.8. Write data to ClickHouse based on Flink
Write Flink to complete the operation of writing data to ClickHouse, and then complete the indicator statistics operation based on CK
3.8.1. Basic introduction of ClickHouse
ClickHouse is a columnar storage database (DBMS) open-sourced by Russia's Yandex in 2016. It is written in C++ language and is mainly used for online analytical processing queries (OLAP). It can use SQL queries to generate analysis data reports in real time.
Conclusion: ClickHouse, like many OLAP databases, has a single-table query speed due to associated queries, and the gap between the two in ClickHouse is more obvious.
3.8.2. ClickHouse installation steps
In this project, we only need to install the stand-alone test version to use (node2 installation). In actual production, you can directly install the distributed cluster version
- 1- Set yum source
sudo yum install yum-utils
sudo rpm --import https://repo.clickhouse.com/CLICKHOUSE-KEY.GPG
sudo yum-config-manager --add-repo https://repo.clickhouse.com/rpm/stable/x86_64
- 2- Install directly based on yum
sudo yum install clickhouse-server clickhouse-client
- 3- Modify the configuration file
vim /etc/clickhouse-server/config.xml
修改178行: 打开这一行的注释
<listen_host>::</listen_host>
- 4- Start the clickhouse server
systemctl start clickhouse-server
停止:
systemctl stop clickhouse-server
重启
systemctl restart clickhouse-server
- 5- Enter the client
3.8.3. Create target table in ClickHouse
create database itcast_ck;
use itcast_ck;
create table itcast_ck.itcast_ck_ems(
id int,
sid varchar(128),
ip varchar(128),
create_time varchar(128),
session_id varchar(128),
yearInfo varchar(128),
monthInfo varchar(128),
dayInfo varchar(128),
hourInfo varchar(128),
seo_source varchar(128),
area varchar(128),
origin_channel varchar(128),
msg_count int(128),
from_url varchar(128),
PRIMARY KEY (`id`)
) ENGINE=ReplacingMergeTree();
3.8.4. Write Flink code to complete writing to CK
import com.itheima.pojo.PulsarTopicPojo;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.io.jdbc.JDBCAppendTableSink;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.pulsar.FlinkPulsarSource;
import org.apache.flink.streaming.connectors.pulsar.internal.JsonDeser;
import org.apache.flink.types.Row;
import java.sql.Types;
import java.util.Properties;
// 基于Flink完成读取Pulsar中数据将消息数据写入到clickhouse中
public class ItcastFlinkToClickHouse {
public static void main(String[] args) throws Exception {
//1. 创建Flinnk流式处理核心环境类对象 和 Table API 核心环境类对象
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//2. 添加Source组件, 从Pulsar中读取消息数据
Properties props = new Properties();
props.setProperty("topic","persistent://public/default/itcast_ems_tab");
props.setProperty("partition.discovery.interval-millis","5000");
FlinkPulsarSource<PulsarTopicPojo> pulsarSource = new FlinkPulsarSource<PulsarTopicPojo>(
"pulsar://node1:6650,node2:6650,node3:6650","http://node1:8080,node2:8080,node3:8080",
JsonDeser.of(PulsarTopicPojo.class),props);
//2.1 设置pulsarSource组件在消费数据的时候, 默认从什么位置开始消费
pulsarSource.setStartFromLatest();
DataStreamSource<PulsarTopicPojo> dataStreamSource = env.addSource(pulsarSource);
//2.2 转换数据操作: 将 PulsarTopicPojo 转换为ROW对象
SingleOutputStreamOperator<Row> rowDataSteam = dataStreamSource.map(new MapFunction<PulsarTopicPojo, Row>() {
@Override
public Row map(PulsarTopicPojo pulsarTopicPojo) throws Exception {
return Row.of(pulsarTopicPojo.getId(), pulsarTopicPojo.getSid(), pulsarTopicPojo.getIp(), pulsarTopicPojo.getCreate_time(),
pulsarTopicPojo.getSession_id(), pulsarTopicPojo.getYearInfo(), pulsarTopicPojo.getMonthInfo(), pulsarTopicPojo.getDayInfo(),
pulsarTopicPojo.getHourInfo(), pulsarTopicPojo.getSeo_source(), pulsarTopicPojo.getArea(), pulsarTopicPojo.getOrigin_channel(),
pulsarTopicPojo.getMsg_count(), pulsarTopicPojo.getFrom_url());
}
});
//2.3: 设置sink操作写入到CK操作
String insertSql = "insert into itcast_ck.itcast_ck_ems (id,sid,ip,create_time,session_id,yearInfo,monthInfo,dayInfo,hourInfo,seo_source,area,origin_channel,msg_count,from_url) values(?,?,?,?,?,?,?,?,?,?,?,?,?,?)";
JDBCAppendTableSink tableSink = JDBCAppendTableSink.builder()
.setDrivername("ru.yandex.clickhouse.ClickHouseDriver")
.setDBUrl("jdbc:clickhouse://node2:8123/itcast_ck")
.setQuery(insertSql)
.setBatchSize(1)
.setParameterTypes(Types.INTEGER,Types.VARCHAR,Types.VARCHAR,Types.VARCHAR,Types.VARCHAR,Types.VARCHAR,Types.VARCHAR,Types.VARCHAR,Types.VARCHAR,Types.VARCHAR,Types.VARCHAR,Types.VARCHAR,Types.INTEGER,Types.VARCHAR)
.build();
tableSink.emitDataStream(rowDataSteam);
//3. 提交执行
env.execute("itcast_to_ck");
}
}
3.9. HBase connects to Phoenix to realize ad hoc query
3.9.1.Phoenix installation operation
Phoenix is an hbase-based tool belonging to apache. This tool provides a new way to operate data (SQL) in hbase. At the same time, Phoenix does
a lot of optimization work on hbase, which allows us to operate hbase more effectively.
For the entire installation operation, you can refer to the installation manual in the data and install it
3.9.2. Creating Tables in Phoenix
create view "itcast_h_ems" (
"id" integer primary key,
"f1"."sid" varchar,
"f1"."ip" varchar,
"f1"."create_time" varchar,
"f1"."session_id" varchar,
"f1"."yearInfo" varchar,
"f1"."monthInfo" varchar,
"f1"."dayInfo" varchar,
"f1"."hourInfo" varchar,
"f1"."seo_source" varchar,
"f1"."area" varchar,
"f1"."origin_channel" varchar,
"f1"."msg_count" integer,
"f1"."from_url" varchar
);
3.9.3. Type declarations in Phoenix
