系列文章目录

文章目录

系列文章目录
前言
一、测试思路
二、测试not enforced代码
三、改为enforce，报错
- 3.1 测试代码
四、 'write.upsert.enabled' = 'true', 设置这个参数实现upsert功能
总结

前言

测试 iceberg读取kafka的数据，能否根据kafka上的id，入湖时，自动更新iceberg的数据，对这个场景进行测试
测试结果：1.iceberg对从kafka流入的数据，默认是追加写的 2.通过给iceberg表设置 ‘write.upsert.enabled’ = 'true 参数，可以实现upsert模式

一、测试思路

从kafka制造数据写入iceberg，iceberg设置pk时，观察是追加写入还是更新。

二、测试not enforced代码

2.1 测试代码

测试思路： 1. select from kafka
2. insert to iceberg
代码如下：

CREATE TABLE IF NOT EXISTS KafkaTableTest2_XXZH (
    `id` bigint,
    `data` STRING
) WITH (
    'connector' = 'kafka',
    'topic' = 'test2_xxzh',
    'properties.bootstrap.servers' = 'hadoop101:9092,hadoop102:9092,hadoop103:9092',
    'properties.group.id' = 'testGroup',
    'scan.startup.mode' = 'latest-offset',
    'csv.ignore-parse-errors'='true',
    'format' = 'csv'
);


CREATE CATALOG hive_iceberg_catalog WITH (
    'type'='iceberg',
    'catalog-type'='hive',
    'uri'='thrift://hadoop101:9083',
    'clients'='5',
    'property-version'='1',
    'warehouse'='hdfs:///user/hive/warehouse/hive_iceberg_catalog'
);
use catalog hive_iceberg_catalog;
CREATE TABLE IF NOT EXISTS ods_base.IcebergTest2_XXZH (
    `id` bigint,
    `data` STRING,
    primary key (id) not enforced
)with(
    'write.metadata.delete-after-commit.enabled'='true',
    'write.metadata.previous-versions-max'='5',
    'format-version'='2'
 );
 

 
 insert into  hive_iceberg_catalog.ods_base.IcebergTest2_XXZH select * from default_catalog.default_database.KafkaTableTest2_XXZH;

2.2 制造数据

[root@hadoop101 conf]#  kafka-console-producer.sh --broker-list  hadoop101:9092,hadoop102:9092,hadoop103:9092  --topic test2_xxzh
>1,abc
[2022-07-22 14:55:51,643] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : {
    
    test2_xxzh=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
>2,bb
>3,cc
>4,dd
>5,ee
>3,cccc
>6,666
>4,ddddd
>

2.3 运行结果

spark-sql (default)> select *  from ods_base.IcebergTest2_XXZH;
22/07/22 15:12:28 WARN HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
id      data
3       cc
4       ddddd
5       ee
3       cccc
6       666
4       dd
Time taken: 0.405 seconds, Fetched 6 row(s)

flink-sql的运行结果：
在这里插入图片描述

2.4 运行结论

无法根据kafka声明的pk对iceberg进行更新。 iceberg是追加的模式写入的。

三、改为enforce，报错

3.1 测试代码

iceberg表的pk 改为enforced，重跑


Flink SQL> CREATE TABLE IF NOT EXISTS KafkaTableTest3_XXZH (
>     `id` bigint,
>     `data` STRING
> ) WITH (
>     'connector' = 'kafka',
>     'topic' = 'test2_xxzh',
>     'properties.bootstrap.servers' = 'hadoop101:9092,hadoop102:9092,hadoop103:9092',
>     'properties.group.id' = 'testGroup',
>     'scan.startup.mode' = 'latest-offset',
>     'csv.ignore-parse-errors'='true',
>     'format' = 'csv'
> );
> 
[INFO] Execute statement succeed.

Flink SQL> CREATE CATALOG hive_iceberg_catalog WITH (
>     'type'='iceberg',
>     'catalog-type'='hive',
>     'uri'='thrift://hadoop101:9083',
>     'clients'='5',
>     'property-version'='1',
>     'warehouse'='hdfs:///user/hive/warehouse/hive_iceberg_catalog'
> );
[INFO] Execute statement succeed.

Flink SQL> use catalog hive_iceberg_catalog;
[INFO] Execute statement succeed.

Flink SQL> CREATE TABLE IF NOT EXISTS ods_base.IcebergTest3_XXZH (
>     `id` bigint,
>     `data` STRING,
>     primary key (id) enforced
> )with(
>     'write.metadata.delete-after-commit.enabled'='true',
>     'write.metadata.previous-versions-max'='5',
>     'format-version'='2'
>  );
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.api.ValidationException: Flink doesn't support ENFORCED mode for PRIMARY KEY constraint. ENFORCED/NOT ENFORCED  controls if the constraint checks are performed on the incoming/outgoing data. Flink does not own the data therefore the only supported mode is the NOT ENFORCED mode

报错信息：
org.apache.flink.table.api.ValidationException: Flink doesn’t support ENFORCED mode for PRIMARY KEY constraint. ENFORCED/NOT ENFORCED controls if the constraint checks are performed on the incoming/outgoing data. Flink does not own the data therefore the only supported mode is the NOT ENFORCED mode

flink没有自己拥有这些数据，所以只支持的模式是非强值的。

结论： iceberg是没有进行根据pk对数据进行update

四、 ‘write.upsert.enabled’ = ‘true’, 设置这个参数实现upsert功能

CREATE TABLE IF NOT EXISTS ods_base.IcebergTest4_XXZH (
    `id` bigint,
    `data` STRING,
    primary key (id) not enforced
)with(
  'format-version' = '2',
  'write.upsert.enabled' = 'true',
  'write.distribution-mode'='hash',
  'write.metadata.delete-after-commit.enabled'='true',
  'write.metadata.previous-versions-max'='3'
 );

[root@hadoop102 module]#  kafka-console-producer.sh --topic test4_xxzh --broker-list hadoop101:9092,hadoop102:9092,hadoop103:9092
>2,222
>3,333  （这里暂停，去spark观察）
>2,bbbb
>3,cccc
>4,444
>5,555

初始化数据

spark-sql (default)> select * from  ods_base.IcebergTest4_XXZH ;
id      data
2       222
3       333

更新数据，id=2,3的内容都更新了

spark-sql (default)> select * from  ods_base.IcebergTest4_XXZH ;
22/07/26 19:24:58 WARN HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
id      data
2       bbbb
4       444
5       555
3       cccc

总结

1.iceberg对从kafka流入的数据，默认是追加写的
2.通过给iceberg表设置 ‘write.upsert.enabled’ = 'true 参数，可以实现upsert模式

实践数据湖iceberg 第三十七课 kakfa写入iceberg的 icberg表的 enfource ,not enfource测试