Series Article Directory

Article directory

Series Article Directory
foreword
1. Test ideas
2. Test not enforced code
3. Change to enforce and report an error
- 3.1 Test code
4. 'write.upsert.enabled' = 'true', set this parameter to realize the upsert function
Summarize

foreword

Test whether iceberg reads data from kafka and whether it can automatically update iceberg data when entering the lake according to the id on kafka. Test
results for this scenario: 1. iceberg writes additional data by default for data flowing in from kafka 2 .The upsert mode can be realized by setting the 'write.upsert.enabled' = 'true parameter to the iceberg table

1. Test ideas

Write iceberg from kafka manufacturing data, and when iceberg sets pk, observe whether it is additional writing or updating.

2. Test not enforced code

2.1 Test code

Test ideas: 1. select from kafka
2. insert to iceberg
The code is as follows:

CREATE TABLE IF NOT EXISTS KafkaTableTest2_XXZH (
    `id` bigint,
    `data` STRING
) WITH (
    'connector' = 'kafka',
    'topic' = 'test2_xxzh',
    'properties.bootstrap.servers' = 'hadoop101:9092,hadoop102:9092,hadoop103:9092',
    'properties.group.id' = 'testGroup',
    'scan.startup.mode' = 'latest-offset',
    'csv.ignore-parse-errors'='true',
    'format' = 'csv'
);


CREATE CATALOG hive_iceberg_catalog WITH (
    'type'='iceberg',
    'catalog-type'='hive',
    'uri'='thrift://hadoop101:9083',
    'clients'='5',
    'property-version'='1',
    'warehouse'='hdfs:///user/hive/warehouse/hive_iceberg_catalog'
);
use catalog hive_iceberg_catalog;
CREATE TABLE IF NOT EXISTS ods_base.IcebergTest2_XXZH (
    `id` bigint,
    `data` STRING,
    primary key (id) not enforced
)with(
    'write.metadata.delete-after-commit.enabled'='true',
    'write.metadata.previous-versions-max'='5',
    'format-version'='2'
 );
 

 
 insert into  hive_iceberg_catalog.ods_base.IcebergTest2_XXZH select * from default_catalog.default_database.KafkaTableTest2_XXZH;

2.2 Manufacturing data

[root@hadoop101 conf]#  kafka-console-producer.sh --broker-list  hadoop101:9092,hadoop102:9092,hadoop103:9092  --topic test2_xxzh
>1,abc
[2022-07-22 14:55:51,643] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : {
    
    test2_xxzh=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
>2,bb
>3,cc
>4,dd
>5,ee
>3,cccc
>6,666
>4,ddddd
>

2.3 Running Results

spark-sql (default)> select *  from ods_base.IcebergTest2_XXZH;
22/07/22 15:12:28 WARN HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
id      data
3       cc
4       ddddd
5       ee
3       cccc
6       666
4       dd
Time taken: 0.405 seconds, Fetched 6 row(s)

The running result of flink-sql:
insert image description here

2.4 Running conclusion

Iceberg cannot be updated according to the pk declared by kafka. iceberg is written in append mode.

3. Change to enforce and report an error

3.1 Test code

Change the pk of the iceberg table to enforced and run again


Flink SQL> CREATE TABLE IF NOT EXISTS KafkaTableTest3_XXZH (
>     `id` bigint,
>     `data` STRING
> ) WITH (
>     'connector' = 'kafka',
>     'topic' = 'test2_xxzh',
>     'properties.bootstrap.servers' = 'hadoop101:9092,hadoop102:9092,hadoop103:9092',
>     'properties.group.id' = 'testGroup',
>     'scan.startup.mode' = 'latest-offset',
>     'csv.ignore-parse-errors'='true',
>     'format' = 'csv'
> );
> 
[INFO] Execute statement succeed.

Flink SQL> CREATE CATALOG hive_iceberg_catalog WITH (
>     'type'='iceberg',
>     'catalog-type'='hive',
>     'uri'='thrift://hadoop101:9083',
>     'clients'='5',
>     'property-version'='1',
>     'warehouse'='hdfs:///user/hive/warehouse/hive_iceberg_catalog'
> );
[INFO] Execute statement succeed.

Flink SQL> use catalog hive_iceberg_catalog;
[INFO] Execute statement succeed.

Flink SQL> CREATE TABLE IF NOT EXISTS ods_base.IcebergTest3_XXZH (
>     `id` bigint,
>     `data` STRING,
>     primary key (id) enforced
> )with(
>     'write.metadata.delete-after-commit.enabled'='true',
>     'write.metadata.previous-versions-max'='5',
>     'format-version'='2'
>  );
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.api.ValidationException: Flink doesn't support ENFORCED mode for PRIMARY KEY constraint. ENFORCED/NOT ENFORCED  controls if the constraint checks are performed on the incoming/outgoing data. Flink does not own the data therefore the only supported mode is the NOT ENFORCED mode

报错信息：
org.apache.flink.table.api.ValidationException: Flink doesn’t support ENFORCED mode for PRIMARY KEY constraint. ENFORCED/NOT ENFORCED controls if the constraint checks are performed on the incoming/outgoing data. Flink does not own the data therefore the only supported mode is the NOT ENFORCED mode

Flink does not own these data itself, so the only supported modes are non-strong values.

Conclusion: iceberg did not update the data according to pk

4. 'write.upsert.enabled' = 'true', set this parameter to realize the upsert function

CREATE TABLE IF NOT EXISTS ods_base.IcebergTest4_XXZH (
    `id` bigint,
    `data` STRING,
    primary key (id) not enforced
)with(
  'format-version' = '2',
  'write.upsert.enabled' = 'true',
  'write.distribution-mode'='hash',
  'write.metadata.delete-after-commit.enabled'='true',
  'write.metadata.previous-versions-max'='3'
 );

[root@hadoop102 module]#  kafka-console-producer.sh --topic test4_xxzh --broker-list hadoop101:9092,hadoop102:9092,hadoop103:9092
>2,222
>3,333  （这里暂停，去spark观察）
>2,bbbb
>3,cccc
>4,444
>5,555

Initialization data

spark-sql (default)> select * from  ods_base.IcebergTest4_XXZH ;
id      data
2       222
3       333

Update data, the contents of id=2 and 3 are all updated

spark-sql (default)> select * from  ods_base.IcebergTest4_XXZH ;
22/07/26 19:24:58 WARN HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
id      data
2       bbbb
4       444
5       555
3       cccc

Summarize

1. Iceberg writes additional data by default to the data flowing in from Kafka.
2. By setting the 'write.upsert.enabled' = 'true parameter to the iceberg table, the upsert mode can be realized

Practice data lake iceberg The thirty-seventh lesson kakfa writes the enfource and not enfource test of the icberg table of iceberg