Series Article Directory

提示：写完文章后，目录可以自动生成，如何生成可参考右边的帮助文档

Article directory

Series Article Directory
foreword
1. Test ideas
2. Case test
Summarize

foreword

insert image description here

In the previous lesson, when we talked about incremental update, the younger brother’s boss asked, for incremental implementation, should we read the incremental data or re-read the historical data? Crit, according to my understanding, is to read increments. . . , the boss does not agree with the understanding! Well, let’s test it, so with this article, readers feel my bleeding heart. May I ask if you have such an experience. . .

1. Test ideas

Continuing from the kafka case above, the producer sends data to kafka, and flink-sql sends the kafka data to iceberg. This section continues to consume from iceberg. Test:
select * from hive_iceberg_catalog.ods_base.IcebergSink_XXZH / + OPTIONS('streaming'= 'true', 'monitor-interval'='1s') /
When the incremental operator of this sql is triggered, is it full or incremental

2. Case test

1. Code

 
     public static void main(String[] args) throws Exception {
    
    
        //TODO 1.准备环境
        //1.1流处理环境
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);
        //1.2 表执行环境
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);


        String sql2 = "CREATE CATALOG hive_iceberg_catalog WITH (\n" +
                "    'type'='iceberg',\n" +
                "    'catalog-type'='hive',\n" +
                "    'uri'='thrift://hadoop101:9083',\n" +
                "    'clients'='5',\n" +
                "    'property-version'='1',\n" +
                "    'warehouse'='hdfs:///user/hive/warehouse/hive_iceberg_catalog'\n" +
                ")";
        String sql3 = "use catalog hive_iceberg_catalog";
        String sql4 = "CREATE TABLE IF NOT EXISTS ods_base.IcebergSink_XXZH (\n" +
                "    `log` STRING,\n" +
                "\t`dt` INT\n" +
                ")with(\n" +
                "    'write.metadata.delete-after-commit.enabled'='true',\n" +
                "    'write.metadata.previous-versions-max'='5',\n" +
                "    'format-version'='2'\n" +
                " )";
      String sql6 = "select * from  hive_iceberg_catalog.ods_base.IcebergSink_XXZH /*+ OPTIONS('streaming'='true', 'monitor-interval'='1s')*/ ";

        tableEnv.executeSql(sql2);
        tableEnv.executeSql(sql3);
        tableEnv.executeSql(sql4);
        tableEnv.executeSql(sql6).print();



        //TODO 6.执行任务
        env.execute();

    }

2. Start the program

The console outputs historical data

| +I |                              e |    20230101 |
| +I |                              e |    20230101 |
| +I |                              e |    20230101 |
| +I |                             >e |    20230101 |
| +I |                              e |     2023010 |
| +I |                            abc |    20240101 |
| +I |                           abcd |    20240101 |
| +I |                           abcd |    20240101 |
| +I |                              ; |      (NULL) |
| +I |                             ； |      (NULL) |
| +I |                              ; |      (NULL) |
| +I |                             ； |      (NULL) |
22/06/16 21:09:21 INFO compress.CodecPool: Got brand-new decompressor [.gz]
| +I |                              1 |    20220601 |
22/06/16 21:09:21 INFO compress.CodecPool: Got brand-new decompressor [.gz]
| +I |                              2 |    20220601 |

3. Start the producer and observe the result

[root@hadoop101 lib]# kafka-console-producer.sh --broker-list  hadoop101:9092,hadoop102:9092,hadoop103:9092  --topic test_xxzh

Enter one:

2,20220606

Observe the console and find that there is one more increment:

22/06/16 21:15:48 INFO compress.CodecPool: Got brand-new decompressor [.gz]
| +I |                              2 |    20220606 |

Summarize

select * from hive_iceberg_catalog.ods_base.IcebergSink_XXZH /+ OPTIONS(‘streaming’=‘true’, ‘monitor-interval’=‘1s’)/

This grammatical confirmation is incremental processing of data, rather than re-reading all data once

Practice data lake iceberg Lesson 35 is based on the stream-batch integration architecture of data lake icerberg--test whether the incremental read is full or only incremental