实践数据湖iceberg 第三十九课 清理快照前后数据文件变化分析

系列文章目录

实践数据湖iceberg 第一课 入门
实践数据湖iceberg 第二课 iceberg基于hadoop的底层数据格式
实践数据湖iceberg 第三课 在sqlclient中，以sql方式从kafka读数据到iceberg
实践数据湖iceberg 第四课 在sqlclient中，以sql方式从kafka读数据到iceberg（升级版本到flink1.12.7）
实践数据湖iceberg 第五课 hive catalog特点
实践数据湖iceberg 第六课 从kafka写入到iceberg失败问题 解决
实践数据湖iceberg 第七课 实时写入到iceberg
实践数据湖iceberg 第八课 hive与iceberg集成
实践数据湖iceberg 第九课 合并小文件
实践数据湖iceberg 第十课 快照删除
实践数据湖iceberg 第十一课 测试分区表完整流程(造数、建表、合并、删快照)
实践数据湖iceberg 第十二课 catalog是什么
实践数据湖iceberg 第十三课 metadata比数据文件大很多倍的问题
实践数据湖iceberg 第十四课 元数据合并(解决元数据随时间增加而元数据膨胀的问题)
实践数据湖iceberg 第十五课 spark安装与集成iceberg(jersey包冲突)
实践数据湖iceberg 第十六课 通过spark3打开iceberg的认知之门
实践数据湖iceberg 第十七课 hadoop2.7,spark3 on yarn运行iceberg配置
实践数据湖iceberg 第十八课 多种客户端与iceberg交互启动命令(常用命令)
实践数据湖iceberg 第十九课 flink count iceberg，无结果问题
实践数据湖iceberg 第二十课 flink + iceberg CDC场景(版本问题，测试失败)
实践数据湖iceberg 第二十一课 flink1.13.5 + iceberg0.131 CDC(测试成功INSERT,变更操作失败)
实践数据湖iceberg 第二十二课 flink1.13.5 + iceberg0.131 CDC(CRUD测试成功)
实践数据湖iceberg 第二十三课 flink-sql从checkpoint重启
实践数据湖iceberg 第二十四课 iceberg元数据详细解析
实践数据湖iceberg 第二十五课 后台运行flink sql 增删改的效果
实践数据湖iceberg 第二十六课 checkpoint设置方法
实践数据湖iceberg 第二十七课 flink cdc 测试程序故障重启：能从上次checkpoint点继续工作
实践数据湖iceberg 第二十八课 把公有仓库上不存在的包部署到本地仓库
实践数据湖iceberg 第二十九课 如何优雅高效获取flink的jobId
实践数据湖iceberg 第三十课 mysql->iceberg，不同客户端有时区问题
实践数据湖iceberg 第三十一课 使用github的flink-streaming-platform-web工具，管理flink任务流，测试cdc重启场景
实践数据湖iceberg 第三十二课 DDL语句通过hive catalog持久化方法
实践数据湖iceberg 第三十三课 升级flink到1.14，自带functioin支持json函数
实践数据湖iceberg 第三十四课 基于数据湖icerberg的流批一体架构-流架构测试
实践数据湖iceberg 第三十五课 基于数据湖icerberg的流批一体架构–测试增量读是读全量还是仅读增量
实践数据湖iceberg 第三十六课 基于数据湖icerberg的流批一体架构–update mysql select from icberg语法是增量更新测试
实践数据湖iceberg 第三十七课 kakfa写入iceberg的 icberg表的 enfource ,not enfource测试
实践数据湖iceberg 第三十八课 spark sql， Procedures语法进行数据治理（小文件合并，清理快照）
实践数据湖iceberg 第三十九课 清理快照前后数据文件变化分析
实践数据湖iceberg 更多的内容目录

---

---

前言

分析hive_iceberg_catalog.system.expire_snapshots()命令前后，数据湖表底层发生的变更

---

一、清理快照

1.1 清理前，现状

续上一课的案例，
上次写了两条数据，每分钟checkpoint一次

spark-sql (default)> select * from icebergtest7_xxzh;
data    dt
1       20220801
2       20220802
Time taken: 0.147 seconds, Fetched 2 row(s)

表的文件数：

[root@hadoop103 conf]# hadoop fs -count  hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/*
           3            2               1345 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data
           1         1553           13349847 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata

1.2执行合并命令

写一个明年的日期（现在在20220805），保留10个快照：

spark-sql (default)>  CALL spark_catalog.system.expire_snapshots('ods_base.IcebergTest7_XXZH', TIMESTAMP '2023-08-06 00:00:00.000', 10);
22/08/05 15:20:45 WARN HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
deleted_data_files_count        deleted_position_delete_files_count     deleted_equality_delete_files_count     deleted_manifest_files_count    deleted_manifest_lists_count
0       0       0       0       1536
Time taken: 25.491 seconds, Fetched 1 row(s)

发现清理了1536个文件：

观察结果，每几秒执行一次count观察

[root@hadoop103 conf]# hadoop fs -count  hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/*
           3            2               1345 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data
           1         1553           13349847 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata
[root@hadoop103 conf]# hadoop fs -count  hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/*
           3            2               1345 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data
           1         1553           13349847 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata
[root@hadoop103 conf]# hadoop fs -count  hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/*
           3            2               1345 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data
           1         1553           13349847 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata
[root@hadoop103 conf]# hadoop fs -count  hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/*
           3            2               1345 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data
           1         1226           11934332 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata
[root@hadoop103 conf]# hadoop fs -count  hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/*
           3            2               1345 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data
           1           17            6700166 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata
[root@hadoop103 conf]# hadoop fs -count  hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/*
           3            2               1345 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data
           1           17            6700166 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata
[root@hadoop103 conf]# hadoop fs -count  hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/*
           3            2               1345 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data
           1           17            6700166 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata

观察结果：

Found 18 items
-rw-r--r--   2 root supergroup    1326958 2022-08-05 15:18 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01662-a3b98f34-6350-4aaf-97c8-5bf5bc322cbb.metadata.json
-rw-r--r--   2 root supergroup    1327818 2022-08-05 15:18 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01663-918cf80c-eee0-404e-ba63-7e4ff7dbcb1a.metadata.json
-rw-r--r--   2 root supergroup    1328678 2022-08-05 15:19 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01664-1dbe7398-ee20-4016-85e0-3f020f868a36.metadata.json
-rw-r--r--   2 root supergroup    1329538 2022-08-05 15:20 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01665-ddbdd7a1-ce62-469c-9082-955eb82288d5.metadata.json
-rw-r--r--   2 root supergroup      10978 2022-08-05 15:20 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01666-4541c2c0-0479-45ca-98f0-fa047047f7d5.metadata.json
-rw-r--r--   2 root supergroup      11833 2022-08-05 15:21 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01667-d99a28ca-8564-43d3-97a6-9d6ffaa65ba5.metadata.json
-rw-r--r--   2 root supergroup       6798 2022-08-04 17:11 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/fe4c8846-b07c-42e4-98c2-68aed69fbfd0-m0.avro
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:13 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-1667142442712269329-1-94a3be8f-1fbb-48c7-87f8-43548cc16a61.avro
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:16 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-1963762773888773433-1-d49babf6-122b-4af9-a43c-efd41c252666.avro
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:13 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-2153830365703656208-1-48ca082d-1e41-4bfd-b8f3-db8cd1b450b4.avro
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:14 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-2187623164859521720-1-f35e216f-f68a-472c-a90d-d70ab84aa7d3.avro
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:17 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-3609130078797535708-1-d2d4674d-1670-4aa0-aca0-7eb51a56a783.avro
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:18 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-3863382103427831766-1-8a43e521-f75c-48cf-99a5-af05695e2237.avro
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:19 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-4767088375083646307-1-e7c384f5-73dc-4644-9042-837d46fae36d.avro
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:15 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-6684373263600938900-1-74780fe1-2e66-4f53-8ce7-b797e223a6c9.avro
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:20 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-7833168760795469341-1-d06fbdfa-b5ca-4eff-a66f-97b053039b3c.avro
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:18 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-8339965443495738233-1-61613f35-7ecb-456a-9e25-cd7be6dfe091.avro
-rw-r--r--   2 root supergroup       4329 2022-08-05 15:21 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-935317679288657184-1-e5a00540-1c34-47f3-9e1b-a847bc334051.avro

1.3 只保留一个快照

发现只有一个manifest文件，删到只有一个快照看看

扫描二维码关注公众号，回复： 14793304 查看本文章

spark-sql (default)>  CALL spark_catalog.system.expire_snapshots('ods_base.IcebergTest7_XXZH', TIMESTAMP '2823-08-06 00:00:00.000', 1);
deleted_data_files_count        deleted_position_delete_files_count     deleted_equality_delete_files_count     deleted_manifest_files_count    deleted_manifest_lists_count
0       0       0       0       11
Time taken: 3.878 seconds, Fetched 1 row(s)

结果如下：

Found 9 items
-rw-r--r--   2 root supergroup    1329538 2022-08-05 15:20 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01665-ddbdd7a1-ce62-469c-9082-955eb82288d5.metadata.json
-rw-r--r--   2 root supergroup      10978 2022-08-05 15:20 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01666-4541c2c0-0479-45ca-98f0-fa047047f7d5.metadata.json
-rw-r--r--   2 root supergroup      11833 2022-08-05 15:21 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01667-d99a28ca-8564-43d3-97a6-9d6ffaa65ba5.metadata.json
-rw-r--r--   2 root supergroup      12694 2022-08-05 15:22 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01668-8e04d8c8-38cc-4b1b-81bc-eb0f9fbcfa5f.metadata.json
-rw-r--r--   2 root supergroup       3237 2022-08-05 15:22 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01669-6206ddc5-830a-4962-bc1a-209c991d6ac7.metadata.json
-rw-r--r--   2 root supergroup       4097 2022-08-05 15:23 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01670-92917778-5f30-4ead-942d-0f05915cb398.metadata.json
-rw-r--r--   2 root supergroup       6798 2022-08-04 17:11 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/fe4c8846-b07c-42e4-98c2-68aed69fbfd0-m0.avro
-rw-r--r--   2 root supergroup       4329 2022-08-05 15:22 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-1911393555202333427-1-fe0fb043-eef8-4755-a34c-91e3d8d94f9a.avro
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:23 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-3064118549879781557-1-5e763c87-757a-46fa-8646-afff2476a51e.avro

二、如果有多个manifest，快照能删到多少个？

2.1 增加数据

增加多条数据：

[root@hadoop101 ~]#  kafka-console-producer.sh --broker-list  hadoop101:9092,hadoop102:9092,hadoop103:9092  --topic test2_xxzh
>22,20220802
>3,20220803
>4,20220804
>5,20220805
>6,20220806
>7,20220807

snapshot在增加：

[root@hadoop103 conf]#  hadoop fs -ls -R  hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/
drwxrwxrwx   - root supergroup          0 2022-08-05 15:44 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data
drwxrwxrwx   - root supergroup          0 2022-08-04 16:12 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220801
-rw-r--r--   2 root supergroup        672 2022-08-04 16:12 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220801/00001-0-989c0c01-b69d-4c66-8c74-7a1a4be08f71-00001.parquet
drwxrwxrwx   - root supergroup          0 2022-08-05 15:38 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220802
-rw-r--r--   2 root supergroup        680 2022-08-05 15:38 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220802/00001-0-52c0f221-3908-447d-9441-ed0be045c3ca-00001.parquet
-rw-r--r--   2 root supergroup        673 2022-08-04 16:46 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220802/00001-0-989c0c01-b69d-4c66-8c74-7a1a4be08f71-00002.parquet
drwxrwxrwx   - root supergroup          0 2022-08-05 15:40 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220803
-rw-r--r--   2 root supergroup        673 2022-08-05 15:40 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220803/00000-0-0b872a75-1956-49e4-9093-e4e418eace05-00001.parquet
drwxrwxrwx   - root supergroup          0 2022-08-05 15:40 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220804
-rw-r--r--   2 root supergroup        673 2022-08-05 15:40 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220804/00001-0-52c0f221-3908-447d-9441-ed0be045c3ca-00002.parquet
drwxrwxrwx   - root supergroup          0 2022-08-05 15:40 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220805
-rw-r--r--   2 root supergroup        672 2022-08-05 15:40 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220805/00001-0-52c0f221-3908-447d-9441-ed0be045c3ca-00003.parquet
drwxrwxrwx   - root supergroup          0 2022-08-05 15:43 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220806
-rw-r--r--   2 root supergroup        673 2022-08-05 15:43 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220806/00001-0-52c0f221-3908-447d-9441-ed0be045c3ca-00004.parquet
drwxrwxrwx   - root supergroup          0 2022-08-05 15:44 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220807
-rw-r--r--   2 root supergroup        673 2022-08-05 15:44 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/data/dt=20220807/00000-0-0b872a75-1956-49e4-9093-e4e418eace05-00002.parquet
drwxrwxrwx   - root supergroup          0 2022-08-05 15:44 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata
-rw-r--r--   2 root supergroup      25977 2022-08-05 15:41 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01695-ff9dfa85-4a33-4557-bd1d-cdc230fa605f.metadata.json
-rw-r--r--   2 root supergroup      26837 2022-08-05 15:42 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01696-f26649f4-877d-4656-a90f-bc72a0f1735d.metadata.json
-rw-r--r--   2 root supergroup      27697 2022-08-05 15:43 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01697-3bad18ac-7b26-4496-9ac0-114cfa49cbaa.metadata.json
-rw-r--r--   2 root supergroup      28652 2022-08-05 15:43 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01698-ef6f5e15-a2ec-44cc-ab4c-9297c0ca3321.metadata.json
-rw-r--r--   2 root supergroup      29512 2022-08-05 15:44 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01699-cfb0ddd6-5c85-499c-866c-9d200eafe965.metadata.json
-rw-r--r--   2 root supergroup      30467 2022-08-05 15:44 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01700-ed75dcf1-c58b-444e-b8b5-b6085370c535.metadata.json
-rw-r--r--   2 root supergroup       6754 2022-08-05 15:40 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/1df9c902-caf9-4703-8c22-1d6a9f7de154-m0.avro
-rw-r--r--   2 root supergroup       6756 2022-08-05 15:40 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/765c23a8-266b-4545-94c1-a0f446a5775e-m0.avro
-rw-r--r--   2 root supergroup       6755 2022-08-05 15:38 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/7a047525-4ecd-46d9-a4ec-9b7321323cfc-m0.avro
-rw-r--r--   2 root supergroup       6755 2022-08-05 15:40 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/7b90e539-a1d9-4f47-83df-76fdb723de45-m0.avro
-rw-r--r--   2 root supergroup       6755 2022-08-05 15:43 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/91501f41-1db7-4307-a2c0-c8041bb936eb-m0.avro
-rw-r--r--   2 root supergroup       6753 2022-08-05 15:44 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/ee7f5d4c-2c3f-4d87-9dab-a89b703dd2e1-m0.avro
-rw-r--r--   2 root supergroup       6798 2022-08-04 17:11 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/fe4c8846-b07c-42e4-98c2-68aed69fbfd0-m0.avro
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:27 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-1035563423599163544-1-2b706b4b-0b08-4f51-9216-1eb64e888188.avro
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:32 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-1737338957084956478-1-be16b42a-2eb1-4bd3-8728-de6f83e67d66.avro
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:28 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-1782562210015627548-1-0a830678-95ed-4cf9-979f-f8dd2f965776.avro
-rw-r--r--   2 root supergroup       4329 2022-08-05 15:22 4521-b5b9-9b854dce0664.avro
。。。删掉中间很多。。。。
-rw-r--r--   2 root supergroup       4330 2022-08-05 15:33 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-8912083785901975773-1-f6a58ef7-03c5-4792-a58c-7bbbffad7b79.avro

2.2 执行删除命令

删除到只有1个shapshot

spark-sql (default)> 
                   >   CALL spark_catalog.system.expire_snapshots('ods_base.IcebergTest7_XXZH', TIMESTAMP '2823-08-06 00:00:00.000', 1);
22/08/05 15:47:24 WARN HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
deleted_data_files_count        deleted_position_delete_files_count     deleted_equality_delete_files_count     deleted_manifest_files_count    deleted_manifest_lists_count
0       0       0       0       34
Time taken: 17.441 seconds, Fetched 1 row(s)

结果如下：发现只剩2个snap 文件， m0文件对应每次写入。

Found 15 items
-rw-r--r--   2 root supergroup      30467 2022-08-05 15:44 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01700-ed75dcf1-c58b-444e-b8b5-b6085370c535.metadata.json
-rw-r--r--   2 root supergroup      31327 2022-08-05 15:45 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01701-117cd2ed-3480-4842-a7dd-4c90eaab83ab.metadata.json
-rw-r--r--   2 root supergroup      32182 2022-08-05 15:46 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01702-0ef6b60a-43aa-4f3a-a636-34298644ebce.metadata.json
-rw-r--r--   2 root supergroup      33043 2022-08-05 15:47 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01703-00fb5e1d-29af-4731-adbe-7f1805c20cd1.metadata.json
-rw-r--r--   2 root supergroup       3237 2022-08-05 15:47 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01704-fb9cc6cb-1492-45b6-8dcb-218c3d56d08b.metadata.json
-rw-r--r--   2 root supergroup       4097 2022-08-05 15:47 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/01705-c6d98e57-d748-460b-93a5-fdfb0d557d67.metadata.json
-rw-r--r--   2 root supergroup       6754 2022-08-05 15:40 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/1df9c902-caf9-4703-8c22-1d6a9f7de154-m0.avro
-rw-r--r--   2 root supergroup       6756 2022-08-05 15:40 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/765c23a8-266b-4545-94c1-a0f446a5775e-m0.avro
-rw-r--r--   2 root supergroup       6755 2022-08-05 15:38 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/7a047525-4ecd-46d9-a4ec-9b7321323cfc-m0.avro
-rw-r--r--   2 root supergroup       6755 2022-08-05 15:40 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/7b90e539-a1d9-4f47-83df-76fdb723de45-m0.avro
-rw-r--r--   2 root supergroup       6755 2022-08-05 15:43 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/91501f41-1db7-4307-a2c0-c8041bb936eb-m0.avro
-rw-r--r--   2 root supergroup       6753 2022-08-05 15:44 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/ee7f5d4c-2c3f-4d87-9dab-a89b703dd2e1-m0.avro
-rw-r--r--   2 root supergroup       6798 2022-08-04 17:11 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/fe4c8846-b07c-42e4-98c2-68aed69fbfd0-m0.avro
-rw-r--r--   2 root supergroup       4649 2022-08-05 15:47 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-3511522569100173178-1-59ecc32e-2aa7-48c6-bbb9-f5aa4b31b442.avro
-rw-r--r--   2 root supergroup       4650 2022-08-05 15:47 hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/snap-4956000582044241255-1-4fa34632-810d-48e7-a470-d5f9c6735f7d.avro

删除快照后，数据是完整的

spark-sql (default)> select * from ods_base.IcebergTest7_XXZH;
data    dt
1       20220801
2       20220802
4       20220804
7       20220807
6       20220806
3       20220803
5       20220805
22      20220802
Time taken: 1.649 seconds, Fetched 8 row(s)

三、元数据特点（查看snap内容）

3.1 snap的内容，包含7个m0文件

[root@hadoop103 snap]# java -jar /opt/software/avro-tools-1.11.0.jar  tojson --pretty  snap-3511522569100173178-1-59ecc32e-2aa7-48c6-bbb9-f5aa4b31b442.avro 
22/08/05 15:58:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
{
    
    
  "manifest_path" : "hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/ee7f5d4c-2c3f-4d87-9dab-a89b703dd2e1-m0.avro",
  "manifest_length" : 6753,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 1695,
  "min_sequence_number" : 1695,
  "added_snapshot_id" : 2845024222990467689,
  "added_data_files_count" : 1,
  "existing_data_files_count" : 0,
  "deleted_data_files_count" : 0,
  "added_rows_count" : 1,
  "existing_rows_count" : 0,
  "deleted_rows_count" : 0,
  "partitions" : {
    
    
    "array" : [ {
    
    
      "contains_null" : false,
      "contains_nan" : {
    
    
        "boolean" : false
      },
      "lower_bound" : {
    
    
        "bytes" : "20220807"
      },
      "upper_bound" : {
    
    
        "bytes" : "20220807"
      }
    } ]
  }
}
{
    
    
  "manifest_path" : "hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/91501f41-1db7-4307-a2c0-c8041bb936eb-m0.avro",
  "manifest_length" : 6755,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 1693,
  "min_sequence_number" : 1693,
  "added_snapshot_id" : 3476529947294323623,
  "added_data_files_count" : 1,
  "existing_data_files_count" : 0,
  "deleted_data_files_count" : 0,
  "added_rows_count" : 1,
  "existing_rows_count" : 0,
  "deleted_rows_count" : 0,
  "partitions" : {
    
    
    "array" : [ {
    
    
      "contains_null" : false,
      "contains_nan" : {
    
    
        "boolean" : false
      },
      "lower_bound" : {
    
    
        "bytes" : "20220806"
      },
      "upper_bound" : {
    
    
        "bytes" : "20220806"
      }
    } ]
  }
}
{
    
    
  "manifest_path" : "hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/765c23a8-266b-4545-94c1-a0f446a5775e-m0.avro",
  "manifest_length" : 6756,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 1689,
  "min_sequence_number" : 1689,
  "added_snapshot_id" : 5462232017147497616,
  "added_data_files_count" : 1,
  "existing_data_files_count" : 0,
  "deleted_data_files_count" : 0,
  "added_rows_count" : 1,
  "existing_rows_count" : 0,
  "deleted_rows_count" : 0,
  "partitions" : {
    
    
    "array" : [ {
    
    
      "contains_null" : false,
      "contains_nan" : {
    
    
        "boolean" : false
      },
      "lower_bound" : {
    
    
        "bytes" : "20220805"
      },
      "upper_bound" : {
    
    
        "bytes" : "20220805"
      }
    } ]
  }
}
{
    
    
  "manifest_path" : "hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/7b90e539-a1d9-4f47-83df-76fdb723de45-m0.avro",
  "manifest_length" : 6755,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 1688,
  "min_sequence_number" : 1688,
  "added_snapshot_id" : 3246455649213713509,
  "added_data_files_count" : 1,
  "existing_data_files_count" : 0,
  "deleted_data_files_count" : 0,
  "added_rows_count" : 1,
  "existing_rows_count" : 0,
  "deleted_rows_count" : 0,
  "partitions" : {
    
    
    "array" : [ {
    
    
      "contains_null" : false,
      "contains_nan" : {
    
    
        "boolean" : false
      },
      "lower_bound" : {
    
    
        "bytes" : "20220804"
      },
      "upper_bound" : {
    
    
        "bytes" : "20220804"
      }
    } ]
  }
}
{
    
    
  "manifest_path" : "hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/1df9c902-caf9-4703-8c22-1d6a9f7de154-m0.avro",
  "manifest_length" : 6754,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 1687,
  "min_sequence_number" : 1687,
  "added_snapshot_id" : 4917712002051492927,
  "added_data_files_count" : 1,
  "existing_data_files_count" : 0,
  "deleted_data_files_count" : 0,
  "added_rows_count" : 1,
  "existing_rows_count" : 0,
  "deleted_rows_count" : 0,
  "partitions" : {
    
    
    "array" : [ {
    
    
      "contains_null" : false,
      "contains_nan" : {
    
    
        "boolean" : false
      },
      "lower_bound" : {
    
    
        "bytes" : "20220803"
      },
      "upper_bound" : {
    
    
        "bytes" : "20220803"
      }
    } ]
  }
}
{
    
    
  "manifest_path" : "hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/7a047525-4ecd-46d9-a4ec-9b7321323cfc-m0.avro",
  "manifest_length" : 6755,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 1684,
  "min_sequence_number" : 1684,
  "added_snapshot_id" : 3096920793835932503,
  "added_data_files_count" : 1,
  "existing_data_files_count" : 0,
  "deleted_data_files_count" : 0,
  "added_rows_count" : 1,
  "existing_rows_count" : 0,
  "deleted_rows_count" : 0,
  "partitions" : {
    
    
    "array" : [ {
    
    
      "contains_null" : false,
      "contains_nan" : {
    
    
        "boolean" : false
      },
      "lower_bound" : {
    
    
        "bytes" : "20220802"
      },
      "upper_bound" : {
    
    
        "bytes" : "20220802"
      }
    } ]
  }
}
{
    
    
  "manifest_path" : "hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/fe4c8846-b07c-42e4-98c2-68aed69fbfd0-m0.avro",
  "manifest_length" : 6798,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 84,
  "min_sequence_number" : 12,
  "added_snapshot_id" : 8562765270417336551,
  "added_data_files_count" : 0,
  "existing_data_files_count" : 2,
  "deleted_data_files_count" : 0,
  "added_rows_count" : 0,
  "existing_rows_count" : 2,
  "deleted_rows_count" : 0,
  "partitions" : {
    
    
    "array" : [ {
    
    
      "contains_null" : false,
      "contains_nan" : {
    
    
        "boolean" : false
      },
      "lower_bound" : {
    
    
        "bytes" : "20220801"
      },
      "upper_bound" : {
    
    
        "bytes" : "20220802"
      }
    } ]
  }
}

snap2也包含7个m0文件

[root@hadoop103 snap]#  java -jar /opt/software/avro-tools-1.11.0.jar  tojson --pretty  snap-4956000582044241255-1-4fa34632-810d-48e7-a470-d5f9c6735f7d.avro 22/08/05 16:15:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
{  "manifest_path" : "hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/ee7f5d4c-2c3f-4d87-9dab-a89b703dd2e1-m0.avro",
  "manifest_length" : 6753,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 1695,
  "min_sequence_number" : 1695,
  "added_snapshot_id" : 2845024222990467689,
  "added_data_files_count" : 1,
  "existing_data_files_count" : 0,
  "deleted_data_files_count" : 0,
  "added_rows_count" : 1,
  "existing_rows_count" : 0,
  "deleted_rows_count" : 0,
  "partitions" : {
    "array" : [ {
      "contains_null" : false,
      "contains_nan" : {
        "boolean" : false
      },
      "lower_bound" : {
        "bytes" : "20220807"
      },
      "upper_bound" : {
        "bytes" : "20220807"
      }
    } ]
  }
}
{
  "manifest_path" : "hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/91501f41-1db7-4307-a2c0-c8041bb936eb-m0.avro",
  "manifest_length" : 6755,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 1693,
  "min_sequence_number" : 1693,
  "added_snapshot_id" : 3476529947294323623,
  "added_data_files_count" : 1,
  "existing_data_files_count" : 0,
  "deleted_data_files_count" : 0,
  "added_rows_count" : 1,
  "existing_rows_count" : 0,
  "deleted_rows_count" : 0,
  "partitions" : {
    "array" : [ {
      "contains_null" : false,
      "contains_nan" : {
        "boolean" : false
      },
      "lower_bound" : {
        "bytes" : "20220806"
      },
      "upper_bound" : {
        "bytes" : "20220806"
      }
    } ]
  }
}
{
  "manifest_path" : "hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/765c23a8-266b-4545-94c1-a0f446a5775e-m0.avro",
  "manifest_length" : 6756,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 1689,
  "min_sequence_number" : 1689,
  "added_snapshot_id" : 5462232017147497616,
  "added_data_files_count" : 1,
  "existing_data_files_count" : 0,
  "deleted_data_files_count" : 0,
  "added_rows_count" : 1,
  "existing_rows_count" : 0,
  "deleted_rows_count" : 0,
  "partitions" : {
    "array" : [ {
      "contains_null" : false,
      "contains_nan" : {
        "boolean" : false
      },
      "lower_bound" : {
        "bytes" : "20220805"
      },
      "upper_bound" : {
        "bytes" : "20220805"
      }
    } ]
  }
}
{
  "manifest_path" : "hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/7b90e539-a1d9-4f47-83df-76fdb723de45-m0.avro",
  "manifest_length" : 6755,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 1688,
  "min_sequence_number" : 1688,
  "added_snapshot_id" : 3246455649213713509,
  "added_data_files_count" : 1,
  "existing_data_files_count" : 0,
  "deleted_data_files_count" : 0,
  "added_rows_count" : 1,
  "existing_rows_count" : 0,
  "deleted_rows_count" : 0,
  "partitions" : {
    "array" : [ {
      "contains_null" : false,
      "contains_nan" : {
        "boolean" : false
      },
      "lower_bound" : {
        "bytes" : "20220804"
      },
      "upper_bound" : {
        "bytes" : "20220804"
      }
    } ]
  }
}
{
  "manifest_path" : "hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/1df9c902-caf9-4703-8c22-1d6a9f7de154-m0.avro",
  "manifest_length" : 6754,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 1687,
  "min_sequence_number" : 1687,
  "added_snapshot_id" : 4917712002051492927,
  "added_data_files_count" : 1,
  "existing_data_files_count" : 0,
  "deleted_data_files_count" : 0,
  "added_rows_count" : 1,
  "existing_rows_count" : 0,
  "deleted_rows_count" : 0,
  "partitions" : {
    "array" : [ {
      "contains_null" : false,
      "contains_nan" : {
        "boolean" : false
      },
      "lower_bound" : {
        "bytes" : "20220803"
      },
      "upper_bound" : {
        "bytes" : "20220803"
      }
    } ]
  }
}
{
  "manifest_path" : "hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/7a047525-4ecd-46d9-a4ec-9b7321323cfc-m0.avro",
  "manifest_length" : 6755,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 1684,
  "min_sequence_number" : 1684,
  "added_snapshot_id" : 3096920793835932503,
  "added_data_files_count" : 1,
  "existing_data_files_count" : 0,
  "deleted_data_files_count" : 0,
  "added_rows_count" : 1,
  "existing_rows_count" : 0,
  "deleted_rows_count" : 0,
  "partitions" : {
    "array" : [ {
      "contains_null" : false,
      "contains_nan" : {
        "boolean" : false
      },
      "lower_bound" : {
        "bytes" : "20220802"
      },
      "upper_bound" : {
        "bytes" : "20220802"
      }
    } ]
  }
}
{
  "manifest_path" : "hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/fe4c8846-b07c-42e4-98c2-68aed69fbfd0-m0.avro",
  "manifest_length" : 6798,
  "partition_spec_id" : 0,
  "content" : 0,
  "sequence_number" : 84,
  "min_sequence_number" : 12,
  "added_snapshot_id" : 8562765270417336551,
  "added_data_files_count" : 0,
  "existing_data_files_count" : 2,
  "deleted_data_files_count" : 0,
  "added_rows_count" : 0,
  "existing_rows_count" : 2,
  "deleted_rows_count" : 0,
  "partitions" : {
    "array" : [ {
      "contains_null" : false,
      "contains_nan" : {
        "boolean" : false
      },
      "lower_bound" : {
        "bytes" : "20220801"
      },
      "upper_bound" : {
        "bytes" : "20220802"
      }
    } ]
  }
}

第三个snap （一分钟后自动生成的）

[root@hadoop103 snap]# java -jar /opt/software/avro-tools-1.11.0.jar tojson --pretty snap-5188266964869746455-1-abd5575f-12fc-43fd-bc9e-5cb69a0e03df.avro
22/08/05 16:19:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
{
“manifest_path” : “hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/ee7f5d4c-2c3f-4d87-9dab-a89b703dd2e1-m0.avro”,
“manifest_length” : 6753,
“partition_spec_id” : 0,
“content” : 0,
“sequence_number” : 1695,
“min_sequence_number” : 1695,
“added_snapshot_id” : 2845024222990467689,
“added_data_files_count” : 1,
“existing_data_files_count” : 0,
“deleted_data_files_count” : 0,
“added_rows_count” : 1,
“existing_rows_count” : 0,
“deleted_rows_count” : 0,
“partitions” : {
“array” : [ {
“contains_null” : false,
“contains_nan” : {
“boolean” : false
},
“lower_bound” : {
“bytes” : “20220807” }, “upper_bound” : {
“bytes” : “20220807”
}
} ]
}
}
{
“manifest_path” : “hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/91501f41-1db7-4307-a2c0-c8041bb936eb-m0.avro”,
“manifest_length” : 6755,
“partition_spec_id” : 0,
“content” : 0,
“sequence_number” : 1693,
“min_sequence_number” : 1693,
“added_snapshot_id” : 3476529947294323623,
“added_data_files_count” : 1,
“existing_data_files_count” : 0,
“deleted_data_files_count” : 0,
“added_rows_count” : 1,
“existing_rows_count” : 0,
“deleted_rows_count” : 0,
“partitions” : {
“array” : [ {
“contains_null” : false,
“contains_nan” : {
“boolean” : false
},
“lower_bound” : {
“bytes” : “20220806”
},
“upper_bound” : {
“bytes” : “20220806”
}
} ]
}
}
{
“manifest_path” : “hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/765c23a8-266b-4545-94c1-a0f446a5775e-m0.avro”,
“manifest_length” : 6756,
“partition_spec_id” : 0,
“content” : 0,
“sequence_number” : 1689,
“min_sequence_number” : 1689,
“added_snapshot_id” : 5462232017147497616,
“added_data_files_count” : 1,
“existing_data_files_count” : 0,
“deleted_data_files_count” : 0,
“added_rows_count” : 1,
“existing_rows_count” : 0,
“deleted_rows_count” : 0,
“partitions” : {
“array” : [ {
“contains_null” : false,
“contains_nan” : {
“boolean” : false
},
“lower_bound” : {
“bytes” : “20220805”
},
“upper_bound” : {
“bytes” : “20220805”
}
} ]
}
}
{
“manifest_path” : “hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/7b90e539-a1d9-4f47-83df-76fdb723de45-m0.avro”,
“manifest_length” : 6755,
“partition_spec_id” : 0,
“content” : 0,
“sequence_number” : 1688,
“min_sequence_number” : 1688,
“added_snapshot_id” : 3246455649213713509,
“added_data_files_count” : 1,
“existing_data_files_count” : 0,
“deleted_data_files_count” : 0,
“added_rows_count” : 1,
“existing_rows_count” : 0,
“deleted_rows_count” : 0,
“partitions” : {
“array” : [ {
“contains_null” : false,
“contains_nan” : {
“boolean” : false
},
“lower_bound” : {
“bytes” : “20220804”
},
“upper_bound” : {
“bytes” : “20220804”
}
} ]
}
}
{
“manifest_path” : “hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/1df9c902-caf9-4703-8c22-1d6a9f7de154-m0.avro”,
“manifest_length” : 6754,
“partition_spec_id” : 0,
“content” : 0,
“sequence_number” : 1687,
“min_sequence_number” : 1687,
“added_snapshot_id” : 4917712002051492927,
“added_data_files_count” : 1,
“existing_data_files_count” : 0,
“deleted_data_files_count” : 0,
“added_rows_count” : 1,
“existing_rows_count” : 0,
“deleted_rows_count” : 0,
“partitions” : {
“array” : [ {
“contains_null” : false,
“contains_nan” : {
“boolean” : false
},
“lower_bound” : {
“bytes” : “20220803”
},
“upper_bound” : {
“bytes” : “20220803”
}
} ]
}
}
{
“manifest_path” : “hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/7a047525-4ecd-46d9-a4ec-9b7321323cfc-m0.avro”,
“manifest_length” : 6755,
“partition_spec_id” : 0,
“content” : 0,
“sequence_number” : 1684,
“min_sequence_number” : 1684,
“added_snapshot_id” : 3096920793835932503,
“added_data_files_count” : 1,
“existing_data_files_count” : 0,
“deleted_data_files_count” : 0,
“added_rows_count” : 1,
“existing_rows_count” : 0,
“deleted_rows_count” : 0,
“partitions” : {
“array” : [ {
“contains_null” : false,
“contains_nan” : {
“boolean” : false
},
“lower_bound” : {
“bytes” : “20220802”
},
“upper_bound” : {
“bytes” : “20220802”
}
} ]
}
}
{
“manifest_path” : “hdfs://ns/user/hive/warehouse/hive_iceberg_catalog/ods_base.db/IcebergTest7_XXZH/metadata/fe4c8846-b07c-42e4-98c2-68aed69fbfd0-m0.avro”,
“manifest_length” : 6798,
“partition_spec_id” : 0,
“content” : 0,
“sequence_number” : 84,
“min_sequence_number” : 12,
“added_snapshot_id” : 8562765270417336551,
“added_data_files_count” : 0,
“existing_data_files_count” : 2,
“deleted_data_files_count” : 0,
“added_rows_count” : 0,
“existing_rows_count” : 2,
“deleted_rows_count” : 0,
“partitions” : {
“array” : [ {
“contains_null” : false,
“contains_nan” : {
“boolean” : false
},
“lower_bound” : {
“bytes” : “20220801”
},
“upper_bound” : {
“bytes” : “20220802”
}
} ]
}
}

3.2 总结snap文件的指向

snap指向多个m0文件，m0指向数据文件，从而得到所有的文件信息。

四、m0文件合并后，后续的m0文件从合并后的文件开始？

下一课继续。。。。

---

总结

例如：以上就是今天要讲的内容，本文仅仅简单介绍了pandas的使用，而pandas提供了大量能使我们快速便捷地处理数据的函数和方法。