1 Small file optimization
Influencing factors: parallelism, rollback time, file size, checkpoint, compaction
1 compaction file generation rules
1 flink-sql writes parquet files to
Summary of rules:
1) At checkpoint, generate a compact file (immediately readable), and delete the uncompact (unreadable) file at the next checkpoint;
2) Only files in a single checkpoint can be merged, and files generated by different checkpoints will not be merged;
3) After the partition ends two checkpoints, the file is available
Reference link:
Combining Small Files in Flink Combat - Programmer Sought