Caused by: java.io.IOException: Checksum mismatch between hdfs://10.48.0.101/ucd-prod-vdp-usdp/user/hive/warehouse/ods.db/ods_dgs_dcs_tth_if_di/dt=20230716/000001_0 and hdfs://ucd-test-vdp-usdp/ucd-test-vdp-usdp/user/hive/warehouse/ods_prod.db/.distcp.tmp.attempt_1689578105872_1498_m_000017_2.1690257654635. Source and target differ in block-size.
Use -pb to preserve block-sizes during copy. You can choose file-level checksum validation via -Ddfs.checksum.combine.mode=COMPOSITE_CRC when block-sizes or filesystems are different. Or you can skip checksum-checks altogether with -skipcrccheck.
The distcp command is a commonly used data copy command between large data clusters. Sometimes the command execution fails due to the inconsistency of block sizes between different clusters. The error is as follows
Source and target differ in block-size. Use -pb to preserve block-sizes during copy
1
Cause Analysis
Distcp does not record the original block size when copying files by default, so the verification fails when the block.size of the original file is not 128M. -pb parameter.
1. HDFS sets the block size when writing, the default is 128M, and the files written by some components or business programs may not be 128M, such as 8M.
<name>dfs.blocksize</name>
<value>134217728</value>
图1 某些组件或者业务程序写入的文件大小
2.distcp 从源集群读文件后写入新集群，默认是使用的MapReduce任务中的dfs.blocksize，默认128M。
3.在distcp写完文件后，会基于块的物理大小做校验，因为该文件在新旧集群中block.size不一致，因此拆分大小不一致，导致校验失败。
如以上文件，在旧集群是17.9/8MB = 3个block， 在新集群 17.9/128M = 1个block. 因此实际在磁盘的物理大小因分割而导致校验失败。
解决办法
distcp时，在地址前增加-pb参数。该参数作用为distcp时候保留block大小，确保新集群写入文件blocksize和老集群一致。
distcp时保留block大小