18/08/22 03:15:58 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[hdfs://jari.com:8020/user/its/warehouse/its_recommend/its_f300v/video_vec_1y.txt], targetPath=/user/client/warehouse/recommend/its_f300v/video_vec_1y.txt, targetPathExists=false, preserveRawXattrs=false}
18/08/22 03:15:59 INFO Configuration.deprecation: io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
18/08/22 03:15:59 INFO Configuration.deprecation: io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
18/08/22 03:15:59 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
18/08/22 03:15:59 INFO mapreduce.JobSubmitter: number of splits:1
18/08/22 03:15:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1476778451795_10217231
18/08/22 03:16:00 INFO impl.YarnClientImpl: Submitted application application_1476778451795_10217231
18/08/22 03:16:00 INFO mapreduce.Job: The url to track the job: http://rm2.tvhadoop.sohuno.com:8008/proxy/application_1476778451795_10217231/
18/08/22 03:16:00 INFO tools.DistCp: DistCp job-id: job_1476778451795_10217231
18/08/22 03:16:00 INFO mapreduce.Job: Running job: job_1476778451795_10217231
18/08/22 03:25:24 INFO mapreduce.Job: Job job_1476778451795_10217231 running in uber mode : false
18/08/22 03:25:24 INFO mapreduce.Job: map 0% reduce 0%
18/08/22 03:25:36 INFO mapreduce.Job: map 100% reduce 0%
18/08/22 03:32:26 INFO mapreduce.Job: Task Id : attempt_1476778451795_10217231_m_000000_0, Status : FAILED
Error: java.io.IOException: File copy failed: hdfs://jari.com:8020/user/its/warehouse/its_recommend/its_f300v/video_vec_1y.txt --> hdfs://jaricluster/user/client/warehouse/recommend/its_f300v/video_vec_1y.txt
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:285)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:253)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://jari.com:8020/user/its/warehouse/its_recommend/its_f300v/video_vec_1y.txt to hdfs://jaricluster/user/client/warehouse/recommend/its_f300v/video_vec_1y.txt
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:281)
... 10 more
Caused by: java.io.IOException: Check-sum mismatch between hdfs://jari.com:8020/user/its/warehouse/its_recommend/its_f300v/video_vec_1y.txt and hdfs://jaricluster/user/client/warehouse/recommend/its_f300v/.distcp.tmp.attempt_1476778451795_10217231_m_000000_0.
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:210)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
解决办法
hadoop distcp -Dmapred.speculative.execution=false -Ddfs.replication=3 -bandwidth 10 -m 25 -update -skipcrccheck $remote_hdfs/user/its/。。。 /user/client/warehouse/recommend/。。。