When we work in the company, we need to package and upload locally developed jobs to the cluster for running. Due to the excessive dependencies on the project, the jar will be relatively large, so each upload is a waste of time. In order to save time, two methods can be used:
1: Upload all dependencies to the cluster at one time, and then use -cp to add the class library to the classpath each time the program is started.
2: When we upload the job jar to the cluster and find that there is a bug, we need to modify it locally and submit it again, but this is time-consuming. At this time, we can use the following two commands to quickly update the jar:
jar -xvf job.jar jar -cvf newJob.jar ./*
The first line of code is to decompress the current jar, then we can selectively replace the class files we need to update and then use the second line of code to package all the source files. You can re-test after packaging.