Spark source dependencies
we know Spark application running load dependent on three places:
SystemClasspath - Spark provide installation time dependencies
dependencies Spark-submit --jars submission
". Spark {driver / executor} .extraClassPath = someJar" Spark-submit --config submitted dependencies
Spark dependencies priority
of the three priority order what is it?
I passed the test found that priority is as follows:
1. spark-submit --config "spark. {Driver / executor} .extraClassPath = someJar" filed dependencies
2. SystemClasspath - Spark provided when the installation dependencies
Dependencies 3. spark-submit --jars submitted
Three purposes:
Spark {Driver / Executor} .extraClassPath highest priority, but generally have different versions of the same jar used when necessary to resolve the conflict;.
SystemClasspath system installed package, use the default priority packet environment, this is more stable and secure.
When the spark-submit --jars not required packages in the default spark environment, upload their own offer.
Summary:
A) when we submitted a spark2 program, the system does not submit the package --jars;
B) When the system and we need to have different versions of the package, using spark. {Driver / executor} .extraClassPath specified.