Solve the jar package conflict between Elasticsearch and spark through maven-shade-plugin

Recently, due to the upgrade of the company's spark version and the use of spark2.1.1 version, I personally feel that spark2.1.1 is really easier to use.
The code I wrote needs to connect to elasticsearch, go to es to query and return the results. At this time, when I use maven to package and run it, the following error will be reported.



Before using spark1.6, this situation has not occurred. Check the error and find that the ES client cannot be created. The reason for viewing the material online is guava18 and the package and

Spark comes with conflict. Then resolve the conflict between Elasticsearch and spark's jar package through maven-shade-plugin

1. First, refer to the es official website blog: https://www.elastic.co/blog/to-shade-or-not-to-shade, create a new maven project and configure the following

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>my.elasticsearch</groupId>
    <artifactId>es-shaded</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <elasticsearch.version>2.3.3</elasticsearch.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>${elasticsearch.version}</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.plugin</groupId>
            <artifactId>shield</artifactId>
            <version>${elasticsearch.version}</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
                <configuration>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <relocations>
                                <relocation>
                                    <pattern>com.google.guava</pattern>
                                    <shadedPattern>my.elasticsearch.guava</shadedPattern>
                                </relocation>
                                <relocation>
                                    <pattern>org.joda</pattern>
                                    <shadedPattern>my.elasticsearch.joda</shadedPattern>
                                </relocation>
                                <relocation>
                                    <pattern>com.google.common</pattern>
                                    <shadedPattern>my.elasticsearch.common</shadedPattern>
                                </relocation>
                                <relocation>
                                    <pattern>com.google.thirdparty</pattern>
                                    <shadedPattern>my.elasticsearch.thirdparty</shadedPattern>
                                </relocation>
                            </relocations>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer" />
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

    <repositories>
        <repository>
            <id>elasticsearch-releases</id>
            <url>http://maven.elasticsearch.org/releases</url>
            <releases>
                <enabled>true</enabled>
                <updatePolicy>daily</updatePolicy>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
    </repositories>
</project>

The above configuration is actually migrating four potentially conflicting jar packages such as org.joda through the maven-shade-plugin plug-in and re-creating a jar package, so that the jar package can be used when the jar package is imported. Not using external dependencies.


2. After the pom file is configured, run:
mvn clean install
as shown in the figure

Then if you see build success, it means that the packaging is successful, and the new dependency package will be in the maven repository folder we configured earlier. As shown in the figure, it appears in the maven environment I installed.

3. Go back to the pom file of the original integration project. Introduce the jar package: (note that the es2.3.3 jar package referenced before needs to be excluded here, otherwise maven will enter the 2.3.3 jar package, causing conflicts)

<dependency>
    <groupId>my.elasticsearch</groupId>
    <artifactId>es-shaded</artifactId>
    <version>1.0-SNAPSHOT</version>
    <exclusions>
        <exclusion>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
        </exclusion>
    </exclusions>
</dependency>

Then package it with maven, test it and you will find that the conflict problem is solved.