The code I wrote needs to connect to elasticsearch, go to es to query and return the results. At this time, when I use maven to package and run it, the following error will be reported.
Before using spark1.6, this situation has not occurred. Check the error and find that the ES client cannot be created. The reason for viewing the material online is guava18 and the package and
Spark comes with conflict. Then resolve the conflict between Elasticsearch and spark's jar package through maven-shade-plugin
1. First, refer to the es official website blog: https://www.elastic.co/blog/to-shade-or-not-to-shade, create a new maven project and configure the following
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>my.elasticsearch</groupId> <artifactId>es-shaded</artifactId> <version>1.0-SNAPSHOT</version> <properties> <elasticsearch.version>2.3.3</elasticsearch.version> </properties> <dependencies> <dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> <version>${elasticsearch.version}</version> </dependency> <dependency> <groupId>org.elasticsearch.plugin</groupId> <artifactId>shield</artifactId> <version>${elasticsearch.version}</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.4.3</version> <configuration> <createDependencyReducedPom>false</createDependencyReducedPom> </configuration> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <relocations> <relocation> <pattern>com.google.guava</pattern> <shadedPattern>my.elasticsearch.guava</shadedPattern> </relocation> <relocation> <pattern>org.joda</pattern> <shadedPattern>my.elasticsearch.joda</shadedPattern> </relocation> <relocation> <pattern>com.google.common</pattern> <shadedPattern>my.elasticsearch.common</shadedPattern> </relocation> <relocation> <pattern>com.google.thirdparty</pattern> <shadedPattern>my.elasticsearch.thirdparty</shadedPattern> </relocation> </relocations> <transformers> <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer" /> </transformers> </configuration> </execution> </executions> </plugin> </plugins> </build> <repositories> <repository> <id>elasticsearch-releases</id> <url>http://maven.elasticsearch.org/releases</url> <releases> <enabled>true</enabled> <updatePolicy>daily</updatePolicy> </releases> <snapshots> <enabled>false</enabled> </snapshots> </repository> </repositories> </project>
2. After the pom file is configured, run:
mvn clean install
as shown in the figure
Then if you see build success, it means that the packaging is successful, and the new dependency package will be in the maven repository folder we configured earlier. As shown in the figure, it appears in the maven environment I installed.
3. Go back to the pom file of the original integration project. Introduce the jar package: (note that the es2.3.3 jar package referenced before needs to be excluded here, otherwise maven will enter the 2.3.3 jar package, causing conflicts)
<dependency> <groupId>my.elasticsearch</groupId> <artifactId>es-shaded</artifactId> <version>1.0-SNAPSHOT</version> <exclusions> <exclusion> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> </exclusion> </exclusions> </dependency>
Then package it with maven, test it and you will find that the conflict problem is solved.