Shaded/Repackaged jar as a dependency

Ján Srniček :

We have a situation where we need one application to be able to connect to to two versions of kafka(0.7.2 and 0.10.0+) and act as a router. I'm trying to omit using two runtimes here as we need this to be stupid fast, so want to prevent additional serialization/deserialization when sending data between runtimes.

To do this, i've tryied to repackage the old kafka driver from package kafka to old.kafka like so:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>kafka-router</artifactId>
        <groupId>org.deer</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>old-kafka</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <kafka.version>0.7.2</kafka.version>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>3.1.1</version>
                <executions>
                    <execution>
                        <id>unpack</id>
                        <phase>compile</phase>
                        <goals>
                            <goal>unpack</goal>
                        </goals>
                        <configuration>
                            <artifactItems>
                                <artifactItem>
                                    <groupId>org.apache.kafka</groupId>
                                    <artifactId>kafka_2.9.2</artifactId>
                                    <version>${kafka.version}</version>
                                    <type>jar</type>
                                    <overWrite>false</overWrite>
                                    <outputDirectory>${project.build.directory}/classes</outputDirectory>
                                    <includes>**/*.class,**/*.xml</includes>
                                </artifactItem>
                            </artifactItems>
                            <includes>**/*.java</includes>
                            <overWriteReleases>false</overWriteReleases>
                            <overWriteSnapshots>true</overWriteSnapshots>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.2</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <relocations>
                                <relocation>
                                    <pattern>kafka.</pattern>
                                    <shadedPattern>old.kafka.</shadedPattern>
                                </relocation>
                            </relocations>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

I'm using dependency plugin to unpack kafka classes to target/classes & shade plugin to repackage them. The reason for this is that the final jar should act as if it is a kafka driver jar(it has no other transitive dependencies, therefore it can't cause some mismatch using kafka instead of old.kafka. But that's not really the point here, just trying to prevent out-of-topic questions.

The main problem here is that when i look at the jar that has been installed to .m2, it looks correct(having the old.kafka package):

But when i try to use this jar as dependency like so ...

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>kafka-router</artifactId>
        <groupId>org.deer</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>router-app</artifactId>
    <version>1.0-SNAPSHOT</version>


    <dependencies>
        <dependency>
            <groupId>org.deer</groupId>
            <artifactId>old-kafka</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>
    </dependencies>
</project>

... and reference it in a class like so ...

package org.deer.test;

import old.kafka.producer.ProducerData;

public class TwoKafkaDriversExample {

    public static void main(String[] args) {
        new ProducerData();
    }
}

... the import it self is not working. I have a suspicion that the shaded jar is missing something maven-related, but haven't noticed anything. Another possible though is that shade plugin or asm doesn't like the bytecode that scala classes are generating.

Ján Srniček :

Ok, so i've been able to figure this out. The import error is a problem with intelij, for some reason it doesn't see the repackaged classes. But the maven does, with using correct constructor and adding scala-lang dependency(it was complaining about missing Seq class) i was able to build this.

Full example uploaded on github - https://github.com/Marssmart/kafka-router