sparkbwa compilation failed
Problems encountered in the case of failure in the internal network
1. The maven mirror source cannot be obtained, and the external network agent is configured to solve it
. 2. The file needs to be modified
(1) Define the JAVA_HOME environment variable in the /src/main/native/Makefile.common file.
(2) The Makefile of the BWA version contains the flag -fPIC. You need to add this option to the end of the CFLAGS variable in the BWA Makefile. Taking into account bwa-0.7.15, the original Makefile contains:
CFLAGS= -g -Wall -Wno-unused-function -O2
After the change, it should be:
CFLAGS= -g -Wall -Wno-unused-function -O2 -fPIC
Problems encountered in the failure of the external network
(1) Error one: jre rather than jdk. Solution: After deleting the original jdk and reconfiguring it, the source environment variable is resolved.
(2) Error two: Maven cannot access the mirror address, and some packages cannot be downloaded. Solution: Replace the maven Ali source mirror, as follows:
<mirror>
<id>nexus-aliyun</id>
<mirrorOf>central</mirrorOf>
<name>Nexus aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>
(3) Error three: Compilation failure, the error is as follows:
Through searching and positioning, it is found that maven has failed to compile the package during the packaging process. In short, it is caused by the compilation failure of the bwa package. Reviewing the compilation process of the bwa package requires installation, gnu , Gcc and other series of software, and the content of the error reported here is: error: zlib.h:no such file or directory , and the apt-get install zlib1g-dev command cannot be used on centos, you can only use yum, and proceed directly to yum -y install gcc will report an error. Therefore, first upgrade gcc, and then download zlib.h with yum.
Note: The environment uses centos7 , the specific operations are as follows:
296 gcc -v
297 yum -y install gcc
298 yum install zlib-devel
Finally, recompile sparkbwa of mvn, and successfully passed the following:
(base) [root@localhost SparkBWA]# mvn package
[INFO] Scanning for projects...
[INFO]
[INFO] --------------------< com.github.sparkbwa:SparkBWA >--------------------
[INFO] Building SparkBWA 0.2
[INFO] --------------------------------[ jar ]---------------------------------
[WARNING] The POM for commons-codec:commons-codec:jar:1.15-SNAPSHOT is missing, no dependency information available
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ SparkBWA ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ SparkBWA ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- exec-maven-plugin:1.5.0:exec (makeBwa) @ SparkBWA ---
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/apache/maven/maven-core/3.0/maven-core-3.0.jar (527 kB at 305 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/sonatype/aether/aether-api/1.7/aether-api-1.7.jar
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/sonatype/aether/aether-spi/1.7/aether-spi-1.7.jar (14 kB at 7.3 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/sonatype/aether/aether-util/1.7/aether-util-1.7.jar
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/apache/maven/maven-model-builder/3.0/maven-model-builder-3.0.jar (148 kB at 77 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/codehaus/plexus/plexus-interpolation/1.14/plexus-interpolation-1.14.jar
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/apache/maven/maven-aether-provider/3.0/maven-aether-provider-3.0.jar (51 kB at 26 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/codehaus/plexus/plexus-classworlds/2.2.3/plexus-classworlds-2.2.3.jar
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/sonatype/aether/aether-impl/1.7/aether-impl-1.7.jar (106 kB at 54 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/apache/maven/maven-compat/3.0/maven-compat-3.0.jar
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/sonatype/aether/aether-api/1.7/aether-api-1.7.jar (74 kB at 35 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/apache/maven/wagon/wagon-provider-api/1.0-beta-6/wagon-provider-api-1.0-beta-6.jar
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/sonatype/aether/aether-util/1.7/aether-util-1.7.jar (108 kB at 44 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/apache/maven/maven-artifact/3.0/maven-artifact-3.0.jar
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/codehaus/plexus/plexus-classworlds/2.2.3/plexus-classworlds-2.2.3.jar (46 kB at 19 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/codehaus/plexus/plexus-utils/3.0.22/plexus-utils-3.0.22.jar
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/codehaus/plexus/plexus-interpolation/1.14/plexus-interpolation-1.14.jar (61 kB at 25 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/ow2/asm/asm/5.0.2/asm-5.0.2.jar
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/apache/maven/wagon/wagon-provider-api/1.0-beta-6/wagon-provider-api-1.0-beta-6.jar (53 kB at 21 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/ow2/asm/asm-commons/5.0.2/asm-commons-5.0.2.jar
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/apache/maven/maven-compat/3.0/maven-compat-3.0.jar (285 kB at 104 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/ow2/asm/asm-tree/5.0.2/asm-tree-5.0.2.jar
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/apache/maven/maven-artifact/3.0/maven-artifact-3.0.jar (52 kB at 18 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/jdom/jdom/1.1/jdom-1.1.jar
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/ow2/asm/asm/5.0.2/asm-5.0.2.jar (53 kB at 17 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/apache/maven/shared/maven-dependency-tree/2.2/maven-dependency-tree-2.2.jar
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/ow2/asm/asm-tree/5.0.2/asm-tree-5.0.2.jar (29 kB at 9.2 kB/s)
Downloaded from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/org/ow2/asm/asm-commons/5.0.2/asm-commons-5.0.2.jar (42 kB at 13 kB/s)
Downloading from alimaven: http://maven.aliyun.com/nexus/content/repositories/central/commons-io/commons-io/2.2/commons-io-2.2.jar
[INFO] Excluding org.glassfish.jersey.containers:jersey-container-servlet:jar:2.22.2 from the shaded jar.
[INFO] Excluding org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.22.2 from the shaded jar.
[INFO] Excluding io.netty:netty-all:jar:4.1.47.Final from the shaded jar.
[INFO] Excluding io.netty:netty:jar:3.9.9.Final from the shaded jar.
[INFO] Excluding com.clearspring.analytics:stream:jar:2.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.5 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-jvm:jar:3.1.5 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-json:jar:3.1.5 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-graphite:jar:3.1.5 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.core:jackson-databind:jar:2.6.7.3 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.7.1 from the shaded jar.
[INFO] Excluding org.scala-lang:scala-reflect:jar:2.11.8 from the shaded jar.
[INFO] Excluding com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.7.9 from the shaded jar.
[INFO] Excluding org.apache.ivy:ivy:jar:2.4.0 from the shaded jar.
[INFO] Excluding oro:oro:jar:2.0.8 from the shaded jar.
[INFO] Excluding net.razorvine:pyrolite:jar:4.13 from the shaded jar.
[INFO] Excluding net.sf.py4j:py4j:jar:0.10.7 from the shaded jar.
[INFO] Excluding org.apache.spark:spark-tags_2.11:jar:2.4.6 from the shaded jar.
[INFO] Excluding org.apache.commons:commons-crypto:jar:1.0.0 from the shaded jar.
[INFO] Excluding org.spark-project.spark:unused:jar:1.0.0 from the shaded jar.
[INFO] Including cz.adamh.utils:native-utils:jar:1.0-SNAPSHOT in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing /home/SparkBWA/target/SparkBWA-0.2.jar with /home/SparkBWA/target/SparkBWA-0.2-shaded.jar
[INFO] Dependency-reduced POM written at: /home/SparkBWA/dependency-reduced-pom.xml
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:02 min
[INFO] Finished at: 2020-09-10T10:32:46-04:00
[INFO] ------------------------------------------------------------------------
(base) [root@localhost SparkBWA]# cd target/
(base) [root@localhost target]# ls
classes maven-archiver maven-status original-SparkBWA-0.2.jar SparkBWA-0.2.jar
Finally solved perfectly!