colegas buen día. He desarrollado una aplicación basada en sparkLauncher que ejecutar un jar ejecutable donde son 5 operaciones. Cada operación en función de la variable específica. Tengo un racimo principal hadoop spark2.3.0-hadoop2.6.5. Funciona bien en ella. Una parte de mi código de trabajo:
private void runSparkJob(String pathToJar, final LocalDate startDate, final LocalDate endDate) {
if (executionInProgress.get()) {
LOGGER.warn("Execution already in progress");
return;
}
Process sparkProcess = null;
try {
LOGGER.info("Create SparkLauncher. SparkHome: [{}]. JarPath: [{}].", sparkHome, vmJarPath);
executionInProgress.set(true);
sparkProcess = new SparkLauncher()
.setAppName(activeOperationProfile)
.setSparkHome(sparkHome) //sparkHome folder on main cluster
.setAppResource(pathToJar) // jar with 5 operation
.setConf(SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS,
String.format("-Drunner.operation-profile=%1$s -Doperation.startDate=%2$s -Doperation.endDate=%3$s", activeOperationProfile, startDate,endDate))
.setConf(SparkLauncher.DRIVER_MEMORY, "12G")
.redirectToLog(LOGGER.getName())
.setMaster("yarn")
.launch();
sparkProcess.waitFor();
int exitCode = sparkProcess.exitValue();
if (exitCode != 0) {
throw new RuntimeException("Illegal exit code. Expected: [0]. Actual: [" + exitCode + "]");
}
} catch (IOException | InterruptedException e) {
LOGGER.error("Error occurred while running SparkApplication.", e);
throw new RuntimeException(e);
} finally {
if (sparkProcess != null && sparkProcess.isAlive()) {
LOGGER.warn("Process still alive. Try to kill");
sparkProcess.destroy();
}
executionInProgress.set(false);
}
}
He comenzado un recipiente estibador, donde una chispa 2.3.0-hadoop6 descargado. Este contenedor necesario para los probadores. He cambiado maestro para .setMaster ( "local"), puse nuevo perfil con rutas de acceso a sparkHome, jarsWithOpertations y tarro envasados sin sombra (tratado con sombra y no funciona para mí). Cuando traté de ejecutar mi aplicación sparkLaunch tengo ahora una excepción:
08/06/2018 14: 47: 53.150 INFO [nmmbrSparkBaseOperationsRunner.runSparkJob] 105: Crear SparkLauncher. SparkHome: [/opt/bigtv/spark/spark-2.3.0-bin-hadoop2.6]. JarPath: [/opt/bigtv/bin/multirating-bigdata-operations-MASTER-SNAPSHOT.jar]. 08/06/2018 14: 47: 54,905 INFO [oaspark.launcher.OutputRedirector.redirect] 63: 08/06/2018 14:47:54 WARN NativeCodeLoader: 62 - No se puede cargar la biblioteca-hadoop nativo para la plataforma ... el uso de clases de Java incorporado, en su caso 06/08/2018 14: 47: 57,042 INFO [oaspark.launcher.OutputRedirector.redirect] 63: 08/06/2018 14:47:57 ERROR SpringApplication: 842 - ejecutar la aplicación falló 2018-08 -06 14: 47: 57,043 INFO [oaspark.launcher.OutputRedirector.redirect] 63:. java.lang.NoSuchMethodError: org.yaml.snakeyaml.Yaml (LORG / yaml / snakeyaml / constructor / BaseConstructor; LORG / yaml / snakeyaml / Represente / Represente; LORG / yaml / snakeyaml / DumperOptions; LORG / yaml / snakeyaml / cargador Opciones ; LORG / yaml / snakeyaml / resolver / resolver;) V08/06/2018 14: 47: 57 043 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.boot.env.OriginTrackedYamlLoader.createYaml (OriginTrackedYamlLoader.java:70) 08/06/2018 14:47: 57043 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.beans.factory.config.YamlProcessor.process (YamlProcessor.java:139) 08/06/2018 14: 47: 57 044 INFO [oaspark.launcher. OutputRedirector.redirect] 63: en org.springframework.boot.env.OriginTrackedYamlLoader.load (OriginTrackedYamlLoader.java:75) 08/06/2018 14: 47: 57 044 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org. springframework.boot.env.YamlPropertySourceLoader.load (YamlPropertySourceLoader.java:50) 08/06/2018 14: 47: 57 044 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.boot.context.config.ConfigFileApplicationListener $ Loader.loadDocuments (ConfigFileApplicationListener.java:547) 08/06/2018 14: 47: 57,044 INFO [oaspark.launcher.OutputRedirector.redirect] 63: a org.springframework.boot.context.config.ConfigFileApplicationListener $ Loader.load (ConfigFileApplicationListener.java:517) 08/06/2018 14: 47: 57.045 INFO [oaspark.launcher.OutputRedirector.redirect] 63: a org.springframework.boot.context.config.ConfigFileApplicationListener $ Loader.loadForFileExtension (ConfigFileApplicationListener.java: 496) 08/06/2018 14: 47: 57.045 INFO [oaspark.launcher.OutputRedirector.redirect] 63: a org.springframework.boot.context.config.ConfigFileApplicationListener $ Loader.load (ConfigFileApplicationListener.java:464) 2018-08 -06 14: 47: 57.045 INFO [oaspark.launcher.OutputRedirector.redirect] 63: a org.springframework.boot.context.config.ConfigFileApplicationListener $ Loader.lambda $ null $ 6 (ConfigFileApplicationListener.java:446) 08/06/2018 14: 47: 57,046 INFO [oaspark.launcher.OutputRedirector.redirect] 63: a java.lang.Iterable.forEach (Iterable.java: 75) 08/06/2018 14: 47: 57 046 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.boot.context.config.ConfigFileApplicationListener $ Loader.lambda $ carga $ 7 (ConfigFileApplicationListener.java:445) 08/06/2018 14: 47: 57 046 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en java.lang.Iterable.forEach (Iterable.java:75) 08/06/2018 14: 47: 57 046 INFO [oaspark .launcher.OutputRedirector.redirect] 63: a org.springframework.boot.context.config.ConfigFileApplicationListener $ Loader.load (ConfigFileApplicationListener.java:442) 08/06/2018 14: 47: 57,046 INFO [oaspark.launcher.OutputRedirector. redirect] 63: en org.springframework.boot.context.config.ConfigFileApplicationListener $ Loader.load (ConfigFileApplicationListener.java:330) 08/06/2018 14: 47: 57,047 INFO [oaspark.launcher.OutputRedirector.redirect] 63: a org.springframework.boot.context .config.ConfigFileApplicationListener.addPropertySources (ConfigFileApplicationListener.java:212) 06/08/2018 14: 47: 57 047 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.boot.context.config.ConfigFileApplicationListener.postProcessEnvironment ( ConfigFileApplicationListener.java:195) 08/06/2018 14: 47: 57 047 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEnvironmentPreparedEvent (ConfigFileApplicationListener.java:182) 2018 -08-06 14: 47: 57,047 INFO [oaspark.launcher.OutputRedirector.redirect] 63:en org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEvent (ConfigFileApplicationListener.java:168) 08/06/2018 14: 47: 57 048 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.context. event.SimpleApplicationEventMulticaster.doInvokeListener (SimpleApplicationEventMulticaster.java:172) 08/06/2018 14: 47: 57 048 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener (SimpleApplicationEventMulticaster.java : 165) 08/06/2018 14: 47: 57 048 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent (SimpleApplicationEventMulticaster.java:139) 08/06/2018 14 : 47: 57.048 INFO [oaspark.launcher.OutputRedirector.redirect] 63: a las org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent (SimpleApplicationEventMulticaster.java:127) 08/06/2018 14: 47: 57 049 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.boot.context.event.EventPublishingRunListener .environmentPrepared (EventPublishingRunListener.java:74) 08/06/2018 14: 47: 57 049 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.boot.SpringApplicationRunListeners.environmentPrepared (SpringApplicationRunListeners.java:54) 2018- 08-06 14: 47: 57 049 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.boot.SpringApplication.prepareEnvironment (SpringApplication.java:358) 08/06/2018 14: 47: 57 049 INFO [oaspark .launcher.OutputRedirector.redirect] 63: en org.springframework.boot.SpringApplication.run (SpringApplication.java:317) 08/06/2018 14: 47: 57 050 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.boot.SpringApplication.run (SpringApplication.java:1255) 08/06/2018 14:47: 57050 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.boot.SpringApplication.run (SpringApplication.java:1243) 08/06/2018 14: 47: 57 050 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en net.mediascope.multirating.bigdata.operations.OperationRunner.main (OperationRunner.java:21) 08/06/2018 14: 47: 57 050 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en sun.reflect. NativeMethodAccessorImpl.invoke0 (nativo Method) 06/08/2018 14: 47: 57 050 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) 08/06/2018 14 : 47: 57.051 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) 08/06/2018 14: 47: 57 051 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en java.lang.reflect. Method.invoke (Method.java:498) 08/06/2018 14: 47: 57 051 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.boot.loader.MainMethodRunner.run (MainMethodRunner.java:48 ) 08/06/2018 14: 47: 57 051 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.boot.loader.Launcher.launch (Launcher.java:87) 08/06/2018 14:47 : 57052 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.springframework.boot.loader.Launcher.launch (Launcher.java:50) 08/06/2018 14: 47: 57 052 INFO [oaspark.launcher.OutputRedirector .redirect] 63: en org.springframework.boot.loader.JarLauncher.principal (JarLauncher.java:51) 08/06/2018 14: 47: 57 052 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en sun.reflect.NativeMethodAccessorImpl.invoke0 (Nativo Method) 08/06/2018 14:47 : 57052 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) 08/06/2018 14: 47: 57 053 INFO [oaspark.launcher.OutputRedirector.redirect] 63 : en sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) 08/06/2018 14: 47: 57 053 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en java.lang.reflect.Method.invoke (Método .java: 498) 08/06/2018 14: 47: 57 053 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.apache.spark.deploy.JavaMainApplication.start (SparkApplication.scala: 52) 2018-08- 06 14: 47: 57,053 INFO [oaspark.launcher.OutputRedirector.redirect] 63: a org.apache.spark.deploy.SparkSubmit $ .org $ Apache $ chispa $ desplegar $ $$ SparkSubmit runMain (SparkSubmit.scala: 879) 08/06/2018 14: 47: 57,054 INFO [oaspark .launcher.OutputRedirector.redirect] 63: a org.apache.spark.deploy.SparkSubmit $ .doRunMain $ 1 (SparkSubmit.scala: 197) 08/06/2018 14: 47: 57,054 INFO [oaspark.launcher.OutputRedirector.redirect] 63: en org.apache.spark.deploy.SparkSubmit $ .submit (SparkSubmit.scala: 227) 08/06/2018 14: 47: 57,054 INFO [oaspark.launcher.OutputRedirector.redirect] 63: a org.apache.spark .deploy.SparkSubmit $ .main (SparkSubmit.scala: 136) 08/06/2018 14: 47: 57,054 INFO [oaspark.launcher.OutputRedirector.redirect] 63: a org.apache.spark.deploy.SparkSubmit.main (SparkSubmit .scala) 08/06/2018 14: 47: 57 058 INFO [oaspark.launcher.OutputRedirector.redirect] 63: 08/06/2018 14:47:57 INFO ShutdownHookManager: 54 - gancho Shutdown llama 08/06/2018 14: 47: 57 060 INFO [oaspark.launcher.OutputRedirector.redirect] 63: 08/06/2018 14:47:57 INFO ShutdownHookManager: 54 - Eliminación de directorio / tmp / chispa 55b54924-e628-43fe-9e43-ed34d7f35a8b 08/06/2018 14: 47: 57 151 INFO [osbalConditionEvaluationReportLoggingListener.logAutoConfigurationReport] 101:
Error al iniciar Application Context. Para mostrar las condiciones informan re-ejecutar la aplicación con la 'depuración' habilitado.
En mi proyecto tengo serpiente yaml 1.19 desde la primavera 5.0 y no hay otros dependicies. No entiendo cuál es el problema, quizás cuando lo puse en el manual de contenedores ventana acoplable que era necesario instalar algo más que la chispa.
Pom del módulo con las operaciones de:
<dependencies>
<dependency>
<groupId>net.mediascope</groupId>
<artifactId>multirating-bigdata-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
<!-- Data Base -->
<dependency>
<groupId>org.jdbi</groupId>
<artifactId>jdbi</artifactId>
<version>2.71</version>
</dependency>
<dependency>
<groupId>com.microsoft.sqlserver</groupId>
<artifactId>sqljdbc42</artifactId>
<version>4.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>org.codehaus.janino</groupId>
<artifactId>commons-compiler</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
</dependency>
<dependency>
<groupId>net.sourceforge.jtds</groupId>
<artifactId>jtds</artifactId>
<version>1.3.1</version>
</dependency>
</dependencies>
<profiles>
<profile>
<id>local</id>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<profiles>
<profile>${profile.active}</profile>
</profiles>
<executable>true</executable>
</configuration>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>hadoop</id>
<build>
<!--Необходимо для адаптации Spring-Boot приложения под запуск через Spark-->
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.handlers</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.schemas</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>META-INF/spring.provides</resource>
</transformer>
<transformer
implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
<resource>META-INF/spring.factories</resource>
</transformer>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>${start-class}</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>I
He encontrado la solución. paquete de origen chispa tiene frascos carpeta donde se snakeyml 1,15, lo cambié a 1,19 y ahora todo el trabajo.