1.美图
2.概述
一个flink程序时好时坏，输出到不同的sink，但是只有一个sink报错
flink界面看到的错误如下
java.lang.RuntimeException: java.util.ConcurrentModificationException
Serialization trace:
classes (sun.misc.Launcher$AppClassLoader)
contextClassLoader (java.lang.Thread)
threads (java.lang.ThreadGroup)
groups (java.lang.ThreadGroup)
group (java.util.concurrent.Executors$DefaultThreadFactory)
val$backingThreadFactory (org.apache.flink.shaded.guava18.com.google.common.util.concurrent.ThreadFactoryBuilder$1)
threadFactory (org.apache.flink.shaded.netty4.io.netty.util.concurrent.ThreadPerTaskExecutor)
executor (org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop)
children (org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoopGroup)
group (org.apache.flink.shaded.netty4.io.netty.bootstrap.Bootstrap)
bootstrap (org.apache.flink.runtime.io.network.netty.NettyClient)
client (org.apache.flink.runtime.io.network.netty.NettyConnectionManager)
connectionManager (org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel)
registeredListeners (org.apache.flink.runtime.io.network.buffer.LocalBufferPool)
allBufferPools (org.apache.flink.runtime.io.network.buffer.NetworkBufferPool)
networkBufferPool (org.apache.flink.runtime.io.network.buffer.LocalBufferPool)
recycler (org.apache.flink.runtime.io.network.buffer.BufferBuilder)
value (java.util.Optional)
bufferBuilders (org.apache.flink.runtime.io.network.api.writer.RecordWriter)
recordWriter (org.apache.flink.streaming.runtime.io.RecordWriterOutput)
val$output (com.dbapp.baas.engine.rule.functions.AssetOperator$1)
collector (org.apache.flink.runtime.operators.util.metrics.CountingCollector)
at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110)
at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89)
at org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45)
at com.dbapp.baas.engine.rule.functions.AssetOperator$1.collect(AssetOperator.java:67)
at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
at com.dbapp.baas.engine.rule.functions.AssetFunction.onReceivedControlEvent(AssetFunction.java:161)
at com.dbapp.baas.engine.rule.functions.AssetFunction.processElement2(AssetFunction.java:71)
at com.dbapp.baas.engine.rule.functions.AssetOperator.processElement2(AssetOperator.java:49)
at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:286)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279)
at org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
Serialization trace:
classes (sun.misc.Launcher$AppClassLoader)
contextClassLoader (java.lang.Thread)
threads (java.lang.ThreadGroup)
groups (java.lang.ThreadGroup)
group (java.util.concurrent.Executors$DefaultThreadFactory)
val$backingThreadFactory (org.apache.flink.shaded.guava18.com.google.common.util.concurrent.ThreadFactoryBuilder$1)
threadFactory (org.apache.flink.shaded.netty4.io.netty.util.concurrent.ThreadPerTaskExecutor)
executor (org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop)
children (org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoopGroup)
group (org.apache.flink.shaded.netty4.io.netty.bootstrap.Bootstrap)
bootstrap (org.apache.flink.runtime.io.network.netty.NettyClient)
client (org.apache.flink.runtime.io.network.netty.NettyConnectionManager)
connectionManager (org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel)
registeredListeners (org.apache.flink.runtime.io.network.buffer.LocalBufferPool)
allBufferPools (org.apache.flink.runtime.io.network.buffer.NetworkBufferPool)
networkBufferPool (org.apache.flink.runtime.io.network.buffer.LocalBufferPool)
recycler (org.apache.flink.runtime.io.network.buffer.BufferBuilder)
value (java.util.Optional)
bufferBuilders (org.apache.flink.runtime.io.network.api.writer.RecordWriter)
recordWriter (org.apache.flink.streaming.runtime.io.RecordWriterOutput)
val$output (com.dbapp.baas.engine.rule.functions.AssetOperator$1)
collector (org.apache.flink.runtime.operators.util.metrics.CountingCollector)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:82)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:348)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:289)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:348)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:289)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:348)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:289)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:82)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:22)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:82)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:22)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:577)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:320)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:289)
at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:577)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:68)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.serialize(KryoSerializer.java:305)
at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.serialize(StreamElementSerializer.java:175)
at org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.serialize(StreamElementSerializer.java:46)
at org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:54)
at org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.serializeRecord(SpanningRecordSerializer.java:78)
at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:152)
at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:120)
at org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:107)
... 14 more
Caused by: java.util.ConcurrentModificationException
at java.util.Vector$Itr.checkForComodification(Vector.java:1184)
at java.util.Vector$Itr.next(Vector.java:1137)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:74)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:22)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
... 105 more
但是在yarn界面看到的错误入下
yarn logs --applicationId application_1585309476321_0048 > a.txt
2020-04-13 19:45:33,407 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --------------------------------------------------------------------------------
2020-04-13 19:45:33,409 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnJobClusterEntrypoint (Version: 1.9.1, Rev:4d56de8, Date:30.09.2019 @ 11:32:19 CST)
2020-04-13 19:45:33,409 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: yarn
2020-04-13 19:45:33,935 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: hdfs
2020-04-13 19:45:33,935 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.112-b15
2020-04-13 19:45:33,935 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 406 MiBytes
2020-04-13 19:45:33,935 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /usr/jdk64/jdk1.8.0_112
2020-04-13 19:45:33,937 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Hadoop version: 2.7.3.2.5.3.0-37
2020-04-13 19:45:33,937 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
2020-04-13 19:45:33,937 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms424m
2020-04-13 19:45:33,937 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx424m
2020-04-13 19:45:33,937 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog.file=/data/ha
2020-04-13 19:45:33,949 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.framesize, 104857600b
2020-04-13 19:45:33,949 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, DETACHED
2020-04-13 19:45:33,949 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2020-04-13 19:45:33,949 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 4
2020-04-13 19:45:33,949 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.ask.timeout, 300s
2020-04-13 19:45:33,949 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.size, 2048m
2020-04-13 19:45:33,949 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.size, 1024m
2020-04-13 19:45:33,950 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: akka.client.timeout, 300s
2020-04-13 19:45:33,987 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.bind-port'
2020-04-13 19:45:34,007 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnJobClusterEntrypoint.
2020-04-13 19:45:34,007 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem.
2020-04-13 19:45:34,008 WARN org.apache.flink.core.plugin.PluginConfig - Environment variable [FLINK_PLUGINS_DIR] is set to [/usr/hdp/2.5.3.0-37/baas/flink/plugins] but the directory doesn't exist
2020-04-13 19:45:34,104 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to hdfs (auth:SIMPLE)
2020-04-13 19:45:34,121 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services.
2020-04-13 19:45:34,379 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at 1.datanode2:0
2020-04-13 19:45:35,058 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2020-04-13 19:45:35,158 INFO akka.remote.Remoting - Starting remoting
2020-04-13 19:45:35,320 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://[email protected]:44799]
2020-04-13 19:45:35,591 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system started at akka.tcp://flink@xxx:44799
2020-04-13 19:45:35,614 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'web.port' instead of proper key 'rest.port'
2020-04-13 19:45:35,627 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /data/hadoop/yarn/local/usercache/hdfs/appcache/application_1585309476321_0048/blobStore-778db280-4cd0-4692-ae13-510a71dc5372
2020-04-13 19:45:35,632 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:43563 - max concurrent requests: 50 - max backlog: 1000
2020-04-13 19:45:35,701 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported.
2020-04-13 19:45:35,704 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start actor system at 1.datanode2:0
2020-04-13 19:45:35,734 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2020-04-13 19:46:59,751 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Asset Engine (1/1) (1355aea587386394ba418b9d81b0f94c) switched from RUNNING to FAILED.
java.lang.StackOverflowError
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:445)
at com.twitter.chill.KryoBase.getRegistration(KryoBase.scala:52)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:488)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:57)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
程序本地启动的时候，不报错,但是部署到服务器就报错
3.怀疑版本
<!-- https://mvnrepository.com/artifact/com.esotericsoftware.kryo/kryo -->
<dependency>
<groupId>com.esotericsoftware.kryo</groupId>
<artifactId>kryo</artifactId>
<version>2.24.0</version>
</dependency>
怀疑是版本的问题，于是想尝试，升级一下版本会不会变好。
先看看哪里引入的这个包，结果发现是flink-core中引入的。
但是抱着试试的态度，升级一把试试。
