【Kafka】Failed to send data to Kafka: Expiring 30 record(s) for xxx 732453 ms has passed since last a

1.美图

2.背景

2020-04-19 21:01:29,612 WARN  org.apache.kafka.clients.consumer.internals.ConsumerCoordinator  - Auto-commit of offsets {com.dbapp.topic.rawevent-5=OffsetAndMetadata{offset=386466448, metadata=''}, com.dbapp.topic.rawevent-2=OffsetAndMetadata{offset=386357064, metadata=''}} failed for group cep.metric: Offset commit failed with a retriable exception. You should retry committing offsets.
2020-04-20 04:47:01,972 INFO  org.apache.kafka.clients.producer.KafkaProducer               - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2020-04-20 04:50:00,089 WARN  org.apache.kafka.clients.producer.internals.Sender            - Got error produce response with correlation id 640010 on topic-partition com.dbapp.ailpha.statistics.metric-1, retrying (2 attempts left). Error: NETWORK_EXCEPTION
2020-04-20 04:50:00,089 WARN  org.apache.kafka.clients.producer.internals.Sender            - Got error produce response with correlation id 640009 on topic-partition com.dbapp.ailpha.statistics.metric-1, retrying (2 attempts left). Error: NETWORK_EXCEPTION
2020-04-20 04:50:00,089 WARN  org.apache.kafka.clients.producer.internals.Sender            - Got error produce response with correlation id 640008 on topic-partition com.dbapp.ailpha.statistics.metric-1, retrying (2 attempts left). Error: NETWORK_EXCEPTION
2020-04-20 04:50:00,089 WARN  org.apache.kafka.clients.producer.internals.Sender            - Got error produce response with correlation id 640007 on topic-partition com.dbapp.ailpha.statistics.metric-1, retrying (2 attempts left). Error: NETWORK_EXCEPTION
2020-04-20 04:50:00,089 WARN  org.apache.kafka.clients.producer.internals.Sender            - Got error produce response with correlation id 640006 on topic-partition com.dbapp.ailpha.statistics.metric-1, retrying (2 attempts left). Error: NETWORK_EXCEPTION
2020-04-20 04:50:00,209 WARN  org.apache.kafka.clients.producer.internals.Sender            - Got error produce response with correlation id 639212 on topic-partition com.dbapp.ailpha.statistics.metric-2, retrying (2 attempts left). Error: NETWORK_EXCEPTION
2020-04-20 04:50:00,209 WARN  org.apache.kafka.clients.producer.internals.Sender            - Got error produce response with correlation id 639211 on topic-partition com.dbapp.ailpha.statistics.metric-2, retrying (2 attempts left). Error: NETWORK_EXCEPTION
2020-04-20 04:50:00,209 WARN  org.apache.kafka.clients.producer.internals.Sender            - Got error produce response with correlation id 639210 on topic-partition com.dbapp.ailpha.statistics.metric-2, retrying (2 attempts left). Error: NETWORK_EXCEPTION
2020-04-20 04:50:00,209 WARN  org.apache.kafka.clients.producer.internals.Sender            - Got error produce response with correlation id 639209 on topic-partition com.dbapp.ailpha.statistics.metric-2, retrying (2 attempts left). Error: NETWORK_EXCEPTION
2020-04-20 04:50:00,209 WARN  org.apache.kafka.clients.producer.internals.Sender            - Got error produce response with correlation id 639208 on topic-partition com.dbapp.ailpha.statistics.metric-2, retrying (2 attempts left). Error: NETWORK_EXCEPTION
2020-04-20 05:02:31,121 ERROR org.apache.flink.streaming.runtime.tasks.StreamTask           - Error during disposal of stream operator.
java.lang.Exception: Failed to send data to Kafka: Expiring 30 record(s) for com.dbapp.ailpha.statistics.metric-2: 732453 ms has passed since last append
        at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.checkErroneous(FlinkKafkaProducerBase.java:376)
        at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.close(FlinkKafkaProducerBase.java:324)
        at org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:43)
        at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.dispose(AbstractUdfStreamOperator.java:117)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:585)
        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:484)
        at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 30 record(s) for com.dbapp.ailpha.statistics.metric-2: 732453 ms has passed since last append
2020-04-20 05:02:31,121 INFO  org.apache.flink.runtime.taskmanager.Task                     - OutputEnricher -> Sink: metricSink (3/3) (3b77e9add4309050ae3f070dd5283a0d) switched from RUNNING to FAILED.
java.lang.Exception: Failed to send data to Kafka: Failed to allocate memory within the configured max blocking time 60000 ms.
        at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.checkErroneous(FlinkKafkaProducerBase.java:376)
        at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer010.invoke(FlinkKafkaProducer010.java:352)
        at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)