バックグラウンド
最近、flinkタスクの開発において、kuduにデータを書き込む必要があり、テスト環境では問題ないのですが、本番環境ではエラーメッセージが多数発生します（以下）。
org.apache.kudu.client.NonRecoverableException: MANUAL_FLUSH is enabled but the buffer is too big
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) ~[blob_p-371f6d8c9621c84079fa2c5ecdaf852455599896-1bb2c1ac8a602c94efbddf261bf4997c:?]
at org.apache.kudu.client.KuduSession.apply(KuduSession.java:93) ~[blob_p-371f6d8c9621c84079fa2c5ecdaf852455599896-1bb2c1ac8a602c94efbddf261bf4997c:?]
at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:50) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableProcessWindowFunction.process(InternalIterableProcessWindowFunction.java:57) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableProcessWindowFunction.process(InternalIterableProcessWindowFunction.java:32) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents(WindowOperator.java:577) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime(WindowOperator.java:533) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:284) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1425) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$16(StreamTask.java:1416) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:344) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:330) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) [flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) [flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) [flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) [flink-dist_2.11-1.13.3.jar:1.13.3]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduSession.apply(AsyncKuduSession.java:596) ~[blob_p-371f6d8c9621c84079fa2c5ecdaf852455599896-1bb2c1ac8a602c94efbddf261bf4997c:?]
at org.apache.kudu.client.KuduSession.apply(KuduSession.java:79) ~[blob_p-371f6d8c9621c84079fa2c5ecdaf852455599896-1bb2c1ac8a602c94efbddf261bf4997c:?]
at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:50) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableProcessWindowFunction.process(InternalIterableProcessWindowFunction.java:57) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.operators.windowing.functions.InternalIterableProcessWindowFunction.process(InternalIterableProcessWindowFunction.java:32) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents(WindowOperator.java:577) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime(WindowOperator.java:533) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:284) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1425) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$16(StreamTask.java:1416) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:344) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:330) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) [flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) [flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) [flink-dist_2.11-1.13.3.jar:1.13.3]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) [flink-dist_2.11-1.13.3.jar:1.13.3]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
解決
まず、ここでの解決策について説明します。ステップは 2 つあります
。1、kuduSession は maxBufferSize を設定する必要があります
。具体的なコードは次のとおりです。
kuduSession.setMutationBufferSpace(maxBufferSize)
maxBufferSize は正の整数であり、10000 など、より大きな値に設定する必要があります。
2. データを手動でフラッシュする 書き込ま
れたデータがしきい値を超えた場合、flush() を呼び出して一度書き込みます。このしきい値は maxBufferSize より小さい必要があります (9000 など)。
i = 0;
for (Tuple2<Boolean, Row> data : dataList) {
errorMsg = kuduWriter.write(table, opsMapper, data);
if (++i > 9000) {
kuduWriter.flush();
i = 0;
}
}
kuduWriter.flush();
上記の 2 つの手順で上記のエラー メッセージを解決できます。
3. 根本原因の分析
上記のログスタックから、エラーが報告されている場所が KuduSession.apply() であることがわかり、次にこのメソッド内を調べて、最終的に次のメソッドでエラーが報告されていることがわかります。
org.apache.kudu.client.AsyncKuduSession.apply(最終操作オペレーション)
このメソッドは比較的長いですが、コードの一部を次に示します。
switch (flushMode) {
case AUTO_FLUSH_SYNC: {
// This case is handled above and is impossible here.
// TODO(wdberkeley): Handle AUTO_FLUSH_SYNC just like other flush modes.
assert false;
break;
}
case MANUAL_FLUSH: {
if (activeBufferSize >= mutationBufferMaxOps) {
Status statusIllegalState =
Status.IllegalState("MANUAL_FLUSH is enabled but the buffer is too big");
throw new NonRecoverableException(statusIllegalState);
}
activeBuffer.getOperations().add(new BufferedOperation(tablet, operation));
break;
}
...
コードのこの部分から、例外を生成するには 2 つの条件があることがわかります:
1. flashMode = MANUAL_FLUSH、つまり手動フラッシュ モード;
2. activeBufferSize >= mutationBufferMaxOps
activeBufferSize は、書き込まれたデータのサイズを指します。単に write() の呼び出し数として理解されます (毎回 1 項目を書き込む場合)。 mutationBufferMaxOps は、バッファー
のサイズを指します。
つまり、手動モードでは、kuduにデータを書き込むと、まずバッファ（バッファ）に書き込まれ、ある程度の量が書き込まれたときにflush()が呼び出されてデータベースにフラッシュされます。
書き込まれたデータの量がバッファの最大許容数を超えた場合、エラーが報告されます。
したがって、解決策は、バッファを大きく設定し、適切なタイミングでフラッシュ() を呼び出すか、手動リフレッシュ モードを使用しないことです。