最近在local模式下部署了一个数据筛选的sparkstreaming程序，主要是对一些用户的交易数据进行实时处理，环境配置如下：
spark ：2.1.0，目前是采用单个节点的Standalone部署模式
scala：2.10.4
jdk：1.7.079
该程序在执行时没有出现异常，但是在执行一段时间之后会报
IllegalArgumentException：Requirement failed的异常。
具体异常如下所示：
17/12/23 04:26:47 INFO BlockManager: Removing RDD 13276
17/12/23 04:26:47 INFO MapPartitionsRDD: Removing RDD 13279 from persistence list
17/12/23 04:26:47 INFO BlockManager: Removing RDD 13279
17/12/23 04:26:47 ERROR JobScheduler: Error in job generator
java.lang.IllegalArgumentException: requirement failed
at scala.Predef$.require(Predef.scala:212)
at org.apache.spark.streaming.scheduler.ReceivedBlockTracker.cleanupOldBatches(ReceivedBlockTracker.scala:168)
at org.apache.spark.streaming.scheduler.ReceiverTracker.cleanupOldBlocksAndBatches(ReceiverTracker.scala:233)
at org.apache.spark.streaming.scheduler.JobGenerator.clearMetadata(JobGenerator.scala:274)
at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:184)
at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:89)
at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:88)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed
at scala.Predef$.require(Predef.scala:212)
at org.apache.spark.streaming.scheduler.ReceivedBlockTracker.cleanupOldBatches(ReceivedBlockTracker.scala:168)
at org.apache.spark.streaming.scheduler.ReceiverTracker.cleanupOldBlocksAndBatches(ReceiverTracker.scala:233)
at org.apache.spark.streaming.scheduler.JobGenerator.clearMetadata(JobGenerator.scala:274)
at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:184)
at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:89)
at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:88)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
17/12/23 04:26:47 INFO StreamingContext: Invoking stop(stopGracefully=false) from shutdown hook
17/12/23 04:26:47 INFO ReceiverTracker: ReceiverTracker stopped
17/12/23 04:26:47 INFO JobGenerator: Stopping JobGenerator immediately
17/12/23 04:26:47 INFO RecurringTimer: Stopped timer for JobGenerator after time 1513974470000
17/12/23 04:26:47 INFO JobGenerator: Stopped JobGenerator
17/12/23 04:26:47 INFO JobScheduler: Stopped JobScheduler
17/12/23 04:26:47 INFO StreamingContext: StreamingContext stopped successfully
17/12/23 04:26:47 INFO SparkContext: Invoking stop() from shutdown hook
17/12/23 04:26:47 INFO SparkUI: Stopped Spark web UI at http://*.*.*.*:4040
17/12/23 04:26:47 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/12/23 04:26:47 INFO MemoryStore: MemoryStore cleared
17/12/23 04:26:47 INFO BlockManager: BlockManager stopped
17/12/23 04:26:47 INFO BlockManagerMaster: BlockManagerMaster stopped
17/12/23 04:26:47 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/12/23 04:26:47 INFO SparkContext: Successfully stopped SparkContext
17/12/23 04:26:47 INFO ShutdownHookManager: Shutdown hook called
17/12/23 04:26:47 INFO ShutdownHookManager: Deleting directory /tmp/spark-21c7303b-7c66-4d0e-a6e6-4731b2770d92
通过StackTrace可以了解到该异常出现的位置，分别是下图所示：
可以看到该方法是在清除元数据时通过ClearMetaData方法调用的，传入cleanupOldBlocksAndBatches的参数为当前时间（time)减入参的时间间隔（maxRemeberDuration)也就是获取该批次开始处理的时刻，结合源码中对于cleanupOldBatches函数的解释，“清除旧的数据块的信息，如果等待完成标志为true，这个方法将只会在文件清理后返回“。
异常也是在该方法中require函数中抛出的，clock.getTimeMillis方法的返回值为获取ReciveTracker类的初始化时间，也就是说该异常是在数据处理后，在清除元数据时发现理论上的批次开始处理时间已经大于该类的初始化时间才会报错，由此可以推断出是处理时间过长所导致的。
所以解决办法:由于部署的环境单核CPU，2G内存所以通过减少了BlockInterval的值直接减少了生成的block的数量，减少接收到数据后生成Job的开销，并且对于接受数据的速率通过“spark.streaming.backpressure”参数设置为true成功解决问题。