Spark streaming kafka problem when the consumer does not perform multiple topic

By breakpoints follow-up data are found each topic can go to, but in the end will be blocked on the floor of the operations performed DataFrame;

Such as:

df.write.format("parquet").mode("append").save(path)

 

Careful observation can discover the type of log: INFO scheduler.JobScheduler: Added Jobs  for Time ××××× log;

 

The reason: Receiver running thread is not enough

 

Solution: increase the available thread

loca mode, consider increasing the local [n], n increasing size;

 

 

 

 

reference:

https://stackoverflow.com/questions/36281665/continuously-info-jobscheduler59-added-jobs-for-time-ms-in-my-spark-stand，

https://www.iteye.com/blog/bit1129-2174751