Thinking of spark speculative execution

And finally to Friday, and turn on the computer to check the running cluster,
a workflow had been performed only 1h, now double the execution time?
Should be a reason to retry mechanism spark, the end result was the successful implementation of the
Here Insert Picture Description
workflow execution history Time 4: 00-5: 14
workflow2 historical execution time 5: 00-5: 56

According Start Time display workflow started at 4:21, 5:00 is workflow2 stuck,
after the release of resources workflow2, 5:52 start again and execute successful.

Then adjust workflow2 footprint, it can not fully occupied clusters lot of resources!

Published 118 original articles · won praise 25 · Views 150,000 +

Guess you like

Origin blog.csdn.net/lhxsir/article/details/90713249