In-depth profiling of wide and narrow dependencies
The concept of wide and narrow dependencies:
- Narrow dependencies: An RDD that has only a simple one-to-one dependency on its parent RDD. That is, the correspondence between the partitions of the parent RDD and the child RDD is one-to-one
- Wide dependency: The English name shuffle dependency is essentially shuffle. That is, the partition data of each RDD may have a part of the data passed to the partition of the next RDD. At this point, there is an intricate interaction between the parent RDD and the child RDD, and shuffle occurs between them.
As shown below: