A Deep Dive into Query Execution Engine of Spark SQL - Maryann Xue

[@αλήθεια-σ4κ]

Excellent session, very well explained

[@megharaina7561]

what is the difference between a normal stage in job and a WSCG . Do multiple pipelines within a single WSCG correspond to two separate stages

[@vinayakalagwadi5866]

according to my understanding the stages are separated only if there is a blocking op which requires aggregations ,,

[@reesewinterspoon9417]

Pipelines are separated by blocking operations, while stages are separated by shuffles (in most cases). A stage can have multiple pipelines. For example: the stage with hash aggregate will have multiple pipelines because hash aggregate is a blocking op, if its just a partial hashagg since that does not need a shuffle, all operations would be in a single stage. Hope that helps