Data validation between source and target table | PySpark Interview Question |

[@beingnagur]

At 6.04 instead of copying the same statement you can use .otherwise("not matching")

[@rishabhkesarwani-br2rx]

I do below steps to compare source vs target table 1) Count should be matching in source and target table 2) Schema should be matching in source and target table 3) Use the except and to check if any records are there which are present in source and not in target or vice versa. 4) Use the left anti join to find out the records which are not matching. 5) Trying to debug why there is record mismatch

[@GeekCoders]

Nice

[@gudiatoka]

exceptAll can be usefull too or anti join

[@GeekCoders]

Except all may miss the null value sometime

[@CeejayPTcoach]

wont the join be a costly operation

[@jhonsen9842]

Main Problem i found in learning Pyspark is brackets every time it gives me some error.

[@GeekCoders]

Yes

[@nishirajnikku969]

I request you to please create a playlist for Pyspark Unit testing .

[@shivamchandan50]

plz make video on pyspark unit testing

[@VinodKumar-gz8bk]

What are the most challenging thing that you faced in your project & how you overcome?