DAG and Lazy Evaluation in spark

In this video I have talked about dag and lazy evaluation in spark in great detail. please follow video entirely and ask doubt in comment section below.
Directly connect with me on:- topmate.io/man...
For more queries reach out to me on my below social media handle.
Follow me on LinkedIn:- / manish-kumar-373b86176
Follow Me On Instagram:- / competitive_gyan1
Follow me on Facebook:- / manish12340
My Second Channel -- / @competitivegyan1
Interview series Playlist:- • Interview Questions an...
My Gear:-
Rode Mic:-- amzn.to/3RekC7a
Boya M1 Mic-- amzn.to/3uW0nnn
Wireless Mic:-- amzn.to/3TqLRhE
Tripod1 -- amzn.to/4avjyF4
Tripod2:-- amzn.to/46Y3QPu
camera1:-- amzn.to/3GIQlsE
camera2:-- amzn.to/46X190P
Pentab (Medium size):-- amzn.to/3RgMszQ (Recommended)
Pentab (Small size):-- amzn.to/3RpmIS0
Mobile:-- amzn.to/47Y8oa4 ( Aapko ye bilkul nahi lena hai)
Laptop -- amzn.to/3Ns5Okj
Mouse+keyboard combo -- amzn.to/3Ro6GYl
21 inch Monitor-- amzn.to/3TvCE7E
27 inch Monitor-- amzn.to/47QzXlA
iPad Pencil:-- amzn.to/4aiJxiG
iPad 9th Generation:-- amzn.to/470I11X
Boom Arm/Swing Arm:-- amzn.to/48eH2we
My PC Components:-
intel i7 Processor:-- amzn.to/47Svdfe
G.Skill RAM:-- amzn.to/47VFffI
Samsung SSD:-- amzn.to/3uVSE8W
WD blue HDD:-- amzn.to/47Y91QY
RTX 3060Ti Graphic card:- amzn.to/3tdLDjn
Gigabyte Motherboard:-- amzn.to/3RFUTGl
O11 Dynamic Cabinet:-- amzn.to/4avkgSK
Liquid cooler:-- amzn.to/472S8mS
Antec Prizm FAN:-- amzn.to/48ey4Pj

[@tnmyk_]

Faadu explanation! Finally someone explained why Lazy evaluation actually works betters for Big Data processing. Amazing examples, very nice code! Loved the way you explained each line and each job step by step

[@prasadBoyane]

I think spark considers 'sum' as action. hence 4 jobs. Greatt series !!!

[@ajinkyadeshmukh2343]

yes, sum is an action in spark

[@PARESH_RANJAN_ROUT]

Great Bhai

[@fervabatool1037]

Excellent

[@ChandanKumar-xj3md]

"Job kaise create hota hai?" ye question pehle kabi clear nai hua tha but thanks Manish for clearing this out and add on was lazy evaluation understanding. 👍

[@vsbnr5992]

NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark please I stuck here

[@SanjayKumar-rw2gj]

Truly impressed Manish bhai. Great explanation as you mentioned already "Itna detail mein kahin nhi milega"

[@krishnavamsirangu1727]

Hi Manish Thanks for explaining the concept in detail by running the code. I have understood the concept of dag ,lazy evaluation and optimization.

[@arijitsamaddar268]

bohot sahi explanation !

[@akhiladevangamath1277]

Thank you Thank you Thank you Manish for this video✨✨✨

[@Watson22j]

wow! very nicely explained. Thank you! :)

[@prabhakarkumar8022]

Awesome bhaiyaji!!!!!

[@bhavindedhia3976]

amazing content

[@souradeep.official]

Detailed Explanation. Better than paid lectures.

[@220piyush]

Maza aa gya lekin video dekh ke... Wahhh❤

[@abinashpandit5297]

Very good Bhaiya. Aaj bhaut kuch isme indepth sikhne ko Mila jo phele pata hi nahi tha. Keep it up 👍

[@AnuragsMusicChannel]

sum() is an action, but the key here is to understand that its not triggerred until an action triggers it. Example : select(sum("value")) is indeed a transformation. It creates a new DataFrame that represents the sum of the value column but does not immediately compute the result, but the actual computation does not happen until an action is triggered. Later stage par, when we call show() (or even collect() ) tab jake, action trigger hoga and will call sum() which is inside select() which creates a job corresponding to sum(). Thats why 4 jobs were seen.

[@SqlMastery-fq8rq]

very well explained Sir..Thank You.

[@ankitachauhan6084]

thank you ! great teaching style

[@prabhatgupta6415]

He has mastered and crunched the spark.

[@maurifkhan3029]

I too got confused as to why sometimes number of jobs as more or less than Actions. Try clearing the state using menu option run ->clear state and option and then run the cell again which has code from reading of file till all the things you want to perform . I think Data bricks intelligently stores state of system and later when you run same read command the Jobs count might not match I tried this and it seems to be working

[@jatinyadav6158]

Jobs count is right it is 4 only because sum() function is an action, which I guess Manish missed by mistake. Btw @Manish thank you so much for the amazing course.

[@deepanshuaggarwal7042]

@@jatinyadav6158 If 'sum' is an action then why it didn't create a job before adding 'show' codeline ?

[@jatinyadav6158]

@deepanshuaggarwal7042 yes sum is an action, I am not sure why it didn't show a job earlier

[@rishav144]

great video Manish bro

[@aasthagupta9381]

You are an excellent teacher, you make lectures so interesting! ye answer dekar to interview ko sikha denge :D

[@a26426408]

Very well explained.

[@yugantshekhar782]

Great explanation sir, really helpful!

[@akashprabhakar6353]

Awesome lecture...thanks a lot!

[@prathapganesh7021]

Thank you so much for clarifying my doubts 🙏

[@kavyabhatnagar716]

Wow! Thank you for such great explanation. ❤

[@arpitchaurasia5132]

bhai gajabe padate ho yar maja hi a gya yar

[@pramod3469]

very well explained...thanks Manish

[@ravikiran3672]

For wide trabsformation there iwll be an extra job will be created. For n transformations there will be n+1 jobs.

[@manojkaransingh5848]

@wow...!..v.nice bro

[@mmohammedsadiq2483]

I have confusion? read and inferSchema are typically used with Spark's DataFrame API, which is part of Spark SQL. They are not transformations or actions ,part of the logical and physical planning phase of Spark, which occurs before any actions are executed

[@welcomefoodies6901]

Hi manish bhaiya, yahan pr 4 actions hit hue ha : read, inferschema, sum, show

[@tahiliani22]

Awesome. By the way, do we know why its creating 4 Spark Jobs instead of 3 ?

[@SanjayKumar-rw2gj]

Is there any cheat sheet to know what all are transformations and actions, like read is a action whereas filter is a transformation?

[@choubeysumit246]

one Action one job is true for rdd api only. one action in dataframe or dataset can lead to multiple actions being generated internally. or sue to adaptive query executions as well multiple jobs are created in databricks which you can see using describe method

[@jasvirsinghwalia401]

Sir Read an inferschema to Transformations hai na and not actions? to inki alag jobs kyu bani hai?

[@abhilovefood4102]

Sir ur teaching is good

[@rohitjhunjhunwala9174]

One thing, spark.read is a transformation not action. Is it an action because we included any options? Pls clarify

[@mission_possible]

Thanks for the session and Please make video on Spark Lineage

[@koushlendrasinghrajput6040]

please give data set so we can practice on thAT

[@krushitmodi3882]

Sir please ye series thodi jaldi finish karo taki ham interview de sake mene apki puri channel dekhli hai Thank you

[@Storytime389]

4 jobs come because u calculated sum and then .show(). I think sum() increases the job number. Correct me if am wrong

[@AmitSharma-ow8wm]

waiting for ur next vidio...

[@AmitSharma-ow8wm]

@@rampal4570 is it true bro

[@manish_kumar_1]

Aaj aa jayega

[@vsbnr5992]

@@AmitSharma-ow8wm NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark please I stuck here

[@roshniagrawal4777]

sum is action

[@ankitas4019]

Where he explained about fligh data download

[@user-ww6yf3iq8q]

Because of group by jobs is created

[@manish_kumar_1]

Nope

[@vaibhavdimri7419]

Sir apko samjh aya ki ek action hit karne par 2 jobs kaise create hui?

[@DevendraYadav-yz2so]

Databricks community ko kaise use karege, Spark kaise setup karege databricks ke sath. Please ye bata dijiye so that code write kr sake

[@aditya_1005]

well explained.....Sir could you please clarify, 3 actions and 4 jobs created?

[@manish_kumar_1]

Aapke 3 actions me 4 jobs create hue hai? Aapne show use kara hai? And aap apna code v paste kar dijiye comment section me

[@hazard-le7ij123]

@@manish_kumar_1 Aapne jo code likha hai usme bhi 4 jobs create hue hain. Can you explain that? Below is my code and same thing is happening. 4 Jobs are getting created. Stage is getting skipped but why do we have an extra job with 4 diff Job Ids? from pyspark.sql import SparkSession from pyspark.sql.functions import * spark = SparkSession.builder.master('local[5]') \ .appName("Lazy Evaluation internal working") \ .getOrCreate() flight_data = spark.read.format("csv")\ .option("header","true")\ .option("inferSchema","true")\ .load("D:\\Spark\\flight_data.csv") flight_data_repartition = flight_data.repartition(3) us_flight_data = flight_data.filter(col("DEST_COUNTRY_NAME")=='United States') us_india_data = us_flight_data.filter((col("ORIGIN_COUNTRY_NAME")=='India') | (col("ORIGIN_COUNTRY_NAME")=='Singapore')) total_flight_ind_sing = us_india_data.groupby("DEST_COUNTRY_NAME").sum("count") total_flight_ind_sing.show() input("Enter to terminate")

[@user-gt3pi6ir5u]

any idea now, where the 4th job came from?

[@pramod3469]

is lazy evaluation consider the partition also like after we have applied orderby on salary col and now we want to show only first two highest salary so will lazy evaluation also works here spark will process only that partition which has these two salary records or it will process all partitions and then extract first two highest salary record for us

[@manish_kumar_1]

Yes until you write .head(2) for 2 highest record your process will not start although in backend it will create DAG.

[@ruinmaster5039]

Bro Plese add summery at the end.

[@anirbanadhikary7997]

Aj apne interview questions bataya Nehi.

[@manish_kumar_1]

Basic questions would be there. Like what is DAG and what is edges and vertices in it.

[@ChetanSharma-oy4ge]

I am trying to find , why 4 jobs are generating here although we have provided only 3 actions

[@techworld5477]

Hi Sir..jab main yeh code run kar raha hoon I am getting error as--name 'col' is not defined isko kaise solve kare?

[@chethanmk5852]

Why do we have 4 jobs when we are using only 3 actions in the application??

[@vsbnr5992]

NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark please I stuck here

[@manish_kumar_1]

Seems like your df is not defined

[@vsbnr5992]

@@manish_kumar_1 ok working now thanks

[@abhayjr11]

Bhai iske phle wala video dedo, mujhe mil nhi rha hai..

[@welcomefoodies6901]

I have 1 doubt bhai , what is difference between apache spark and apache airflow ?

[@manish_kumar_1]

Completely different purpose and different technology. Google kijiye aap

[@welcomefoodies6901]

@@manish_kumar_1 thanku bhaiya , ap Apache airflow pr bhi series bna do please, i really connect with you , jo paid courses se samj nhi aata wo ap ache se samjha rhe ho , thanku bhai , love you bro 🙌❤️

[@deepaliborde25]

where is the practical session link ?

[@mahendrareddych334]

Bro, you are explaining superbly but why don't you explain in English. Everyone doesn't know Hindi. I don't know Hindi but watching your videos to understand the concepts but not getting it fully because it was explained in Hindi.

[@chandanpatra1053]

ek code likha hai using spark .usse dekhkar kese bataya ja sakta hai ki wo code 'action' hai ya 'transformation' hai.

[@manish_kumar_1]

Aapko google karke pata karna chahiye ki kon kon se actions hai. Rest are transformation

[@prateekpawar1871]

Do you have theory notes for spark?

[@ordinary_indian]

where to find the files ? I just have started the course

[@snehalkathale98]

Where I get CSV file

[@abhishekchaturvedi9855]

Hello Manish. When you mentioned the sql query gets optimized by spark. Just wanted to know will it help improve the execution time if we use the optimized query in our code itself so that spark need not do it ?

[@manish_kumar_1]

Spark optimization is very limited. So as a developer we should write optimized code to run our process faster

[@prabhatsingh7391]

Hi Manish Bhaiya, in the code snippet you told there are three actions in this applications(read, infer schema and show) but in spark ui there are 4 jobs created ,can you please explain this.

[@manish_kumar_1]

1 job skip hua hoga. Agar data Kam hai to explain karke dekhiye 3 aana chahiye

[@ajaysinghjadoun9799]

please make a video in Windows function

[@manish_kumar_1]

Sure

[@ajaysinghjadoun9799]

Sir also considers spark 5s problems.spill, shuffle, storage, etc

[@asif50786]

How many more videos to come on Apache spark??

[@manish_kumar_1]

Around 20-30. It's just beginning of spark

[@DevendraYadav-yz2so]

Lec 7 Tak view kr liya Jo aap code dika rahe hai usko kise databricks and pyspark

[@manish_kumar_1]

Aapko practical and fundamentals sath me dekhne hai. First video me hi bataya tha

[@shrikantpandey6401]

could you provide notebook link?. It will good for hands on

[@manish_kumar_1]

I don't provide notebook or pdf. Take notes and type every line of code by yourself. This will give you confidence

[@sankuM]

@@manish_kumar_1 this is indeed really great point! However, if possible, do share your own reference material for our benefit! Thanks! This series is really helpful, I've 4+ YoE in DE but never tried to go into spark internals, now while interviewing for switch, I'm definitely going to utilize all this! Keep 'em coming!! 🙌🏻👏🏻

[@raghavsisters]

Why it is called acyclic?

[@manish_kumar_1]

Because it doesn't make cycle. If it's get into cycle or you can consider it as a circle then it will run endlessly

[@khurshidhasankhan4700]

Sir csv read karne pr two jobs kaise create ho raha hai, read only one action call kr rahe hain. If possible please clearify

[@manish_kumar_1]

Aur inferschema v use kiye honge. Isliye aa rha hoga

[@khurshidhasankhan4700]

@@manish_kumar_1 thank you sir, can you please share the action list how many action hai spark me, if possible please share sir

[@akhilgupta2460]

Hi manish Bhai, Could u provide the flight data file.

[@manish_kumar_1]

Kisi ek video me bataya tha. Please follow all videos in sequence

[@avanibafna6207]

In my case same code has created 5 jobs?I have import col so it will also be treated as action and new job will be created is it so?

[@manish_kumar_1]

Can you please paste your code in comment section

[@avanibafna6207]

@@manish_kumar_1 from pyspark.sql.functions import col flight_data=spark.read.format("csv")\ .option("header","true")\ .option("inferSchema","true")\ .load("dbfs:/FileStore/tables/flight_data.csv") flight_data_reparition=flight_data.repartition(3) us_flight_data=flight_data_reparition.filter("DEST_COUNTRY_NAME='United States'") us_india_data=us_flight_data.filter((col("ORIGIN_COUNTRY_NAME")=='India')|(col("ORIGIN_COUNTRY_NAME")=='Singapore')) total_flight_ind_sing=us_india_data.groupby("DEST_COUNTRY_NAME").sum("count") total_flight_ind_sing.show() (5) Spark Jobs Job 22 View(Stages: 1/1) Job 23 View(Stages: 1/1) Job 24 View(Stages: 1/1) Job 25 View(Stages: 1/1, 1 skipped) Job 26 View(Stages: 1/1, 2 skipped) flight_data:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field] flight_data_reparition:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field] us_flight_data:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field] us_india_data:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, ORIGIN_COUNTRY_NAME: string ... 1 more field] total_flight_ind_sing:pyspark.sql.dataframe.DataFrame = [DEST_COUNTRY_NAME: string, sum(count): long] +-----------------+----------+ |DEST_COUNTRY_NAME|sum(count)| +-----------------+----------+ | United States| 100| +-----------------+----------+

[@shivakrishna1743]

Where can I get the flight_data.csv file? Please help.

[@shivakrishna1743]

Got the file, thanks

[@3mixmusic564]

Guru ibutton khi nhi aaya na idhr na udhr😂😂😂

[@manish_kumar_1]

Bhaari mistake ho gaya 😂

[@Tushar0797]

bhai please vo extra job kese create hua ye doubt clear krdo

[@manish_kumar_1]

Aap sql tab me jaake dekho. Kitne jobs skip hue hai. And share me your code and screenshot of the sql tab on LinkedIn or Instagram.

[@rohitgade2382]

@@manish_kumar_1 abe chutiya tere video ka bol Raha he wo 😂

[@amlansharma5429]

us_india_data = us_flight_data.filter((col("ORIGIN_COUNTRY_NAME") == 'India') | (col("ORIGIN_COUNTRY_NAME") == 'Singapore')) Ismein error bata raha hai : NameError: name 'col' is not defined Isko kaise define kare?

[@AliKhanLuckky]

Col ko import karna padenga voh ek function hai toh import functions karo I think so

[@manish_kumar_1]

Correct "from pyspark.sql.functtions import *"

[@vsbnr5992]

@@AliKhanLuckky NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark please I stuck here

[@vsbnr5992]

@@manish_kumar_1 NameError: name 'flight_data_repartition' is not defined what to do in this case even i import functions and types from pyspark please I stuck here

[@navjotsingh-hl1jg]

bhai iski file de do iss lecture ki

[@Tanc369]

csv kaha milegi sir ?

[@manish_kumar_1]

2 playlist hai. Parallely dekhiye. Practical wale me data milega description me usko copy karke save Kar lijiye as csv

[@manish_kumar_1]

Directly connect with me on:- topmate.io/manish_kumar25

[@PARESH_RANJAN_ROUT]

App kar pate ho toh, mein bhi kar sakta Manish Bhai

[@aishwaryamane5732]

Hi sir.. In which video series u have explained about schema @manish_kumar_1