Spark Session vs Spark Context | Spark Internals

[@arundhingra4536]

Very useful video. I have been working with spark for more than two years now but never really bothered about SparkSession vs SparkContext. For me its just the entry point and you go from there. But the idea of having multiple sparkSessions with a single underlying SparkContext makes great sense and was an eye opener. Thanks

[@The1Rafvas]

I would say that this is by far the best explanation I have found after hours of search on the topic. Congrats!!!

[@swatisneha9393]

Today i understood exact meaning of sparkContext and sparkSession. Thanks a lot, your video helped!!!

[@DataSavvy]

Thanks... I am happy that our is useful... Please provide your feedback on other videos of this channel

[@randommoments1263]

Detailed, Clear and straightforward, all at the same time. Superb..!

[@VivekKBangaru]

very informative one. Thanks Buddy.

[@rakeshchaudhary8255]

still relevent as of today and frequently asked. The practical on databricks made things crystal clear.

[@saurav0777]

In the older version of spark like spark 1.6 we had the entry point in the spark application created using spark context sc ,but the later version like spark 2.0 spark context has been deprecated and added all the context in one level above the abstraction and added them in spark session which contains all the spark context,spark-sql context,hive context etc..

[@shayshaswishes373]

can you please post the vedio , how to add SparkSession.builder in existing code.

[@Smoked93POL]

Short and to the point. I like your explanation.

[@Pramodkumar-mn6jd]

Nice ans it was very clear explaination, thanku sir 🙏

[@meenakshisundaraar7267]

Neat and clean presentation... 😊

[@DataSavvy]

Thanks a lot 😊

[@unmeshasreeveni]

The best explanation. Congrats

[@amitSingh-je8vh]

Congrats or Thanks 😅

[@kushagra_nigam95]

Best explanation till date 👍

[@DataSavvy]

Thanks Kushagra :)

[@jayashankarnallam6945]

What is the advantage of creating multiple spark sessions instead of having multiple spark contexts.

[@meenakshisundaraar7267]

Think spark context as server And spark session as client

[@kneelakanta8137]

Very good information, can you please help in clarifying this doubts: 1. What are included in configurations and properties of different spark sessions of the spark context, and it's effect on cluster 2. What is purpose of spark context and for what spark context is responsible for can you make a video to understand the spark context in full fledge?

[@SatishKumar-yz4tn]

Nicely explained. Thank you!!

[@anandhusk7794]

very clear and simple explanation. Thanks :)

[@truptiwaghmare3376]

What if same table was being updated by two users at a time..which one would be updated,let's say if we change the datatype of column rename it to same as previous column and store it to table..back again..and by table I mean global table

[@saurabhgarud6690]

Very useful stuff thank you so much

[@ravulapallivenkatagurnadha9605]

Nice

[@saurabh7337]

Under which scenarios it will be meaningful to have separate sparkContext for each user?

[@tirupatiraosambangij607]

Nice explanation.. Thank you

[@DataSavvy]

Thanks for appreciation :)

[@nashaeshire6534]

Thx a lot, really clear.

[@kthiru5168]

Nice Explanation.

[@priyachauhan813]

Hi , Thanks for nice explanation, Scala works with datasets and python with dataframes and they both generate RDDs as end Results, is my uinderstanding correct

[@SagarSingh-ie8tx]

Yes it’s nice

[@rahulberry4806]

thanks, clearly explained

[@narendrak36]

This is what exactly I am looking for. Niw I got to know exact difference between Context and session. Thank you dude. Do you know which is the best certification on spark as a Spark developer?

[@snehakavinkar2240]

Sorry, but I am little confused here. What do you mean when you say every spark context represents one application? When I submit a spark application aren't I am the only user who is attached to that application. How do multiple users make configuration changes to my spark application? Don't they have to submit their own copy of spark application again with config they wish to set? Thank you!

[@DataSavvy]

Imagine where u have already running app on cluster. Whatever code needs to be run you are getting at run time... That will be good use case for multiple spark sessions... Drop me an email at [e-mail]. will share more content to read on this

[@N12SR48SLC]

stackoverflow.com/questions/52410267/how-many-spark-session-to-create#:~:text=4%20Answers&text=No%2C%20you%20don't%20create,in%20the%20same%20spark%20job. Why is it saying 1 SS per application then?

[@nandhannandhan8155]

Superb

[@NiRmAlKuMaRindia]

Great details

[@gurumoorthysivakolunthu9878]

Hi Sir... Very useful topic and very well explained... Thank you, Sir... 1. "Each user can have different spark session... " -- Does this mean -- Different jobs submission...? That means only one Spark Context for the entire cluster which handles many jobs... Right...? 2. Then what about Driver... Is it similar to Spark Context... Only one Driver for all jobs...? 3. In the demo you showed creating many spark sessions in the same job... Each sessions are different within the same job itself... Am I right...? But why creating different sessions in the same code / job...? Thank you, Sir...

[@jasbirkumar7770]

sir can you tell me some about housekeeping executive spark deta. i dont understand spark word. facility company JLL requird he have spark exprience

[@priteshpatel4316]

Hi Harjeet thanks for the clear and simple explanations of all your videos. Can you upload videos serial wise pyspark tutorial if you have because in most of the tutorials around its starts with creation of spark dataframe using Sparksession and operations on dataframe. You can also suggest any tutorial/blog to read regarding pyspark. Thanks Man....your explanation are great

[@alphacharith]

Thank You

[@dineshedala]

In one of my interviews I faced this question. What happens if the executor got crashed unexpectedly which has already processed 50 records. Will it continues from 51 or from 0? Do we have any service that tracks the execution status of a executor?

[@rahuldey1182]

yes by creating checkpoint and mentioning the checkpoint folder location in ur program

[@indiannewyorkbabies6872]

Doesn’t rdds store those lineage information and when does the executor fails, the rdds gives that info to another new executor and starts the execution…!! Thtsy rdds ade fault tolerant

[@rvalusa]

Thanks, Sir for a wonderful video explaining the differences. one qq, when we close/stop a sparkSession which is created from a sparkContext, then this makes other sparkSessions as well get stopped which are created from the same sparkContext?

[@rvalusa]

found this, which is weird implementation and apparently a bug in spark - apache-spark-developers-list.1001551.n3.nabble.com/Closing-a-SparkSession-stops-the-SparkContext-td26932.html

[@ankan1627]

so what happens when different users create their own spark context. ( say before spark session was introduced) ? are multiple spark contexts created in such cases ? if yes, what are we gaining by moving the abstraction away from spark context to spark session ?

[@ajithkannan522]

only one spark context avaialable. You can create multiple sparkSessions under the spark context

[@krish808]

excellent content in a simple and easy format. Are you providing any trainings on databricks? if so, how do I contact you

[@ravikrish006]

Can we have multiple contexts. Could you show with some examples

[@sudheeryarramaneni2218]

I have a doubt,Can we apply actions directly on RDD with out transformations?

[@DataSavvy]

Loading of a file and creating a rdd is also a transformation... So logically you cannot run action without transformation... If you you don't count creating an rdd as transformation, then you can say that you run action action without transformation

[@phanikumar4915]

can we call stop on spark session what will happen if we call.

[@karthikram1954]

Great video sir. Just one question. In which node does the spark context and spark session run?

[@srikanthchillapalli1037]

I don't think we can create multiple spark context in spark 1.x as well. There is a parameter spark.driver.allowMultipleContexts=true, but this is only used on test scripts but cannot be used to create multiple contexts when coding in IDE. And in spark 2.x we will create multiple spark sessions. Please let me know if I'm wrong.

[@murifedontrun3363]

There can be only one SparkContext per JVM process. If there would have been multiple SCs running in the same JVM then it would be very difficult to handle GC tuning up, communication overhead among the executors etc.

[@srikanthchillapalli1037]

@@murifedontrun3363 Yes, but here in video the tutor explained that in old versions multiple spark contexts were created. So, I got a doubt how it is possible.

[@vijaybigdata752]

I have a doubt, in this scenario if we have 4 spark sessions for a single spark context, when spark context goes down will all 4 spark sessions killed? Please confirm.

[@DataSavvy]

Yes Vijay...all spark session will be killed

[@deepakkini3835]

I had an interview and he asked me on spark process. Could you please explain what happens when the spark job is stopped in midway of execution? Will it start from the beginning or from where it left off?

[@DataSavvy]

It depends on how the job was stopped... Do you mean that you killed spark context and stopped or only the job running action had failed... Recovery will depend on this...

[@DataSavvy]

It will also depend on if you have any checkpoints in your job

[@vijaydas2962]

Very informative content... I've a doubt.... I opened 2 separate spark2-shell using 2 different Ids... When I hit spark.sparkContext in two different terminals, the reference numbers were different. Shouldn't they be same as you explained at the beginning of this video where multiple users shared the same sparkContext object?

[@atnafudargaso8374]

same here

[@yashdeepkumar2495]

He is talking when working in a clustered environment with more than one worker node I think...usually that will be the scenario. If you open 2 spark shells and check it will create two seperate contexts.I am new to this and pls let me know if you found the correct ans to your question after two years.

[@sandeshhegde2847]

If you open 2 shells, they're 2 different applications. this video talks about having multiple spark sessions within a single application

[@rajrajan51]

Thanks for the video bro . I have doubt suppose user 1 is sharing table 1 and user 2 is updating a value for the column in the table 1 will the change also got update user 1 shared table too.

[@srikanthchillapalli1037]

It wont happen as user1 and user2 will have isolated sessions from one another and so one user operation doesnt have any impact on other user table. Actually u can have different data for both these users though the table name is same.

[@sachinhugar]

Hi harjeet, when this type of use case come any example bcz in batch processing there will one spark session is enough.

[@DataSavvy]

When u want your users to have live connection for data analysis etc

[@souravsardar]

@datasavvy thanks for the video. Could you please make a video on where we can practice production level scenarios in pyspark .

[@DataSavvy]

Sure Saurav... Let me know if you have list of scenario which u want me to cover. Drop me a email at [e-mail]

[@max6447]

By executor do u mean node manager?

[@projjalchakraborty1806]

Hi harjeet..why we are using multiple sparksession instead of multiple sparkcontext....any advantage is there???

[@DataSavvy]

It makes it easier to share tables and share cluster resources among your users... As you well know starting different application for each user usually cause cluster contention

[@vijaypandey5371]

What will happen if driver program fails in Spark. And how to recover it?

[@DataSavvy]

It depends on what settings you have for that job. If you have checkpoints and retry enabled. spark will start to recreate those objects... otherwise the job will fail..

[@santhoshsandySanthosh]

What is the tool or software that is used in this demo of creating sessions. . is it python based or scala ?

[@vamshi878]

Hi harjeet, can you make a video for how to read hbase table data into spark dataframe, and how to insert spark dataframe into hbase table. is there any spark-hbase connector available for cloudera?

[@DataSavvy]

Sure Vamsi... I will add this in my to-do list... Thanks for suggestion :)

[@rohi1350]

We can something SparkSession is similar to sqlContext in spark 1.6

[@sreepaljsp]

you said some thing at 2:39.. i did not get that word.. the sentence is "I can ___ a spark context per user" what is that missing word?

[@srikd9829]

I think the missing word is: Spun. its the past tense of the word: Spin. generally the word is used as 'spun a server' means different meanings like "introducing a new server or node, or starting or booting the server or node". This is because, starting or booting the server, spins the hard-disk to load the OS. This is how the word came into practise. hope this helps.

[@TheVijju89]

Please post the python code sheet..

[@mohdrayyankhan6623]

What are the users here ???

[@mineb1842]

Plz

[@MyTravelingJourney]

I think you missed many points

[@DataSavvy]

Please suggest what are you pointing Towards... Will cover in another video

[@dalwindersingh9282]

please suggest, raise few of them.

[@underlecht]

3 ads in 5 minutes. did not finish.

[@DataSavvy]

KZbin has increased the ads based on user stats. I unfortunately don't have method to decrease that. Any suggestion is welcomed (except a complete switch off)