Apache Spark Tutorial | Spark Tutorial for Beginners | Apache Spark Training | Edureka

[@edurekaIN]

Got a question on the topic? Please share it in the comment section below and our experts will answer it for you. For Edureka Apache Spark Certification Training Curriculum, Visit the website: bit.ly/2KHSmII

[@arunasingh8617]

Well explained the concept of Lazy Evaluation!

[@edurekaIN]

Good To know our videos are helping you learn better :) Stay connected with us and keep learning ! Do subscribe the channel for more updates : )

[@sarthakverma5921]

his teaching is pure gold

[@AdalarasanSachithanantham]

1st time really impressed in how the way you are teaching God bless you

[@draxutube]

SO GOOD TO WATCH I UNDERSTANDED SO MUCH

[@leojames22]

One of the best video I ever watched. MapReduce was not explained in this way wherever i checked. Really thank you to post this. Use Cases are really good. Worth the time watching almost 2 hrs. 5 star to you the instructor. Very impressed.

[@moview69]

you are undoubtedly the king of all instructors...you rock man

[@vinulovesutube]

Before starting this session I had no clue of Bigdata nor Spark . Now I have pretty decent insight . Thanks

[@edurekaIN]

Thank you for watching our videos and appreciating our work. Do subscribe to our channel and stay connected with us. Cheers :)

[@arunasingh8617]

I have a question here, if we have almost 60M data then creating RDD while processing the data will helps in handling such huge data or some other processing steps required?

[@kag1984007]

So far this is 4th course I am watching, Instructors from Edureka are amazing. Very well explained RDD in first half. Worth watching !!!

[@edurekaIN]

Hey Kunal, thanks for the wonderful feedback! We're glad we could be of help. We thought you might also like this tutorial: kzbin.info/www/bejne/q3XComeIopmcaLM. You can also check out our blogs here: www.edureka.co/blog Do subscribe to our channel to stay posted on upcoming tutorials. Cheers!

[@daleoking1]

This makes things more clear after my Data Science class lol. Thank you so much for a great tutorial, I think this will sharpen me up.

[@edurekaIN]

Hey, thank you for watching our video. Do subscribe and stay connected with us. Cheers :)

[@Successtalks2244]

I love this edureka tutorials very much

[@mix-fz7ln]

Awesome session! Hats off to the instructor, I was searching hard to understand spark and nothing pop up to me and explained this session amazing I love how the instructor clarify every concept and frames you are amazing!

[@edurekaIN]

Good to know our contents and videos are helping you learn better . We are glad to have you with us ! Please share your mail id to send the data sheets to help you learn better :) Do subscribe the channel for more updates 😊 Hit the bell icon to never miss an update from our channel

[@yitayewsolomon4906]

thanks very much, I'm biggner for data science i got clear explanation for spark thanks alooot.

[@edurekaIN]

Thank you so much for the review ,we appreciate your efforts : ) We are glad that you have enjoyed your learning experience with us .Thank You for being a part of our Edureka team : ) Do subscribe the channel for more updates : ) Hit the bell icon to never miss an update from our channel : )

[@kadhirn4792]

Great Video. He is my tutor for ML

[@ranjeetkumar2051]

thank you sir for making this video

[@edurekaIN]

Most welcome

[@ramsp35]

This one of the best and simplified Spark tutorial I have come across. 5 stars...!!!

[@edurekaIN]

Thank you for appreciating our efforts, Ramanathan. We strive to provide quality tutorials so that people can learn easily. Do subscribe, like and share to stay connected with us. Cheers!

[@nileshdhamanekar4545]

Awesome session! Hats off to the instructor, you are amazing! The RDD explanation was the best

[@edurekaIN]

Hey Nilesh, we are delighted to know that you liked our video. Do subscribe to our channel and stay connected with us. Cheers :)

[@krutikachauhan3299]

It was a totally new topic for me.. but still I was able to grasp it easily. Thanks to the whole team.

[@edurekaIN]

Hey:) Thank you so much for your sweet words :) Really means a lot ! Glad to know that our content/courses is making you learn better :) Our team is striving hard to give the best content. Keep learning with us -Team Edureka :) Don't forget to like the video and share it with maximum people:) Do subscribe the channel:)

[@hymavathikalva8959]

Very helpful section. Now I have some idea on hadoop. Nice explanation sir. Tq

[@edurekaIN]

Thank you so much : ) We are glad to be a part of your learning journey. Do subscribe the channel for more updates : ) Hit the bell icon to never miss an update from our channel : )

[@suvradeepbanerjee6801]

Great tutorial. Really explained things up! thanks a lot

[@edurekaIN]

You're welcome 😊 Glad you liked it!! Keep learning with us..

[@sahanashenoy5895]

Amazing way of explanation. crystal clear.. way to go edurekha

[@ashishpaul8557]

Thank You @ Edureka for doing such excellent work.

[@nshettys]

Brilliant Explanation!!! Thank you

[@srividyaus]

This is the best spark demo I have ever heard. Very clear and planned way of explaining things! Have taken up Hadoop basics classes with Edureka, which are great! Planning to enroll for spark as well. Would you explain more realtime use cases in spark training? Hadoop basics doesn't have use case explanation, which is the only drawback of the course! Great going , thanks a lot for this video.

[@edurekaIN]

+Srividyaus thanks for the thumbs up! :) We're glad you liked our tutorial and the learning experience with Edureka! We have communicated your feedback to our team and will work towards coming up with more real time use case videos on top of existing hands-on projects. Meanwhile, you might also find this video relevant: kzbin.info/www/bejne/sJanhquVf8tka5Y. Do subscribe to our channel to stay posted on upcoming videos and please feel free to reach out in case you need any assistance. Cheers!

[@shamla08]

Very detailed presentation and a very good instructor! Thank you!

[@JanacMeena]

Jump to 21:25 for the example

[@moneymaker2328]

Excellent session no words to describe anything about it ...trainer is too good...worth watching

[@edurekaIN]

Hey Apurv, thank you for watching our video and appreciating our effort. Do subscribe and stay connected with us. Cheers :)

[@ajanthanmani1]

1.5x for people who don't have 2 hours to watch this video :)

[@edurekaIN]

Whatever rocks your boat, Ajanthan! :) Since we have learners from all backgrounds and requirements we make our tutorials as detailed as possible. Thanks for checking out our tutorial.Do subscribe to stay posted on upcoming tutorials. We will be coming up with shorter tutorial formats too in the future. Cheers!

[@taniakhan71]

thank you so much for this wonderful tutorial.. I have a question.. while discussing about lazy evaluation, you mentioned that for B1 to B6 RDD memory is allocated, but they remain empty till collection is invoked. My qs is.. what is the size size of the memory that is allocated for each RDD? How does the framework predict the size before hand for each RDD without processing the data? eg, B4, B5 , B6 might have different sizes and smaller or equal to B1, B2, B3 respectively... I didn't get this part. Could you please clarify?

[@edurekaIN]

What is the size of the memory that is allocated for each RDD? 1. There is no easy way to estimate the RDD size and approximate methods were used in Spark Size Estimator's methods). 2. By default, Spark uses 60% of the configured executor memory (--executor- memory) to cache RDDs. The remaining 40% of memory is available for any objects created during task execution. In case your tasks slow down due to frequent garbage-collecting in JVM or if JVM is running out of memory, lowering this value will help reduce the memory consumption. How does the framework predict the size before hand for each RDD without processing the data? 1. One can determine how much memory allocated to each RDD by looking at the Spark Context logs on the driver program. 2. A recommended approach when using YARN would be to use --num-executors 30 --executor-cores 4 --executor-memory 24G. Which would result in YARN allocating 30 containers with executors, 5 containers per node using up 4 executor cores each. The RAM per container on a node 124/5= 24GB (roughly). Hope this helps :)

[@laxmipriyapradhan8087]

thank u sir , i just love ur teching style. is there any other vdos of urs in youtube, plz give that link

[@edurekaIN]

Hi Laxmipriya glad to hear this from you please feel free to visit our channel for more informative videos and don't forget to subscribe to get notified on our new videos

[@rmuru]

Excellent session...very informative..trainer is too good and explained all concepts in detail...thanks lot

[@niveditha-7555]

Wow!! extremely impressed with this explanation

[@seenaiahpedipina1165]

Good explanation and useful tutorial. Conveyed a lot just in two hours. Thank you edureka !

[@edurekaIN]

Hey Srinu, thanks for the wonderful feedback! We're glad we could be of help. Here's another video that we thought you might like: kzbin.info/www/bejne/q3XComeIopmcaLM. Do subscribe to our channel to stay posted on upcoming tutorials. Cheers!

[@SkandanKA]

Nice, brief explanation @edureka. Keep going with more such good tutorials.. 👍

[@gyanpattnaik520]

Its an amazing video . Gives a complete concept of spark as well as its implementation in real world. Thanks

[@sagnikmukherjee5108]

Its an awesome session. The way you explain everything with examples, its remarkable. Thanks mate.

[@edurekaIN]

Thanks for the wonderful feedback! We are glad we could help. Do subscribe to our channel to stay posted on upcoming tutorials.

[@pritishkumar6514]

Loved the way how the trainer explained about it. Watched for the first time and it cleared all my doubts. Thanks, edureka.

[@edurekaIN]

Thanks for the compliment Pritish! we are glad you loved the video. Do subscribe to the channel and hit the bell icon to never miss an update from us in the future. Cheers!

[@shubhamshingi9618]

wow. Such an amazing content. Thanks, edureka for this

[@puneethunplugged]

Thank you for the crisp session. Good content and flow. Appreciate it.

[@deepikapatra1065]

amazing video! too much concepts got cleared in just 2 hours:)Keep up the good work,edureka!

[@darisanarasimhareddy4311]

I completed hadoop coaching few days back. I would like to learn spark and scala .Is this 39 videos good enough for Spark AND Scala Training?

[@edurekaIN]

+Darisa NarasimhaReddy, thanks for choosing Edureka to learn Hadoop. About your query, these tutorials will give you a basic introduction to Spark but you will miss out on the hands-on components, assignments & doubt clarification since these are pre-recorded sessions. We'd suggest that you take up our Spark course as the next step in your learning path since Hadoop + Spark will give you tremendous career growth. Would you like us to get in touch with you and assist you with your queries? Hope this helps. Cheers!

[@iiitsrikanth]

Good work Edureka Team! Really Helpful to the beginners.

[@areejabdelaal4446]

thanks a lot!

[@Yashyennam]

This is top notch 👍👍👌

[@nagendrag2441]

Explanation is very good.. Thank you Now i understood overview completely...

[@nikitagupta6174]

Hi, I have few questions: 1.) Its about difference between Hadoop and Spark, you told that there are lot of i/p o/p operations in hadoop whereas in spark you said it happens only once when blocks are copied in memory and rest of the operations are performed in memory itself, so i wanted to ask when entire operation is completed so i/p o/p operation might be again required to copy the result to disk or result stays in memory itself in case of spark? 2.) Also, when we use map and reduce functions in spark python, how does those things works then? All the map operations are done in memory like that of hadoop? but what about reduce thing as reduce will merge result of two blocks so, don't you think that again network overhead will occur when we pass data from another disk to the disk in which we need to do reduce operation and the that disk will again copy that data to its memory? Can you explain how exactly it will work in case of spark?

[@edurekaIN]

Hey Nikita, thanks for checking out our tutorial! Here are the answers to your questions: 1. Spark doesn't work in a strict map-reduce manner and map output is not written to disk unless it is necessary. To disk are written shuffle files. It doesn't mean that data after the shuffle is not kept in memory. Shuffle files in Spark are written mostly to avoid re-computation in case of multiple downstream actions. The difference between Spark storing data locally (on executors) and Hadoop MapReduce is that: i.The partial results (after computing ShuffleMapStages) are saved on local hard drives not HDFS which is a distributed file system with a very expensive saves. ii.Only some files are saved to local hard drive (after operations being pipelined) which does not happen in Hadoop MapReduce that saves all maps to HDFS. 2. Also, when we use map and reduce functions in spark python, The Spark Python API (PySpark) exposes the Spark programming model to Python (Spark Programming Guide). PySpark is built on top of Spark's Java API. Data is processed in Python and cached / shuffled in the JVM. 3. All the map operations are done in memory like that of hadoop? Yes, all the operations will be done in memory only and all the reduce operations also will be done in the same way as Hadoop because data is processed in Python and cached / shuffled in the JVM. Hope this helps. Cheers!

[@gaurisharma9039]

Kindly put a video on spark pipelining. I would really appreciate that. Thanks much in advance

[@chandraiyer9835]

Very nicely explained Thank You :)

[@edurekaIN]

You are welcome👍 You can also check out our complete training here: Apache Spark and Scala Certification Training - bit.ly/3bWTxQh Use code "KZbin20" to get Flat 20% off on this training.

[@theabhishekkumardotcom]

Thank you for the quick introduction on the architecture of spark....

[@edurekaIN]

Hey Abhishek, Thank you for watching our video. Do subscribe, like and share to stay connected with us. Cheers :)

[@chandan02srivastav]

Very well explained!! Amazing Tutor

[@ManishKumar-ni4pi]

The way of representation is wonderful. Thank you

[@edurekaIN]

Thanks for the compliment Manish! We are glad you loved the video. Do subscribe to the channel and hit the bell icon to never miss an update from us in the future. Cheers!

[@ankitas7293]

This Shivank sirs voice .. he is a very very good trainer.

[@hemanthgowda5855]

Good lecture. An action is a trigger for lazy eval to start right? .collect() is not equivalent to printing..

[@edurekaIN]

Hey Hemanth, sorry for the delay. Yes, action is a trigger for lazy evaluations to start. To print all elements on the driver, one can use the collect() method to first bring the RDD to the driver node. Hope this helps!

[@sasikumar-gp9zd]

hi , useful information ...What are the pre-requisite to learn Apache Spark and Scala ? is it useful to a fresher to do this course

[@edurekaIN]

+Sasi Kumar, thanks for checking out our tutorial! To learn Spark, a basic understanding of functional programming and object oriented programming will come in handy. Knowledge of Scala will definitely be a plus, but is not mandatory. Spark is normally taken up by professionals with some knowledge of Hadoop. You could either up-skill with Hadoop and then follow the learning path to Apache Spark and Scala or you can directly take up Spark training. Hadoop basics will be touched upon in our Spark training also. You can find out more about our Hadoop training here: www.edureka.co/big-data-and-hadoop and learn more about our Spark training here: www.edureka.co/apache-spark-scala-training. Hope this helps. Cheers!

[@coolprashantmailbox]

very useful video for beginners.. awesome.thank u

[@tsuyoshikittaka6636]

wonderful tutorial ! thank you :)

[@sunithachalla7840]

awesome session...

[@ajiasahamed8814]

Excellent session.. Trainer is fantastic and attitude . Edureka.. You are amazing in online coaching.

[@joa1paulo_]

Thanks for sharing!

[@girish90]

Excellant session!

[@safiaghani4078]

Hi, It is very much informative lecture ...I have a plan to write my thesis in apache spark ...could please suggest me good topic ..please it will be a great help thanks.

[@edurekaIN]

Hey! You can refer to this thread on quora: www.quora.com/I-want-to-do-my-thesis-in-Apache-Spark-What-are-a-few-topics-or-areas-for-that Hope this helps. Cheers :)

[@foradvait7591]

Excellent. Dear trainer sir, you have amazing hold on Spark concepts. Regards

[@bobslave7063]

Thanks, for amazing tutorials! Very well explained.

[@rajashekarpantangi9673]

Very good Explanation. Awesome content. I have a question. When Map function is executed the results are given as a block in memory. This is fine. In the example provided in the video, the map function doesn't require any further computation( since the job is to take numbers less than 10). What about for a job like Word count. 1. How would the output of the map function be? Is it same as Map function in MapReduce (apple,1 (apple,1) (apple,1) (banana,1),(banana,1),(banana,1),(orange,1),(orange,1),(orange,1))? Or we can write the code for reducing also in the same map function giving output as ((apple,3) (orange,3)(banana,3))?? 2. And are the blocks from each data node will be sent to a single data node to execute the further computation?? (as in reduce in map reduce)?? Thanks in Advance

[@edurekaIN]

Hey Rajashekar, thanks for the wonderful feedback! We're glad you liked our tutorial. This error (Unsupported major.minor version) generally appears because of using a higher JDK during compile time and lower JDK during runtime. The default java version and you Hadoop's java version should match. For java version type in terminal >java -version this will display your current java version. For knowing the java version used by hadoop you will have to find hadoop-env.sh(in etc folder) file which contains an entry for JAVA_HOME like "export JAVA_HOME = /usr/lib/jvm/jdk1.7.0_67" or something like that. If the version of java shown by both the command are different or your hadoop-env.sh file are different then this error arises. Try setting JAVA_HOME to the path of jdk correctly to the version shown by java -version. Hope this helps. Cheers!

[@rajashekarpantangi9673]

I don't think u answered to my question. please read my question again and reply thanks.

[@edurekaIN]

Hey Rajashekar, here's the explanation: 1. Word count code in spark - map function is similar to hadoop mapreduce but not the same map(func) : Return a new distributed dataset formed by passing each element of the source through a function func. Consider the word count code - in scala val ip =sc. textFile("file:///home/edureka/Desktop/example.txt") // loading the sample example file val wordCounts = ip.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a + b) // flatmap splits the word according to space delimiter and map, here assigns each word,a value of 1 and reduceByKey will add up the values having same key ie words here wordCounts.collect // this will give output as given below: res: Array[(String, Int)] = Array((banana,2), (orange,6), (apple,4)) 2. As spark does in memory processing, only needed data is pushed to memory and processed .Here in the example flatmap , map and reduceByKey are transformation functions used , this will do Lazy evaluation, ie data will not be pushed immediately to memory/ram (transformation function will create a linage graph of RDD's) and when ever an action (collect in the code example) happens on final RDD - spark will use the lineage details and push the required data to memory Spark does not work like hadoop - blocks are not send to a single node for processing , instead , computation/processing will happen in memory of each nodes where needed data exists and aggregated result will be send to the spark master node / client. In this way spark is faster, there is no i/o disk operations as in hadoop. Hope this helps. Cheers!

[@rajashekarpantangi9673]

Thanks!!

[@nihanthsreyansh2480]

Cheers to Edureka ! Very Well explained . Please Upload " Using Python With Apache Spark " Videos too !!

[@edurekaIN]

Hey Nihanth, thanks for checking out our tutorial. We're glad you liked it. We do not have such a tutorial at the moment, but we have communicated your request to our team and we might come up with it in the future. Do subscribe to our channel to stay posted. Cheers!

[@nihanthsreyansh2480]

Thanks for the reply !

[@taniakhan71]

Thank you for the explanation.

[@edurekaIN]

Thank you for watching our video. Do subscribe, like and share to stay connected with us. Cheers :)

[@gurukasiviswanathyadavmeka6048]

Hi,I don't have knowledge on hadoop and I am willing to learn spark. Please let us know the details like when I join in your institute on spark course I can get a clear understanding on topics of spark or I need to do hadoop training first then after I have to take spark training???Also let us know the duration of courses and timing details.Kindly share the contact details as well

[@edurekaIN]

Hey Viswanath, thanks for checking out our tutorial! To learn Spark, a basic understanding of functional programming and object oriented programming will help. Knowledge of Scala will definitely be a plus, but is not mandatory. If you have these skills then you can go ahead and learn Spark. You can also master Hadoop and then go for Spark since it's a career path that's normally taken. You can check out our Spark course and batch details, curriculum etc here: www.edureka.co/apache-spark-scala-training. Hope this helps. Cheers!

[@dhruveshshah1872]

Loved your video. Explained the basic details in a best possible way. Would wait for your new videos on this topic..Can you share the github link for the earthquake project?

[@edurekaIN]

Hey Dhruvesh, thanks for checking out our tutorial. We're glad you liked it. Please check out this blog for the code: www.edureka.co/blog/spark-tutorial/ You can fill in your request on the google form in the blog. Hope this helps. Do subscribe to our channel to stay posted on upcoming tutorials. Cheers!

[@muhammadrizwanali907]

Excellent video from the tutor. Very well defined the concepts and technology. Really appreciable

[@edurekaIN]

Thank you for watching our video. Do subscribe, like and share to stay connected with us. Cheers :)

[@rahulmishra4111]

Great session ..very informative .. Can you please share the sequence of videos in Apache Spark and Scala learning playlist.. Thanks in advance

[@manishdev71]

Excellent session.

[@theinsanify7802]

Thank you very much this was an amazing course .

[@edurekaIN]

Thanks for the compliment, Mahdi! We are glad you loved the video. Do subscribe to the channel and hit the bell icon to never miss an update from us in the future. Cheers!

[@theinsanify7802]

@@edurekaIN i sure did .. can't miss these contents.

[@AdalarasanSachithanantham]

Superb 🎉

[@1203santhu]

Session is really fantastic and informative...

[@edurekaIN]

Hey Santhosh, thanks for checking out our tutorial! We're glad you found it useful. :) Here's another video that we thought you might like: kzbin.info/www/bejne/q3XComeIopmcaLM Do subscribe to our channel to stay posted on upcoming videos. Cheers!

[@umashankarsaragadam8205]

Excellent explanation ...Thank you

[@girishahb01]

Nicely explained, i am in the process of learning Machine learning algorithm in Python & R. I may have to learn Spark in future :)

[@edurekaIN]

Hey Girisha, thanks for checking out our tutorial. We're glad you found it useful. Please feel free to check out our Spark course here: www.edureka.co/apache-spark-scala-training. You can get in touch with us anytime if you need any information or assistance. Hope this helps. Cheers!

[@sagarsinghrajpoot3832]

Awesome video sir 🙂

[@tabitha3302]

Excellent Video, Super explanation , we want like real time examples and use cases,,worth it,Awesome

[@edurekaIN]

Hey Tabitha, thanks for the wonderful feedback! We're glad you found it useful. Do follow our channel to stay posted on upcoming tutorials. You can also check out our complete training here: www.edureka.co/apache-spark-scala-training. Hope this helps. Cheers!

[@kavyaa1053]

thanks for this video .

[@edurekaIN]

Hey Kavya, thank you for appreciating our work. Do subscribe and stay connected with us. Cheers :)

[@umeshsawant135]

Excellent session!! trainer is well experienced and good teacher as well..All the best edureka..

[@edurekaIN]

Thank you for watching our video. Do subscribe, like and share to stay connected with us. Cheers :)

[@rakesh4a1]

From where i can read this kinda core information about Spark and Hadoop....any links or way to find documents...

[@edurekaIN]

Hi Rakesh, please check this link www.edureka.co/blog/spark-tutorial/. Hope this is helpful.

[@rishinepal4371]

Best explanation !

[@edurekaIN]

Hey Rishi, glad you loved the video. Do subscribe to the channel and hit the bell icon to never miss an update from us in the future. Cheers!

[@deankommu3137]

nice video with a brief explanation

[@2007selvam]

It is very useful session.

[@edurekaIN]

+Rangasamy Selvam, thanks for checking out our tutorial! We're glad you found it useful. Here's another video that we thought you might like: kzbin.info/www/bejne/rn-kdWmZd7Csl6M. Do subscribe to our channel to stay posted on upcoming tutorials. Cheers!

[@kalpanagyara9982]

good & useful turorial.. Can I get a copy if the ppt that was used?

[@edurekaIN]

Hey Kalpana, thanks for checking out our tutorial! We're glad you found it useful. This exact PPT is not available, but you will find related material here: www.slideshare.net/EdurekaIN. You can also get access to our complete material and training when you enroll into our course here: www.edureka.co/apache-spark-scala-training. Hope this helps. Cheers!

[@manedinesh]

NIcely explained. Thanks!

[@edurekaIN]

Hey Dinesh, thank you for watching our video. Do subscribe, like and share to stay connected with us. Cheers :)

[@MrAK92]

awesome class..thank u sir for proving very useful information

[@edurekaIN]

Hey Arun! Thank you for the wonderful feedback. Do subscribe yo our channel and check out our website to know more about Apache Spark training : www.edureka.co/apache-spark-scala-training Hope this helps. Thanks :)

[@ainunabdullah2140]

Very good Tutorial

[@edurekaIN]

Hey Abdullah, thanks for the wonderful feedback! We're glad we could be of help. You can check out our complete Apache Spark course here: www.edureka.co/apache-spark-scala-training. Do subscribe to our channel to stay posted on upcoming tutorials. Hope this helps. Cheers!

[@nithingeorge517]

Hello. This is very informative. I think the resiliency concept which you explained here is a bit improper. Resiliency in Spark is with respect to the lineage and not the replication factor as RDD can be written at the speed of DRAM so there is no need to replicate each byte. Awaiting your reply.

[@edurekaIN]

Hey Nitin, thanks for checking out our tutorial! Yes, Resiliency in Spark is with respect to the lineage and not the replication factor . Spark creates a lineage. A lineage keeps track of what all transformations has to be applied on that RDD.Now imagine you are going to perform an action multiple times on the same RDD which has a long lineage, this will cause an increase in execution time. Caching stores the computed result of the RDD in the memory thereby eliminating the need to recompute it every time. You can think of caching as if it is breaking the lineage, but it does remember the lineage so that it can be recomputed in case of a node failure. So, it is needed to replicate each byte. Hope this helps. Cheers!

[@IsabellaYuZhou]

1:01:07

[@JarinTasnimAva]

very well-described ! Amazing !

[@edurekaIN]

Thank you for watching our video. Do subscribe, like and share to stay connected with us. Cheers :)

[@prabhathkota107]

Very well explained the overview of spark

[@JohnWick-zc5li]

In case if the File Size is 5GB or 10GB than how RDD would be helpful when there is less Memory.

[@edurekaIN]

Hey John, sorry for the delay. First of all you have distributed memory on different slave nodes, so you'll have good amount of memory. But still if the memory is full, then Spark will place the RDDs on disk. Hope this helps!

[@JohnWick-zc5li]

Good Jobs Guys thanks.

[@edurekaIN]

Thank you for watching our video. Do subscribe, like and share to stay connected with us. Cheers :)

[@vjtareddy]

Good explanation. May we know the Instructors name.

[@edurekaIN]

Hey Vijetha, thanks for checking out our tutorial! We're glad you liked it. Better than telling you the instructor's name, let us redirect you to his live classes. :) He will be taking an Apache Spark batch starting on 27th Feb. You can check out the details here: www.edureka.co/apache-spark-scala-training. Hope this helps. Cheers!

[@abhishekmane5885]

Thank you, it was very good explanation. Can you please share ppt used in video ?

[@edurekaIN]

Hey Abhishek, thanks for watching our spark tutorial! We're glad you found it useful. Here are the slides of this video: www.slideshare.net/EdurekaIN/apache-spark-tutorial-spark-tutorial-for-beginners-apache-spark-training-edureka Hope this helps. Cheers!

[@atindrabhattacharya2263]

I love shivank, He is awesome. thanks for this wonderful session

[@edurekaIN]

Hey Atindra, thank you for appreciating our trainers. We are glad that you found the videos helpful Do subscribe and stay connected with us. Cheers :)

[@krishna.s738]

Explanation is great. May i know the Instructor "name".

[@edurekaIN]

Hey Krish, thanks for the wonderful feedback! The instructor is Shivang and he has a batch coming up on 13th May, 2017. You can check out the batch details and our course curriculum here: www.edureka.co/apache-spark-scala-training. Please feel free to get in touch with us if you have any more questions or need assistance. Alternatively, you can also call us at 88808 62004; we're available 24X7 at this number. Cheers!

[@efgh7906]

great explianation and great session