Spark Join and shuffle | Understanding the Internals of Spark Join | How Spark Shuffle works

Spark Programming and Azure Databricks ILT Master Class by Prashant Kumar Pandey - Fill out the google form for Course inquiry.
forms.gle/Nxk8...
-------------------------------------------------------------------
Data Engineering using is one of the highest-paid jobs of today.
It is going to remain in the top IT skills forever.
Are you in database development, data warehousing, ETL tools, data analysis, SQL, PL/QL development?
I have a well-crafted success path for you.
I will help you get prepared for the data engineer and solution architect role depending on your profile and experience.
We created a course that takes you deep into core data engineering technology and masters it.
If you are a working professional:
1. Aspiring to become a data engineer.
2. Change your career to data engineering.
3. Grow your data engineering career.
4. Get Databricks Spark Certification.
5. Crack the Spark Data Engineering interviews.
ScholarNest is offering a one-stop integrated Learning Path.
The course is open for registration.
The course delivers an example-driven approach and project-based learning.
You will be practicing the skills using MCQ, Coding Exercises, and Capstone Projects.
The course comes with the following integrated services.
1. Technical support and Doubt Clarification
2. Live Project Discussion
3. Resume Building
4. Interview Preparation
5. Mock Interviews
Course Duration: 6 Months
Course Prerequisite: Programming and SQL Knowledge
Target Audience: Working Professionals
Batch start: Registration Started
Fill out the below form for more details and course inquiries.
forms.gle/Nxk8...
--------------------------------------------------------------------------
Learn more at www.scholarnes...
Best place to learn Data engineering, Bigdata, Apache Spark, Databricks, Apache Kafka, Confluent Cloud, AWS Cloud Computing, Azure Cloud, Google Cloud - Self-paced, Instructor-led, Certification courses, and practice tests.
========================================================
SPARK COURSES
-----------------------------
www.scholarnes...
www.scholarnes...
www.scholarnes...
www.scholarnes...
www.scholarnes...
KAFKA COURSES
--------------------------------
www.scholarnes...
www.scholarnes...
www.scholarnes...
AWS CLOUD
------------------------
www.scholarnes...
www.scholarnes...
PYTHON
------------------
www.scholarnes...
========================================
We are also available on the Udemy Platform
Check out the below link for our Courses on Udemy
www.learningjo...
=======================================
You can also find us on Oreilly Learning
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
=========================================
Follow us on Social Media
/ scholarnest
/ scholarnesttechnologies
/ scholarnest
/ scholarnest
github.com/Sch...
github.com/lea...
========================================

[@ScholarNest]

Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/

[@rishigc]

Hi, your videos are very interesting. Could you please provide me the URL of the video where you discuss Spark UI ?

[@duckthishandle]

I have to say that your explanations are better than the actual trainings provided by Databricks/Partner Academy. Thank you for your work!

[@Manapoker1]

one of the best if not the best video I've seen explaining joins in spark. Thank you!

[@davidezrets439]

Finally a clear explanation to shuffle in Spark

[@MADAHAKO]

BEST EXPLANATION EVER!!! THANK YOU!!!!

[@MegaSb360]

The clarity is exceptional

[@akashhudge5735]

Thanks for sharing the information, very few people knows the internals of the spark

[@vincentwang6828]

Short, informative and easy to understand. Thanks.

[@SATISHKUMAR-qk2wq]

Love you sir . I joined the premium

[@mertcan451]

Awesome easy explanation thanks!

[@chetansp912]

Very clear and crisp..

[@TE1gamingmadness]

When we'll see the next part of this video on Tuning the join operations ? Eagerly waiting for that.

[@umuttekakca6958]

Very neat and clear demo, thanks.

[@harshal3123]

Concept clear👍

[@plc12234]

really good one, thanks!!

[@mallikarjunyadav7839]

Amazing sir!!!!!

[@sudeeprawat5792]

Wow what an explanation ✌️✌️

[@sudeeprawat5792]

One question i have while reading the data in dataframe. Data is distributed across the executor on the basis of algorithm or randomly distributed across executor??

[@npl4295]

I am still confused about what happens in the map phase.Can you explain this "Each executor will map based on the join key and send it to an exchange. "?

[@tanushreenagar3116]

Nice

[@hmousavi79]

Thanks for the nice video. QQ: When I read from S3 with a bunch of filters on (partitioned and non-partitioned) columns, how many Spark RDD partitions should I expect to get? Would that be different if I use DataFrames? Effectively, All I need to achieve is to read from a massive dataset (TB+), perform some filtering, and writing the results back to S3. I'm trying to optimize the cluster size and number of partitions. Thank you.

[@fernandosouza2388]

Thanksssss!!!!

[@nebimertaydin3187]

do you have a video for sort merge join?

[@akashhudge5735]

one point you mentioned that if the partitions from both the dataframe is present in the same Executor then shuffling doesn't happen. but as per the other sources one task work on single partition hence even if we have required partition on the single executor still they are many partitions of the dataframe which contains the required join key data e.g. ID=100. Then how join is performed in this case.

[@meghanatalasila1309]

can you please share video on Chained Transformations?

[@WilliamBonnerSedutor]

What if the number of shuffle partitions is too much bigger than the number of nodes ? In the company I've just joined, they run the spark-submit in the developer cluster using 1 node, 30 partitions, 8GB each and shuffle partitions = 200. Maybe this 200 partitions can slow everything. The datasets are by the order of hundreds of GB

[@WilliamBonnerSedutor]

I'm not quite sure if I understood something: an exchange / shuffling in Spark is always basically a map-reduce operation ? ( so it uses the HDFS ?) Am I mixing things or am I right ? Thank you so much!

[@sanjaynath7206]

What would happen if the shuffle.partition is set to > 3 but we have only 3 unique keys for join operation? please help.

[@chald244]

The courses are quite interesting. Can I get the order in which I an take Apache Spark courses with my monthly subscription.

[@ScholarNest]

Follow the playlist. I have four Spark playlists. 1. Spark programming using Scala. 2. Spark programming using Python. Finish one or both depending on your language preference. Then start one or both of the next. 1. Spark Streaming in Scala 2. Spark Streaming in Python. I am hoping to get some more playlists in near future.

[@star-302]

Keeps repeating himself it’s annoying