How do nested loop, hash, and merge joins work? Databases for Developers Performance #7

[@khalidbadawi8216]

bro, not exaggerating, your way of teaching is legendary, keep making videos on different topics and ur views will reach the top.

[@PeterBogaert-fz1kn]

I'm an Oracle DBA for 30 years and your explanation is the best I ever seen !!! Congrats

[@TheMagicofSQL]

You're welcome Peter - glad you found this useful!

[@Variable.Constant.]

Hi, I am really surprised you got so less views. You have explained something I took years to understand. Subscribed.

[@TheMagicofSQL]

Thanks Kapil, glad this helped you understand :)

[@rahulbansal3811]

Exactly your videos are highly underrated on youtube

[@raobuctdeerf]

Very much agree your explanation was spot on ...and solid too..

[@VicUXR]

I feel so lucky to have come across this video on my second day of SQL home study. Looks like I saved myself years of mystification/confusion!

[@TheMagicofSQL]

You're welcome, glad this helped!

[@dalvandi]

Great explanation! A word of note for those who were also not understanding (like me) due to the suit system ; The suit strength goes clubs < diamonds < hearts < spades I just found out that apparently that there's two systems of suit strength : clubs < diamonds < hearts < spades (alphabetical) and diamonds < clubs < hearts < spades (alternating) I know the latter. I didn't know the alphabetical order even existed.

[@TheMagicofSQL]

I was just ordering alphabetically for this video! Thanks for digging out the other suit sorting method

[@nishanksoni7120]

1.Nested Join - nested for loop - O(N * N) - easiest to implement - time consuming for large dataset - better with either small datasets or index on join attributes 2.Merge Join - sort them first and compare - O(NlogN) + O(NlogN) - Efficient for large datasets - pre-join preparation required (sorting) - scanning of relation happens once while joining - can leverage indexes if available to make it faster 3.Hash Join - using hashing - not good for range queries - creating a hashtable for the query you are searching ( user_id is the key) - used for equi joins - efficient for large datasets - requires additional memory - pre join preparation required -> hash table construction - Hash function should distribute data evenly SQL Engine : Take a look at data , look statistics across all table using cardinality

[@TheMagicofSQL]

Nice summary. Though - at least in Oracle Database - hash join is worse than "not good" for range queries. The optimizer won't use it at all!

[@danishhussain8029]

This is a core question of DBA Interviews. Very well explained. Heartiest thanks Sir

[@TheMagicofSQL]

You're welcome; glad this helps!

[@sandhu01]

Although it should technically be a developer's question :)

[@galeop]

Great vidéo! My sum up: Definitions: [from other source]The Optimiser will decide which table will be the _inner_ or _outer_ table. - The outer table is the source of rows to match against the inner table. It is usually read from disk. - The inner table is the table that is probed for matches. It is usually held in memory, is usually the source table for hashing, and if possible, is the smallest table of the two being joined. Nested Loops: TL;DR : good if you only have a small subset of rows to join from the outer table, AND you have an index on the inner table. For each row in the outer table, it will look for all the row in the inner table. Without any index, you get a complexity of o(number of rows in Table1 * number of rows in Table2). This is very inefficient, unless you only have a small number of rows from the outer table to join (or a small subset of the outer table to join), AND you have an index on the joining column of the inner table. Thanks to the index of the inner table, the nested loop (that looks for the matching row in the inner table) will be fast. And as there are few rows triggering the nested loop, the whole operation will be pretty Note that if you also have an index for the outer table, finding the subset of rows that we want to join is even faster; but this is optional. If the number of rows to join from the outer table is small, but not that small, the Optimiser may have a hard time deciding whether to use a Nested Loop or Hash Join, so having up to date statistics is important. Merge Join: TL;DR: good even if you have lots of rows to join, but requires an index on the outer table to be efficient. Both tables are first sorted by the joining column (¿sorted in memory?), and then joined. The sorting allows to read the the inner table bit by bit : the RDMS reeds the 1st row from the sorted outer table, and then looks in the inner table for matching rows. When the next row no longer matches, it means that there won’t be any other matching row, as the table is sorted. This allows to only have to read each table once. Merge joins are thus efficient even if many rows have to be joined. But the sorting operation is expensive… Sorting both tables means a complexity roughly of o(#Table1 * Log #Table1 + #Table2 * Log #Table2) However, if you have an index for the outer table (on the joining column), then only the inner table will have to be sorted. Indeed, indexes are sorted, so the RDBMS will use the outer table index to read it in a sorted manner. Sadly, even if you have an index for the inner table, Oracle DB will still have to sort it [10:28]. Hash Join: TL;DR: works only for equality joins (e.g. not for “joincolumTable1 > joincolumnTable2”). Good for large amount of rows to join. It is the most efficient most of the time. Complexity is o(number of rows in Table1 + number of rows in Table2) A hash table of table1 is computed in memory, and then the value of each of table2 is hashed, and an equality of hash value is looked for in the hash table.

[@chungusmcjaxx5799]

Thanks. Explained visually nested loops in 90 seconds. Appreciate your method with cards. Thank you.

[@TheMagicofSQL]

You're welcome; glad you appreciate it!

[@AndrewCoffman-jq9ld]

Been doing MSSQL for 20 years and this was by far the best explanation of these joins. Thanks!

[@TheMagicofSQL]

You're welcome!

[@jessiebessie3616]

Woah this is good stuff. Easily one of the best explaination I seen in a long time. Clear and Concise. The anim also nicely done. Totally can visualise and relate in split seconds. Kudos!

[@TheMagicofSQL]

You're welcome, glad you found this useful :)

[@bobfar]

I am incredibly grateful to Chris for their invaluable tutorials on hash join, merge join, and nested loop. These concepts always seemed daunting to me when I encountered them in execution plans, but their clear and concise explanations have helped me gain a much deeper understanding. Their expertise and dedication to educating others are truly commendable. A big thank you for demystifying these complex topics and making them accessible to all!

[@TheMagicofSQL]

You're welcome

[@guptaashok121]

this is super good. just for my understanding, you said when we just have few card (5) from outer deck to match nested loop becomes faster as it can start matching from the first card however, hash join still needs to create hash table for all 52 cards from outer table. why should it create hash table for 52 instead of 5, assuming the filter is already applied and it knows those 5 rows already.

[@TheMagicofSQL]

Thanks Perhaps I wasn't clear on this - I was thinking of a Top-N, "get the first 5 rows then stop" query. Instead of a where clause that only matches 5 rows, the result set (could) be bigger. But we'll stop as soon as we've returned 5 rows. Because nested loops join rows immediately, it can stop as soon as it reads 5 rows from the outer table (assuming they all join to a row in the inner table). A hash join always builds the hash table on the whole outer table first. If the where clause identifies 5 rows from the outer table, you're right the hash join will only build on these 5 rows.

[@iamcerba]

It was unclear for me too, thanks for the explanation.

[@mohaktrivedi9591]

Thank you very much! This is hands-down the easiest and most concise explanations I've seen!

[@gururaoprabhakar5677]

Excellent Explanation. I don't think even those who wrote the optimizer's algorithms could have explained it this lucid and simple.

[@TheMagicofSQL]

Thanks! Glad you found this informative :)

[@grzekozak]

How you can explain complex matter and in the entertaining way ?! Congratulations ! You nailed it !

[@TheMagicofSQL]

You're welcome, glad you found these useful and enjoyable :)

[@abruenin236]

Probalby the best explanation on this I have ever seen

[@mukeshbashal5865]

i always confused in this, But after your card explaction i understand way u explained. Subscribed

[@TheMagicofSQL]

You're welcome!

[@snaidu70]

This is an amazing level of detail. I'm so glad I found your channel. It is priceless. Thank you so much for sharing all this knowledge with us.

[@TheMagicofSQL]

You're welcome, great to hear you find this useful :)

[@BettForester]

Thank you for a clear explanation! But I am still wondering why the merge sort needs to start on the previous value if we know it was already joined with the last value from the outer deck. Is there something I have missed? Thanks :)

[@TheMagicofSQL]

There could be many rows with the same value. Going back to the previous value is simple way to ensure you always capture them all. It also works for greater/less than comparisons - if the join is C1 > C2 then you'll be on the last row/value in the inner deck after the first row from the outer. But likely need to revisit most of the rows in the inner deck when you go to the second in the outer.

[@bodoziada6242]

that was super easy to understand such complicated concepts and with good English for all people. Thank you

[@TheMagicofSQL]

You're welcome, thanks!

[@Somosphere]

First time came across such good explanation of these 3 kind of joins. Kudos !!!

[@zelalembayissa4181]

This is amazing how you explain and make it simple. Thank you very much!

[@vijayhul]

This video deserves more views

[@TheMagicofSQL]

Thanks, glad you found it useful :)

[@88spaces]

Excellent explanation of join strategies. And your use of decks of cards helps to visualize each strategy in your head for a better understanding of each one. Thank you.

[@TheMagicofSQL]

You're welcome, glad you found this useful :)

[@oah8465]

This channel is pure gold. Thx a ton.

[@TheMagicofSQL]

Thanks :)

[@BharathMarrivada]

The best explanation that I have ever seen.

[@GSCVirus]

Great explanation and the example with the deck of cards is brilliant

[@robertczaja176]

Perfect video! You can explain things that everyone can understand it!

[@jamsher731]

Excellent explaination Chris Saxon

[@amanpandey4907]

Finest video on this topic. How is it possible that it was hidden for so long? Just a suggestion, I think you should include NESTED Loops, Merge join, Hash join in the description so more people searching on youtube will come across this. include more hashtags and all. Thanks for the video, if possible please explain REGEX.

[@TheMagicofSQL]

Thanks - these terms are already in the description though; what exactly are you suggesting I do differently? REGEX is a big topic! Maybe I'll cover it one day ;)

[@amanpandey4907]

@@TheMagicofSQL sry I wrote description, I meant video title.

[@strokeracer]

Such a good video!! Finally making Joins make sense for me!

[@TheMagicofSQL]

Glad this helped :)

[@payalbhatia5244]

This is really magic , I could understand all of it with an engaging interest. Thanks for sharing.

[@TheMagicofSQL]

Great to hear :)

[@arunsundar3739]

this video helps me to understand the joining strategies as well helps to make sense & reason out the usage of some of the complex concepts used in Apache Spark, fortunate to find this video, fantastic explanation, very easy to understand the concepts that felt really confusing, thank you very much :)

[@TheMagicofSQL]

You're welcome! Glad you found this useful

[@kavindashehanrajapakshe2890]

It's truely magical. Thank you

[@AndrewCoding]

Thank you so much. You make my day. Best explanation ever.

[@nikhilv199138]

very informative, easily and affectively explains the three join strategies

[@TheMagicofSQL]

Great, glad you found this informative :)

[@israaezzat2353]

your explanation is beyond amazing bravo

[@TheMagicofSQL]

Thanks; glad you found this useful!

[@Speak12truth]

I don't think anyone can explain better than this.

[@ArthurMontgomery-d2p]

Great video. Thanks. But I didn't understand the reason why the example of joining only five cards in the outer deck shows the disadvantage of hash join (from 8:17 to 8:25). I mean, why can't we simply apply the hash function to those five values? That only costs five operations to construct the hash memory structure?

[@TheMagicofSQL]

How do you know what the top five values are before doing the join? If you want to (inner) join t1 to t2 and get the first five joined rows there's no general way to pick five from t1 that will join to t2. The values you pick from t1 may have no match in t2. Using a hash, you have to read all the rows from t1. Then join to t2, stopping as soon as you have five rows in the results.

[@ArthurMontgomery-d2p]

@@TheMagicofSQL Thanks for the clarification! I took the five cards as already known.😅

[@SilasSWF]

This is one the bests videos about this subject. Thank you a lot!!!

[@TheMagicofSQL]

Thanks; glad it helped

[@vialvial1246]

Best explanation! Thank you sir!

[@FrankyWeber]

Very nice Chris. I’ve watched some other video in the past from Connor, but this one is much more elaborated. I was wondering if you could explain in one video of this series about bloom filters. Thank you again for doing this. Excellent work.

[@TheMagicofSQL]

Thanks Franky! I've added bloom filters to my list of things to create videos about ;)

[@nguyennguyenkhoi7111]

Thanks bro, It's really nice and clear explaination. However, I still have 1 question. Why do we need to run "Hash Function" in the whole table? Why don't we hash 5 cards and do the JOIN?

[@TheMagicofSQL]

See my response to ArthurMontgomery-d2p - how do you know which 5 rows to pick from the first table?

[@TheAbbassss]

very simple explaining..good work bro

[@abdullahyahya2471]

Awesome Explanation. I said "Yaaar Kamaaal" when watching this video. Which mean " Dudeee, Terriffic"

[@TheMagicofSQL]

:) Great to hear!

[@Noone-bb5qh]

Thank you so much for this video. It really helped me for my interview.

[@TheMagicofSQL]

You're welcome; glad this helped you

[@josedial1]

Excellent explanation for something that took me years to learn and I even bought some books to try to understand this concepts. Still have questions in my mind like - what is considered a big table ( how many rows - at the end normally the answer is -> It depends). Also how many rows are a few rows to return base on the total numbers of rows in a table. Also explain this with two tables is kind of Ok, now in real life RL when there are 3,4, ... tables - I know at the end is always join two tables but when the query is complex is very tiring to try to figure out what is the best join between table t1 and t3 and now t1 and t4 etc.. - Any way a Big thank you again.

[@TheMagicofSQL]

Yeah "big table" doesn't really have a fixed definition! Remember that when deciding join order & method it's not the total number of rows in the table that matters. It's how many rows the optimizer expects to fetch from that table. In general the optimizer tries to start by joining the two tables that you fetch the fewest rows from. Then adding the tables with more rows & finish by joining the table that returns the most. This is because starting with the smallest data set and adding to it is more efficient than starting with the biggest data set. When the optimizer gets the "wrong" join order & method, it's often because the number of rows it estimates is significantly different (an order of magnitude or more) to the actual number it processes.

[@josedial1]

@@TheMagicofSQL Wow Chris. I really appreciate that you had take the time to answer my comment. Thanks again.

[@TheMagicofSQL]

You're welcome!

[@mustafakalayciDBA]

This might be the first comment that I have ever make in youtube :) I am already familiar to join types but watching this was quite nice. you explained it so simple and I love it Chris 👍👍👍. I would like to add this video to my blog also if it is okay for you. Nice moustache by the way :)

[@TheMagicofSQL]

Cool, glad you enjoyed this

[@bisratgezahgne914]

Wow, Easy to understand with a detailed explanation. Thank you

[@TheMagicofSQL]

You're welcome, thanks!

[@AlexYeusiukou]

Thank you for this clear explanation! Subscribed! One question: if both tables are indexed and we use a Merge Join, you said that it would still sort the second table. Could you elaborate on the reason behind it? Excellent content!

[@TheMagicofSQL]

"It just does!" Sorry, I don't know the exact reason why Oracle Database always sorts the second table. It may be that this changes in the future.

[@veeramani8478]

this is freaking awesome i learnt something in depth one of the best channel i have seen so far i think using hints we can command the optimizer we want this join type while doing certain query using pinned explain plan

[@TheMagicofSQL]

Thanks! You have to be careful using hints. To ensure the optimizer picks a particular plan, typically you need LOTS of hints to ensure it does this. When using Oracle Database we recommend you use SQL Plan Management (SQL profiles & baselines) to manage plans instead.

[@sdef719]

Absolutely Loved the way you explained. More power to you. Subscribed. :)

[@dbajubin]

Fantastic explanation. You made it so easy to understand . Thanks

[@mohan1958]

Thanks very much for the detailed study.

[@datawitharyan]

Thanks bro , Excellent Explanation

[@RamaRavuri]

nice explanation, but stuck with a doubt, so if we have indexes, won't it be 52 iterations to find match in other table which is lesser than compared to hash join having 104 hits, plz explain.

[@TheMagicofSQL]

You mean for nested loops? To get all the rows in both tables? Both nested loops and hash joins will read all the data in both tables. A (covering) index means the nested loop only needs to read the joined row on the inner table. But it also needs to read the index too! A hash join full scans both tables, reading each row once. The nested loop could full scan the outer table and use an index to lookup in the inner table. So while it reads the same number of rows, it's still more work. This is because it has to read the index too. It's also faster to full scan all the rows rather than read them one-by-one via an index.

[@ajay-jm1ni]

I got addicted to see deeper and deeper in SQL now Thanks for the such nice explanation

[@pranavsharma7479]

this video shd be viral among all advance database students

[@TheMagicofSQL]

Then please share it :)

[@MuhammadUmar-dx7wh]

Very Good explanation

[@sandhu01]

Very well put 👏 @6:20 - 52*log52 equals 296.4 because the base is 2 not 10.

[@TheMagicofSQL]

52 * ln 52 ~ 205 isn't it? :) In any case, the exact numbers don't matter too much here - they will vary depending on the algorithm used. It's more the relative size of operations needed for each join type.

[@sandhu01]

@@TheMagicofSQL thanks for the reply! Understood! That's why appreciation was in first place :)

[@baouevangelia5140]

I have a question, when you say to sort the values of the two decks, you mean by creating a clustered index on each bucket or am i missing something?

[@TheMagicofSQL]

No - just that the database has to order all the rows from each table on the join columns. Like when you add an ORDER BY to a SELECT statement.

[@JKhalaf]

This was really good, thank you.

[@junaidahmed3981]

such clear explanation, thank you sir

[@indudwivedi4068]

Nicely explained!

[@Maen963]

awesome, great and terrefic explanation. so underrated !

[@TheMagicofSQL]

Thanks!

[@PradeepKanaparthy]

HI please one more favour. need post on tuning complex query having multiple joins with example to see changes before tuning and after tuning. some of the tuning changes or tips and how they affect the query and make quick access or give response

[@TheMagicofSQL]

I recently ran an Ask TOM Office Hours session going through some real world tuning examples - see the recording at kzbin.info/www/bejne/m6itkqCEipt8j5I

[@PradeepKanaparthy]

@@TheMagicofSQL Thank you 😊

[@hy1011k]

Perfect explanation👏🏻

[@TheMagicofSQL]

Thanks!

[@todorowael]

Great video, thank you.

[@kumarparimi676]

Best explanation for joins ever 👍👍

[@rahil8304]

Absolutely incredible Big Cheer to this guy

[@guptaashok121]

when we compare the complexity we only compared based on the number of matches however the hash join also needs to build hash table, how can we estimate the cost of that to compare with other two.

[@satyendrakumar6667]

Awesome way of explaining the joins 👍.

[@TheMagicofSQL]

You're welcome, thanks!

[@fabkury]

Your content (and presentation) is excellent, thank you for your work!

[@TheMagicofSQL]

You're welcome; glad you found this useful

[@sidnayak4395]

Thanks to yt and you for recommending and giving a beautiful explanation on this topic ❤️ choosing cards was best....was unable to understand from theory session from various sources

[@TheMagicofSQL]

Thanks and welcome

[@chilukanand]

wow this is awesome! Thanks for sharing this video.

[@RomanPeralta]

Great explanation! Thank you so much for sharing your knowledge.

[@likethebeer]

This was a very good explanation. Thanks!

[@Ytakies]

This is GOLD ❤❤❤❤

[@YusanTRusli]

Great explanation! Thank You

[@chandnigupta4933]

Really wonderful! Very well explained ! You have got less likes .. your video deserves lot of likes and appreciation as Its really outstanding👌🏼

[@TheMagicofSQL]

You're welcome, glad you enjoyed this - please share it with anyone you think would benefit!

[@aliadel1723]

I'm lucky to found this treasure.

[@govindpalariya3173]

This is really outstanding session.

[@mayfly0]

such a pleasure to watch, thanks for the video

[@TheMagicofSQL]

You're welcome, glad you enjoyed it!

[@tasty4644]

excellent video

[@ludwigettner3777]

Saved my a**. Really nice video.

[@ChoothamT]

Thank you for well explained. Excellent!

[@TheMagicofSQL]

You're welcome, thanks!

[@aniketurankar9568]

Great learning, Thank you.

[@TheMagicofSQL]

You're welcome!

[@lawalrasheed_]

Absolute gem!!

[@fandeslyc]

Hello At 8:15, you said that even if we only wanted the join for 5 cards in the first deck, we still had to go through the entire deck I don't really understand why Can't we just put the 5 cards in the hash table ?

[@TheMagicofSQL]

The problem is you don't know *which* five cards from the first deck will join to cards in the second deck - if any! In general there is NOT a one-to-one match like with decks of cards. Say the second deck only has aces in; there's no guarantee any of the first five cards you draw from the first deck will be aces. So you have to build the hash table from all the cards in the first deck before you start reading the second deck. Does this help?

[@fandeslyc]

@@TheMagicofSQL Yes thanks I was thinking of a left join, but in that case, the 5 card set would be on the probing side i guess

[@TheMagicofSQL]

It doesn't really matter whether it's an inner or outer; a hash join processes all the rows in the first table before accessing the second

[@LuisMi-vm7ri]

Around 6min for complexity, do you use binary or decimal log? I believe it should be binary but one of your calculations (the one for Merge Join = 180) leads me to think you used decimal. Was this a mistake when using your calculator, or is it really meant to be decimal log?

[@LuisMi-vm7ri]

Very nice video btw! Really enjoy the visual and step by step explanations

[@TheMagicofSQL]

It was a decimal log. The numbers are just approximations to give some idea of complexity. Particularly for sorting the precise figures depend on the sort algorithm and order of the input data set. Whether you use LN or LOG, merge join comes out between nested loops and hash join in the example.

[@LuisMi-vm7ri]

Makes sense! Thank you

[@FiruzzZ-777]

flawless explanations!, you deserve your hair back

[@TheMagicofSQL]

😂 thanks!

[@looneytr]

Great video

[@rawgyanlearnandinnovate9781]

Very nice video 👍

[@anuragdhyani1712]

Great vid 😀

[@mehdibouchene3208]

this was magic thank you

[@mauricioroldanramirez821]

Excellent, Thank you :)

[@skhochay]

you are very good teacher !!!!!

[@TheMagicofSQL]

Thanks :)

[@IgorSadovskii]

usefull explanation ! thanks a lot!

[@taniyasaini6830]

Awesome video! Love it!!!