Learn MapReduce with Playing Cards

The special extended preview of my new MapReduce screencast available for purchase at pragprog.com/screencasts/v-jam....
To get access to my updated and in-depth course, go to my site at www.jesse-anderson.com/big-dat... and sign up. You'll get a free mini-course and then have the option to purchase the full 8 week course.

[@kart00nher0]

This is by far the best explanation of the MapReduce technique that I have come across. I especially like how the technique was explained with the least amount of technical jargon. This is truly an ELI5 definition for MapReduce. Good work!

[@jessetanderson]

+Subramanian Iyer Thanks!

[@smushti]

An innovative idea to use a pack of cards to explain the concept. Getting fundamentals right with an example is great ! Thank you

[@djyotta]

Very well done - not too slow, yet very clear and well structured.

[@Useruytrw]

Jesse may you get all SUCCESS and BLESSINGS

[@doud12011990]

really cool one. It is always nice to come back to the basics. Thanks for that one

[@sukanyaswamy]

Great presentation. The visualization makes it so much easier to understand.

[@vivek3350]

Really liked your way of presentation....."Simple" and "Informative". Thanks for sharing!!

[@ekdumdesi]

Great explanation !! You Mapped the Complexity and Reduced it to Simplicity = MapReduce :)

[@vamsikrishnachiguluri8510]

what a great effort, i am astonished by your teaching skills.we need teachers like you.Thanks for your best explanation .

[@thezimfonia]

That was very helpful Jesse. Thank you for sharing this!!

[@aravindsingirikonda1569]

Wonderful explanation ! Made it very simple to understand! Thanks a ton!

[@prasann26]

Wow.. You have made this look so simple and easy... Thanks a ton !!!

[@menderbrains]

Great explanation! This is how a tutor should simplify the understanding! Thanks

[@ahmedatallahatallahabobakr8712]

Your explanation is majic! Well done

[@vscid]

and that's how you explain any technical concept. simple is beautiful!

[@mgpradeepa554]

The explanation is wonderful.. You made me understand things easily.

[@rodrigofuentealbafuentes695]

Really good illustration.... really easy to understand for people as me that we are not computer experts.. thanks

[@abhishekgowlikar]

Nice video explaining the Map Reduce Practically.

[@tkousek1]

Great explanation!! worth a bookmark. Thank you sir!

[@rohitgupta025]

Just wow...very nicely explained

[@amitprakashpandeysonu]

loved the idea. Now I understood how map reduce works. Thank you.

[@rahulx411]

an ounce of example is better than a ton of precept! --Thanks, this was great!

[@davidy2535]

amazing explanation! I love it. Huge Thanks!

[@user-or7ji5hv8y]

best explanation of mapReduce. Thanks!

[@gboyex]

Great video with good explanation technique.

[@nkoradia]

Brilliant approach to teach the concept

[@arindamdalal3988]

really nice video and explain the terms in a simple way...

[@krupakapadia2498]

Great Explanation! Thanks!

[@rogerzhao1158]

Nice tutorial! Easy to understand

[@mahari999]

Superb. Thank you Jesse Anderson

[@AlexChetcuti]

Thanks this really helped me for my exam !!

[@anandsib]

Good Explanation with simple example

[@abdulrahmankerim2377]

Very useful explanation.

[@amandeepak8640]

Thank You sir for such a wonderful explanation. :-)

[@hazelmiranda8587]

Good to understand for a layman! So its quite crucial to identify the basis of the grouping i.e. the parameters based on which the data should be stored in each node. Is it possible to revisit that at a later stage?

[@scottzeta3067]

The only one I watched which can clearly introduce mapreduce to newbie

[@go_better]

Thanks! Great explanation

[@patrickamato8839]

Great summary - thanks!

[@tichaonamiti4616]

Thats wonderful ..... you are a gret teacher

[@TheDeals2buy]

Good illustration using a practical example...

[@anmjubaer]

Great explanation. Thanks.

[@trancenut81]

Excellent explanation!

[@vincentvimard9019]

just great explanation !

[@Dave-lc3cd]

Thanks for the great video!

[@rodrigoborjas7727]

Thank u very much for the explanation.

[@victorburnett6329]

If I understand correctly, the mapper divvies up the data among nodes of the cluster and subsequently organizes the data on each node into key-value pairs, and the reducer collates the key-value pairs and distributes the pairs among the nodes.

[@jessetanderson]

Almost. Hadoop divvies up the data, the mapper creates key value pairs, and the reducer processes the collated pairs.

[@sebon11]

Great explanation!

[@gypsyry]

Best explanation. Thanks a lot

[@grahul007]

Excellent video explanation

[@arnavanuj]

Good illustration. 😃

[@mmuuuuhh]

To wrap this up: Map = Split data Reduce = Perform calculations on small chunks of data in parallel Then combine the subresults from each reduced-chunk. Is that correct?

[@jessetanderson]

+mmuuuuhh Somewhat correct. I'd suggest buying the screencast to learn more about the code and how it works.

[@alphacat03]

+mmuuuuhh merge-sort maybe?

[@kemchobhenchod]

divide and conquer

[@BULLSHXTYT]

Map transforms data too

[@dennycrane2938]

No no... Map = Reduce the Data, Reduce = Map the Data . .... ....

[@vigneshrachha8362]

Superb video....thanks a lot sir

[@irishakazakyavichyus]

thanks! that is an easy explanation!

[@rrckguy]

Great lesson. Thanks..

[@piyushmajgawali1611]

I actually did this with cards.Thanks

[@Luismunoz-jf2zv]

Now I get it, thanks!

[@sarthakmane2977]

great video by the way!!

[@alextz4307]

Very nice, thanks a lot.

[@LetsBeHuman]

4:51 - - i'm kind of lost. so you said two papers as two sets of nodes. left is node1 and right is node2. then you said, "I have two nodes, where each node has 4 stacks of cards". I also understood that you are merging two varieties of cards in node1 and another two varieties of cards in node2. " a cluster is made of tens, hundreds or even thousands of nodes all connected by a network". so in this example, let's say two papers(nodes) are one cluster. the part I get confused is , when you say " the mapper on a node operates on that smaller part. the magic takes the mapper data from every node and brings it together on nodes all around the cluster. the reducer runs a node and knows it has access to everything with same key ". So if there are two nodes A and B that has mapper data, then the reducer part will happen on two other nodes C and D. I'm confused when you say "on nodes all around the cluster".

[@LetsBeHuman]

When you say nodes and clusters, does an input file of 1TB should definitely be run in more than one computer or we can install hadoop in a single laptop and virtually create nodes and clusters ?

[@logiprabakar]

Wonderful, you have used the right tool(cards) and made it simpler. Thank you. Am i correct in saying, in this manual shuffle and sort, the block size is 52 cards where as in a node it would be 128.

[@SamHopperton]

Brilliant - thanks!

[@devalpatel7243]

Hat's of man. very well understood

[@AnirudhJas]

Thanks Jesse! This is a wonderful video! I have 2 doubts. 1. Instead of sum, if it is a sort function, how will splitting it into nodes work? Because then every data point should be treated in one go. 2. The last part on scaling, how will different nodes working on a file and then combining based on key, be more efficient than one node working on one file? I am new to this and would appreciate some guidance and help on the same.

[@jessetanderson]

1. This example goes more into sorting github.com/eljefe6a/CardSecondarySort 2. It isn't more efficient, but more scalable.

[@AnirudhJas]

@@jessetanderson Thank you!

[@guessmedude9636]

i like this technique nice keep it up

[@ZethWeissman]

Might be a bit clearer to understand the advantage of this if instead of having the same person run the cards on each node sequentially and have two people do it at the same time. Or go further and have four people show it. Then each person can grab all the cards of the suit from each node and can sum their values up, again, at the same time. Show a timer showing how long it took for the one person to do everything on one node and the time of having all four running at the same time.

[@kabirkanha]

Never trust a man whose deck of playing cards has two 7s of Diamonds.

[@furkanyigitozgoren3847]

It was very nice. But I could not find the video that you showed the shuffling "magic part"

[@bijunair3807]

Good explanation

[@bit.blogger]

6:16 got a question! Would you please elaborate more on those moving data? Since there is two separate reduce task on those two nodes how does two different reduce tasks combine together? How do we choose which cards move to which node?

[@jessetanderson]

That is called the shuffle sort. See more about that here www.inkling.com/read/hadoop-definitive-guide-tom-white-3rd/chapter-6/shuffle-and-sort.

[@chandrakanthpadi]

Does the actual data in the node moves or copies of the data is moved?

[@IvanRubinson]

Well, that explains the interview question: How would you sort a ridiculously large amount of data?

[@user-ho2kf2xr7v]

Great video

[@hemanthpeddi4129]

awesome explanation super

[@hexenkingTV]

So it follows mainly the principle of divide and conquer?

[@jessetanderson]

Following that analogy, it would be divide, reassemble, and conquer.

[@MincongHuang]

Great video, thanks for sharing !

[@ajuhaseeb]

Aiwa. Simply explained.

[@__-to3hq]

wow this was great

[@paulfsch3108]

Hi Jesse, can I use map reduce only on document-oriented DBs, or also e.g. on Graph databases?

[@jessetanderson]

Hessebub you can use it for both, but the processing Algorithms are very different between them.

[@paulfsch3108]

Alright, thanks very much for answering & doing the video in the first place!

[@iperezgenius]

Brilliant!

[@yash6680]

awesome

[@amirkazemi2517]

greta video. why is there performance issues with hadoop however?

[@jessetanderson]

I'm not sure what you mean by performance issues.

[@ShoaibKhan-hy5nf]

The magic part u mentioned in the video resides in reducer or Map?

[@jessetanderson]

Shoaib Khan mostly in between those two phases

[@urvisharma7243]

What if the node with clubs and hearts breaks down during the reduce operation? Will data be lost? Or will the complete Map Reduce job be repeated using the replicated data?

[@jessetanderson]

The data is replicated and the reduce would be re-run on a different node.

[@pamgg1663]

excellent!!!

[@RawwestHide]

thanks

[@niamatullahbakhshi9371]

so nice

[@moofymoo]

huge 1Tb file.. anyone watching this in 2065?

[@NuEnque]

February 2019 (Go RAMS)

[@anmjubaer]

@@NuEnque July 21 2019

[@devanshsrivastava4265]

feb 2020

[@jonathannimmo9293]

more like 2025

[@omrajpurkar]

August 11, 2020!!

[@mudassarm30]

spade clubs ... I think you used the wrong suite names for them :)

[@covelus]

awesome

[@abdellahi.heiballa]

my friend: i wish i had ur calm we having an exam tomorrow you watching how playing cards....

[@sarthakmane2977]

dude, whats the name of that magic??

[@lerneninverschiedenenforme7513]

little bit long explanation. could be done faster (e.g. card-sorting). But after watching, you know what's happening. So all thumbs up!

[@ZFlyingVLover]

The 'scalability' of hadoop has to do with the fact that the data being processed CAN be broken up and processed in parallel in chunks and then the results can be tallied by key. It's not an inherent ability of the tech other than HDFS itself. Like most technology or jobs for that matter the actual 'process' is simple it's wading through the industry specific terminology that has makes it unnecessarily complicated. Hell you can make boiling an egg or making toast complicated too if that's your intent.

[@jessetanderson]

Sorry, you misunderstood.

[@ZFlyingVLover]

@@jessetanderson I didn't misunderstand you. Your explanation was great.

[@thiery572]

Interesting. Now I want to request a bunny comes out from a hat.

[@haroonrasheed9739]

Great

[@MuhammadFarhan-ny7tj]

Which music is this in start of this video

[@jessetanderson]

I'm not sure where they got it from.

[@sumantabanerjee9728]

Easiest explanation.

[@varshamehra8164]

Cool

[@kart00nher0]

IMO the key takeaway from the video is that MR only works when: a. There is one really large data set (e.g. a giant stack of playing cards) b. Each row in the data set can be processed independently. (e.g. sorting or counting playing cards does not require knowing the sequence of cards in the deck - each card is processed based on information on the face of card) To process real-world problems using MR, the data sets will need to be massaged and joined to satisfy the criteria listed above. This is where all the challenges lie. MR itself is the easy part.

[@jessetanderson]

+Subramanian Iyer agreed MR is difficult, but the understanding of how to use and manipulate the data is far more complex. This is why I think data engineering should be a specific discipline and job title. www.jesse-anderson.com/big-data-engineering/

[@glennt1962]

This is a great example video without the accent to deal with.