Learn MapReduce with Playing Cards

[@kart00nher0]

This is by far the best explanation of the MapReduce technique that I have come across. I especially like how the technique was explained with the least amount of technical jargon. This is truly an ELI5 definition for MapReduce. Good work!

[@jessetanderson]

+Subramanian Iyer Thanks!

[@smushti]

An innovative idea to use a pack of cards to explain the concept. Getting fundamentals right with an example is great ! Thank you

[@ekdumdesi]

Great explanation !! You Mapped the Complexity and Reduced it to Simplicity = MapReduce :)

[@djyotta]

Very well done - not too slow, yet very clear and well structured.

[@Useruytrw]

Jesse may you get all SUCCESS and BLESSINGS

[@rodrigofuentealbafuentes695]

Really good illustration.... really easy to understand for people as me that we are not computer experts.. thanks

[@bit.blogger]

6:16 got a question! Would you please elaborate more on those moving data? Since there is two separate reduce task on those two nodes how does two different reduce tasks combine together? How do we choose which cards move to which node?

[@jessetanderson]

That is called the shuffle sort. See more about that here www.inkling.com/read/hadoop-definitive-guide-tom-white-3rd/chapter-6/shuffle-and-sort.

[@chandrakanthpadi]

Does the actual data in the node moves or copies of the data is moved?

[@mmuuuuhh]

To wrap this up: Map = Split data Reduce = Perform calculations on small chunks of data in parallel Then combine the subresults from each reduced-chunk. Is that correct?

[@jessetanderson]

+mmuuuuhh Somewhat correct. I'd suggest buying the screencast to learn more about the code and how it works.

[@alphacat03]

+mmuuuuhh merge-sort maybe?

[@ienjoysandwiches]

divide and conquer

[@BULLSHXTYT]

Map transforms data too

[@dennycrane2938]

No no... Map = Reduce the Data, Reduce = Map the Data . .... ....

[@sukanyaswamy]

Great presentation. The visualization makes it so much easier to understand.

[@kabirkanha]

Never trust a man whose deck of playing cards has two 7s of Diamonds.

[@vamsikrishnachiguluri8510]

what a great effort, i am astonished by your teaching skills.we need teachers like you.Thanks for your best explanation .

[@LetsBeHuman]

4:51 - - i'm kind of lost. so you said two papers as two sets of nodes. left is node1 and right is node2. then you said, "I have two nodes, where each node has 4 stacks of cards". I also understood that you are merging two varieties of cards in node1 and another two varieties of cards in node2. " a cluster is made of tens, hundreds or even thousands of nodes all connected by a network". so in this example, let's say two papers(nodes) are one cluster. the part I get confused is , when you say " the mapper on a node operates on that smaller part. the magic takes the mapper data from every node and brings it together on nodes all around the cluster. the reducer runs a node and knows it has access to everything with same key ". So if there are two nodes A and B that has mapper data, then the reducer part will happen on two other nodes C and D. I'm confused when you say "on nodes all around the cluster".

[@scottzeta3067]

The only one I watched which can clearly introduce mapreduce to newbie

[@amitprakashpandeysonu]

loved the idea. Now I understood how map reduce works. Thank you.

[@furkanyigitozgoren3847]

It was very nice. But I could not find the video that you showed the shuffling "magic part"

[@menderbrains]

Great explanation! This is how a tutor should simplify the understanding! Thanks

[@doud12011990]

really cool one. It is always nice to come back to the basics. Thanks for that one

[@vscid]

and that's how you explain any technical concept. simple is beautiful!

[@victorburnett6329]

If I understand correctly, the mapper divvies up the data among nodes of the cluster and subsequently organizes the data on each node into key-value pairs, and the reducer collates the key-value pairs and distributes the pairs among the nodes.

[@jessetanderson]

Almost. Hadoop divvies up the data, the mapper creates key value pairs, and the reducer processes the collated pairs.

[@vivek3350]

Really liked your way of presentation....."Simple" and "Informative". Thanks for sharing!!

[@rahulx411]

an ounce of example is better than a ton of precept! --Thanks, this was great!

[@ahmedatallahatallahabobakr8712]

Your explanation is majic! Well done

[@rohitgupta025]

Just wow...very nicely explained

[@davidy2535]

amazing explanation! I love it. Huge Thanks!

[@mgpradeepa554]

The explanation is wonderful.. You made me understand things easily.

[@nkoradia]

Brilliant approach to teach the concept

[@prasann26]

Wow.. You have made this look so simple and easy... Thanks a ton !!!

[@abhishekgowlikar]

Nice video explaining the Map Reduce Practically.

[@hazelmiranda8587]

Good to understand for a layman! So its quite crucial to identify the basis of the grouping i.e. the parameters based on which the data should be stored in each node. Is it possible to revisit that at a later stage?

[@thezimfonia]

That was very helpful Jesse. Thank you for sharing this!!

[@asin0136-y6g]

Wonderful explanation ! Made it very simple to understand! Thanks a ton!

[@user-or7ji5hv8y]

best explanation of mapReduce. Thanks!

[@mahari999]

Superb. Thank you Jesse Anderson

[@arnavanuj]

Good illustration. 😃

[@sarthakmane2977]

dude, whats the name of that magic??

[@tkousek1]

Great explanation!! worth a bookmark. Thank you sir!

[@TheDeals2buy]

Good illustration using a practical example...

[@gboyex]

Great video with good explanation technique.

[@anandsib]

Good Explanation with simple example

[@urvisharma7243]

What if the node with clubs and hearts breaks down during the reduce operation? Will data be lost? Or will the complete Map Reduce job be repeated using the replicated data?

[@jessetanderson]

The data is replicated and the reduce would be re-run on a different node.

[@amirkazemi2517]

greta video. why is there performance issues with hadoop however?

[@jessetanderson]

I'm not sure what you mean by performance issues.

[@hexenkingTV]

So it follows mainly the principle of divide and conquer?

[@jessetanderson]

Following that analogy, it would be divide, reassemble, and conquer.

[@AnirudhJas]

Thanks Jesse! This is a wonderful video! I have 2 doubts. 1. Instead of sum, if it is a sort function, how will splitting it into nodes work? Because then every data point should be treated in one go. 2. The last part on scaling, how will different nodes working on a file and then combining based on key, be more efficient than one node working on one file? I am new to this and would appreciate some guidance and help on the same.

[@jessetanderson]

1. This example goes more into sorting github.com/eljefe6a/CardSecondarySort 2. It isn't more efficient, but more scalable.

[@AnirudhJas]

@@jessetanderson Thank you!

[@rogerzhao1158]

Nice tutorial! Easy to understand

[@piyushmajgawali1611]

I actually did this with cards.Thanks

[@sebon11]

Great explanation!

[@arindamdalal3988]

really nice video and explain the terms in a simple way...

[@patrickamato8839]

Great summary - thanks!

[@tichaonamiti4616]

Thats wonderful ..... you are a gret teacher

[@LetsBeHuman]

When you say nodes and clusters, does an input file of 1TB should definitely be run in more than one computer or we can install hadoop in a single laptop and virtually create nodes and clusters ?

[@abdulrahmankerim2377]

Very useful explanation.

[@trancenut81]

Excellent explanation!

[@grahul007]

Excellent video explanation

[@Dave-lc3cd]

Thanks for the great video!

[@rodrigoborjas7727]

Thank u very much for the explanation.

[@gypsyry]

Best explanation. Thanks a lot

[@moofymoo]

huge 1Tb file.. anyone watching this in 2065?

[@NuEnque]

February 2019 (Go RAMS)

[@anmjubaer]

@@NuEnque July 21 2019

[@devanshsrivastava4265]

feb 2020

[@jonathannimmo9293]

more like 2025

[@omrajpurkar]

August 11, 2020!!

[@MrSpun1090]

Thanks this really helped me for my exam !!

[@MuhammadFarhan-ny7tj]

Which music is this in start of this video

[@jessetanderson]

I'm not sure where they got it from.

[@IvanRubinson]

Well, that explains the interview question: How would you sort a ridiculously large amount of data?

[@sarthakmane2977]

great video by the way!!

[@amandeepak8640]

Thank You sir for such a wonderful explanation. :-)

[@vincentvimard9019]

just great explanation !

[@vigneshrachha8362]

Superb video....thanks a lot sir

[@ZethWeissman]

Might be a bit clearer to understand the advantage of this if instead of having the same person run the cards on each node sequentially and have two people do it at the same time. Or go further and have four people show it. Then each person can grab all the cards of the suit from each node and can sum their values up, again, at the same time. Show a timer showing how long it took for the one person to do everything on one node and the time of having all four running at the same time.

[@rrckguy]

Great lesson. Thanks..

[@MincongHuang]

Great video, thanks for sharing !

[@alextz4307]

Very nice, thanks a lot.

[@irishakazakyavichyus]

thanks! that is an easy explanation!

[@ajuhaseeb]

Aiwa. Simply explained.

[@Luismunoz-jf2zv]

Now I get it, thanks!

[@SamHopperton]

Brilliant - thanks!

[@wetterauerbub]

Hi Jesse, can I use map reduce only on document-oriented DBs, or also e.g. on Graph databases?

[@jessetanderson]

Hessebub you can use it for both, but the processing Algorithms are very different between them.

[@wetterauerbub]

Alright, thanks very much for answering & doing the video in the first place!

[@周大鹏-o1j]

Great video

[@abdellahi.heiballa]

my friend: i wish i had ur calm we having an exam tomorrow you watching how playing cards....

[@devalpatel7243]

Hat's of man. very well understood

[@hemanthpeddi4129]

awesome explanation super

[@bijunair3807]

Good explanation

[@guessmedude9636]

i like this technique nice keep it up

[@logiprabakar]

Wonderful, you have used the right tool(cards) and made it simpler. Thank you. Am i correct in saying, in this manual shuffle and sort, the block size is 52 cards where as in a node it would be 128.

[@__-to3hq]

wow this was great

[@thiery572]

Interesting. Now I want to request a bunny comes out from a hat.

[@iperezgenius]

Brilliant!

[@yash6680]

awesome

[@RawwestHide]

thanks

[@pamgg1663]

excellent!!!

[@ZFlyingVLover]

The 'scalability' of hadoop has to do with the fact that the data being processed CAN be broken up and processed in parallel in chunks and then the results can be tallied by key. It's not an inherent ability of the tech other than HDFS itself. Like most technology or jobs for that matter the actual 'process' is simple it's wading through the industry specific terminology that has makes it unnecessarily complicated. Hell you can make boiling an egg or making toast complicated too if that's your intent.

[@jessetanderson]

Sorry, you misunderstood.

[@ZFlyingVLover]

@@jessetanderson I didn't misunderstand you. Your explanation was great.

[@mudassarm30]

spade clubs ... I think you used the wrong suite names for them :)

[@niamatullahbakhshi9371]

so nice

[@lerneninverschiedenenforme7513]

little bit long explanation. could be done faster (e.g. card-sorting). But after watching, you know what's happening. So all thumbs up!

[@covelus]

awesome

[@sumantabanerjee9728]

Easiest explanation.

[@Nyocurio]

Why did they come up with such a terribly unintuitive name as "MapReduce" ??? It's basically just "bin by attribute, then process each bin in parallel". BinProcess.

[@jessetanderson]

It's a well-known functional programming paradigm.

[@varshamehra8164]

Cool

[@haroonrasheed9739]

Great

[@kart00nher0]

IMO the key takeaway from the video is that MR only works when: a. There is one really large data set (e.g. a giant stack of playing cards) b. Each row in the data set can be processed independently. (e.g. sorting or counting playing cards does not require knowing the sequence of cards in the deck - each card is processed based on information on the face of card) To process real-world problems using MR, the data sets will need to be massaged and joined to satisfy the criteria listed above. This is where all the challenges lie. MR itself is the easy part.

[@jessetanderson]

+Subramanian Iyer agreed MR is difficult, but the understanding of how to use and manipulate the data is far more complex. This is why I think data engineering should be a specific discipline and job title. www.jesse-anderson.com/big-data-engineering/

[@glennt1962]

This is a great example video without the accent to deal with.

[@gregrell2441]

This is just a sales pitch

[@jessetanderson]

I think the description is pretty clear that it's an extended preview of the screencast.

[@joseblazquez8417]

like si vienes por riwb

[@hank-l6s]

keep kinging