Understanding HDFS using Legos

You've been hearing about Hadoop and HDFS. How does it work?
In this video, we use an innovative method to show how HDFS works with Legos. Jesse Anderson shows how HDFS handles files and replicates the data, then covers the read and write paths for the data. Finally, he talks about how HDFS handles failure scenarios and the importance of data locality.
** Hadoop & Apache Spark training from NewCircle: newcircle.com/category/big-data

[@stackinglittlesats]

Nice explanation man, if I could, I would buy you an air conditioner. You deserve it.

[@SIRabhinav]

may the force be with you

[@miguelchris6374]

sorry to be offtopic but does anyone know a trick to log back into an Instagram account? I was stupid forgot my account password. I would love any tricks you can give me.

[@miguelchris6374]

@Billy Dariel it worked and I actually got access to my account again. Im so happy! Thank you so much you really help me out!

[@billydariel9140]

@Miguel Chris Happy to help :D

[@ZeTamboh]

@@billydariel9140 nice ad

[@davidvella7141]

This video was uploaded 5 years ago and it still one of the best explanations I've ever seen.

[@ThorstenStaerk]

the first explanation at all that could tell me what Hadoop has to do with MapReduce

[@taxatlanticinc6611]

This is a best explanation I have seen yet! It's a lot more engaging and informative than the traditional PowerPoint! Thank You!!

[@Coffingdw]

Nice job Jesse. Very informative and creative. Thank you. TeraTom

[@AdrianRodriguezWebDevelopment]

This video just made my day! Thank you New Circle Training! And thank you Doug Cutting for sharing this video on Twitter.

[@wisdomandpeace4897]

Excellent video. I actually understand Hadoop somewhat after watching this video.

[@TheMrsStinsfire]

11:06 R.I.P. Data Node 3

[@supermonkey965]

He was a good node, admired by his node friends.

[@ElCuchu]

I'm still crying, can't get over it, such a good node dude rippp

[@parthnagdev]

He is happy in the Node Heaven and is saving all the replicated data it ever wanted to save.

[@AhlamLamo]

Amazing explanation !! one of the best videos I've seen about HDFS

[@ravianantharamaiah7567]

Excellent teaching. Conceptually things are very clear now. Thank you.

[@gustavogbfBR]

Nice work. Really help me to understand how HDFS works.

[@amitprakashpandeysonu]

Really nice and innovative way to teach hdfs concept. loved it and understood it very clearly. Thank you.

[@klausdupont6335]

Incredible illustration! Would love to see more on this topic in this form :-)

[@maryoleary8660]

I love learning with legos, even watching it at 1.5x, I was able to follow along easily. Well Done.

[@MuzamilKhan-rl2sh]

Wow man, you explain it in a creative way.

[@Happymoon789]

Thanks for your efforts! Smart display!

[@prohouse6088]

very nice teaching methodology jesse, thanks for sharing

[@dingman081130]

gorgeous presentation, thanks

[@ThomasEhardt]

Great introduction to HDFS!

[@thandekilenzungu7240]

The explanation is so clear I understood everything

[@i_e_she]

This was great, thank you! Should have more views.

[@MSDlublin]

Very good work for begginers - THANKS A LOT!

[@joecordingley7071]

This was great, thanks.

[@abhijeet_r]

Very innovative presentation thanks a lot!

[@CarlosMercadoINIGTDY]

Excellent video, thanks!

[@olesyagorbacheva6991]

Thank you for such a good explanation!

[@thndesmondsaid]

Jesse! Great explanation as always.

[@markhellel3371]

Great Job Jesse! Nicely done! :-)

[@1234abcd2139]

nice illustrative way of teaching HDFS. Would have been wonderful if some more information was given about fallback mechanism of Name node or coordinator

[@nirupamaj6140]

very informative, thank you

[@kausaralam2605]

Great explanation!

[@yicai7]

Great explanation! Voted!

[@haydo8373]

Superb, I had it running at 1.5x and it was still easy to follow! Thanks :) Can you explain every CS concept with Lego? - that would be amazing

[@juiliussmith5702]

thanks dude lol

[@CB-fz3li]

Nice clear explanation

[@mrdhksan]

Excellent, thank you. A serious question: What would happen if two of the four nodes crash?

[@manojprabhakar5522]

Awesome, Thank you for the explanation, Could you please make videos of Spark with Yarn and how the communication is handled?

[@underlecht]

looks like we have a perfect explaination!

[@arisweedler4703]

Great explanation! I assume that another benefit of the HDFS is that reading large files will be quicker, because you would be able to effectively "BitTorrent" from your cluster. Does HDFS do that?

[@BabtaOfEinGedi]

Perfect! Thanks so much

[@jesusoliveros9950]

Amazing !!! great Job

[@rimchatti3807]

Nice job, it is helping getting familiar with HDFS. I'm new to Data Engineering and so on.. Could you please explain to me what is a cluster. Thanks;

[@Guitarman01]

Good Presentation, however I do have question. Since the file is split to other nodes, doesn't replication also take places so that if a node does go down, then you can retrieve. Node 3 went down, but I would have figured I could get it from another node. Does master save a copy of all files as well? I didn't see how that works on the video.

[@alaayari6391]

thanks for the explanation

[@bugs181]

I'm just now learning about the methods used in distributed file systems. I'm an application developer and it's a bit difficult to wrap my head around the lower level storage systems like HDFS. This video explained replication in an easy to understand way. Now only if I could have one other BIG question answered. What kind of file system would we use if we want an application to use a virtualized file system stored over many nodes? For example, we want each node to add additional storage capacity. To the application layer, this would look like a single big storage drive but to the lower level facilities this would use network coordination to serve the files to the application.

[@kishorpatil6260]

ĺm ĺ

[@buzz-uk]

Hi, While setting up pseudo or full cluster, do we need to format data node with HDFS file system or we only have to format namenode. I am asking this because, I have read this on many blogs that, HDFS stores the data in sequential order on the hard disk and it is an abstract layer which stores data on big blocks rather than default block size storage provided host file system. If we are not formatting datanode than the powerful feature of HDFS will come to toss.

[@CosmeJunior]

Nice Job. Brazil thanks you!

[@TzGiwrgos15]

Brilliant!

[@mahdiamrollahi8456]

Hello, nice job, I have a question, as a file system how hadoop can manage a database file(like mssql or mysql) file? how can it devide an .MDF file to other separeted files to store them on different machines? Because such files, have meta data and overhead and they are not like a basic txt file. Does hadoop have special system to treat each file type differently? Regards.

[@abhimanyukarkara4218]

Question: when we have to read from let's say the red file. Would all three nodes be processing simultaneously different data (chunks) and give us an combined output or would only one node process the complete the data processing alone?

[@shyland20]

why s3 service streaming with embedded link is slow (get stock every 2-3 second) when embedded on wp site? after understanding what you saying how can i improve the speed? i read something about the hdfs but i don't understand how it's related to s3 if at all. thanks in advance

[@ulrikkallblad6698]

Very nice video! Only one question: If node 3 is down, how can the data from node 3 be moved to the other nodes?

[@forbin80]

@@brianbitchballs3902 thanks for the great explanation BrianBitchBalls

[@marflem12]

Thank You

[@JackyA123]

yoou haveabsolutely no need to be nervous! Doing a great job here

[@ahmedaj2000]

Thanks 😊

[@taharhalloub8721]

Thanks a lot

[@joseenrique6723]

For the red file, are EACH of the replicas still 64 mb in size?

[@Gorlung]

what does happen when you add a new and empty node?

[@myeverymusic]

What will happen once Data Node 3 is alive again? will the Name Node asks other nodes to copy some data to Data Node 3?

[@AbhinavSingh-oq7dk]

if a data node malfunctions, then name node instructs remaining data nodes to create replicas of files that malfunctioned data node held. why create another replica when there are two others already? I mean they are there for the backup, right? Do correct if I am missing something. Thanks.

The fact that hbase write on a node as you say is the reason why it corrupts the HDFS filesystem so easily?

[@elwyndude]

If a node goes down, why does it need 3 replications to pull the data, could it not just read from the existing two?

[@stivstivsti]

thanx!

[@wow376]

feel like buying Legos already!

[@Irresponsibleful]

did you get a AC by now ?

[@nocontentnoname5922]

Did we find who broke node 3 yet?

[@satwindersetia4367]

Very creative, indeed.

[@malesamuel7736]

Cool

[@amni5tianone263]

tnx

Interesting, it's very similar to how Elasticsearch works

[@KenedyYinkfuChuye]

Now I like Legos :)

[@barefeg]

What if hulk smashes the naming node?

[@GiacomoMilazzo]

I don't understand. If each set of blocks is "one" file (red, yellow, blue) why he says that blocks are replicated? He should say "distributed", not replicated! Replicating involve data resiliency, erasure code and so on. Is it? Then he put the case that one of the cluster's node crash. So in this case replications come on play. And, of course, he should not call the set of blocks "one" file. But it should say there's one file composed of "n" chunks that are replicated among nodes of cluster.

[@draganglumac]

The way I understood it, = . I suppose (if my understanding is correct of course) the confusion then comes from the fact that at the beginning of the video he said that a = . He really should have started with just one row of lego bricks for each file, and just explained that a data node sends a copy of the block it just wrote to one its data peers as directed by the control node.

[@user-oi3ce5nj3m]

狡兔三窟说的就是这个意思

[@user-ml2ci7wl1f]

英语不太好，但是觉得很棒

[@danielleu.877]

SUPER informative, but also i hear "Hadoop" and just think "Hadooken" just me? yeah okay hahaha

[@samiulsaeef2076]

play in 1.25x

[@ravatmehul3906]

Nican

[@guille.p]

It started off pretty well but then it got very confusing. He didn't seem so sure of what he was saying. It didn't work for me. Thank you, anyway.

[@marcelscherzer8385]

its lego, not legos... but nice vid.

[@JM-fp3gf]

Why is he so sweaty?

[@jessetanderson]

Yeah, it was the lighting. We tried moving the lights around, but I didn't have any makeup on which mitigates the lights.

[@musasall5740]

You should not answer this moron. u doing a good job for free

[@vishusingh008]

Looks like you are a moron!

[@vishusingh008]

He replied so kindly and genuinely, you are calling him moron.

[@stonemysterioserusss]

Pretty sure he was referring to the initial commenter, not Jesse. Rude remark nevertheless.

[@viewerone]

It's been quite a challenge to hear this video. Headphones are in but it doesn't seem to help.

[@FredroStarr12]

audio volume is fine to me, must be an issue on your machine

[@viewerone]

Freddy yes, that’s what it was. Guess my Mac needed a reboot. Worked fine afterwards.

[@cafecapes]

Why do Americans call Lego bricks Legos? Lego is a company name and small building bricks is what they make, they don't make Legos! You can't implicitly type Lego bricks as Legos it sounds silly.

[@lucaborzani56]

we do the same in Europe. Where are you from?

[@cafecapes]

I've been thinking deeply about this and decided I'm the worst person to be dictating English. Briton mate.

[@bugs181]

cafecapes Every nation has it's own way of speaking. There's a very elaborate section on the Stack Exchange website that goes into great depth on the difference in languages, pronunciation, and word usage. If this is a serious inquiry, I'd suggest you go there. It's a very informative place to learn anything you want - and if the question hasn't already been asked, you can ask it yourself. One example is that there is a topic on how Americans pronounce the word solder as "sodder" while other countries pronounce it as "sold-er" and where this distinction came from. You might be surprised to know that the language variations have a lot to do with heritage dating way way back. Every language and dialect, regardless of what it is has become bastardized - and it's just a part of how languages evolve. For what it's worth, I used to pronounce it as "sold-er" until I got tired of being corrected - and I have no indication of where I learned this from. I now colloquially pronounce it as "sodder" just because of tradition and geographical linguistics. Also, pronunciations and accent vary widely in the United States from coast to coast also. Apologize for the long comment.