Understanding HDFS using Legos

[@davidvella7141]

This video was uploaded 5 years ago and it still one of the best explanations I've ever seen.

[@ThorstenStaerk]

the first explanation at all that could tell me what Hadoop has to do with MapReduce

[@stackinglittlesats]

Nice explanation man, if I could, I would buy you an air conditioner. You deserve it.

[@SIRabhinav]

may the force be with you

[@miguelchris6374]

sorry to be offtopic but does anyone know a trick to log back into an Instagram account? I was stupid forgot my account password. I would love any tricks you can give me.

[@miguelchris6374]

@Billy Dariel it worked and I actually got access to my account again. Im so happy! Thank you so much you really help me out!

[@billydariel9140]

@Miguel Chris Happy to help :D

[@ZeTamboh]

@@billydariel9140 nice ad

[@taxatlanticinc6611]

This is a best explanation I have seen yet! It's a lot more engaging and informative than the traditional PowerPoint! Thank You!!

[@maryoleary8660]

I love learning with legos, even watching it at 1.5x, I was able to follow along easily. Well Done.

[@TheMrsStinsfire]

11:06 R.I.P. Data Node 3

[@supermonkey965]

He was a good node, admired by his node friends.

[@ElCuchu]

I'm still crying, can't get over it, such a good node dude rippp

[@parthnagdev]

He is happy in the Node Heaven and is saving all the replicated data it ever wanted to save.

[@wisdomandpeace4897]

Excellent video. I actually understand Hadoop somewhat after watching this video.

[@shaikabdussalaam5431]

You really have a " hands-on" approach of teaching this. : )

[@Coffingdw]

Nice job Jesse. Very informative and creative. Thank you. TeraTom

[@MuzamilKhan-rl2sh]

Wow man, you explain it in a creative way.

[@AdrianRodriguezWebDevelopment]

This video just made my day! Thank you New Circle Training! And thank you Doug Cutting for sharing this video on Twitter.

[@thandekilenzungu7240]

The explanation is so clear I understood everything

[@amitprakashpandeysonu]

Really nice and innovative way to teach hdfs concept. loved it and understood it very clearly. Thank you.

[@thndesmondsaid]

Jesse! Great explanation as always.

[@Ahlambabes]

Amazing explanation !! one of the best videos I've seen about HDFS

[@ravianantharamaiah7567]

Excellent teaching. Conceptually things are very clear now. Thank you.

[@olesyagorbacheva6991]

Thank you for such a good explanation!

[@haydo8373]

Superb, I had it running at 1.5x and it was still easy to follow! Thanks :) Can you explain every CS concept with Lego? - that would be amazing

[@juiliussmith5702]

thanks dude lol

[@klausdupont6335]

Incredible illustration! Would love to see more on this topic in this form :-)

[@TheJesus172]

Thanks, Nice explanation

[@ThomasEhardt]

Great introduction to HDFS!

[@prohouse6088]

very nice teaching methodology jesse, thanks for sharing

[@gustavogbfBR]

Nice work. Really help me to understand how HDFS works.

[@CB-fz3li]

Nice clear explanation

[@bugs181]

I'm just now learning about the methods used in distributed file systems. I'm an application developer and it's a bit difficult to wrap my head around the lower level storage systems like HDFS. This video explained replication in an easy to understand way. Now only if I could have one other BIG question answered. What kind of file system would we use if we want an application to use a virtualized file system stored over many nodes? For example, we want each node to add additional storage capacity. To the application layer, this would look like a single big storage drive but to the lower level facilities this would use network coordination to serve the files to the application.

[@kishorpatil6260]

ĺm ĺ

[@yicai7]

Great explanation! Voted!

[@dingman081130]

gorgeous presentation, thanks

[@abhijeet_r]

Very innovative presentation thanks a lot!

[@markhellel3371]

Great Job Jesse! Nicely done! :-)

[@Happymoon789]

Thanks for your efforts! Smart display!

[@1234abcd2139]

nice illustrative way of teaching HDFS. Would have been wonderful if some more information was given about fallback mechanism of Name node or coordinator

[@joecordingley7071]

This was great, thanks.

[@CarlosTheGreat-j7i]

Excellent video, thanks!

[@i_e_she]

This was great, thank you! Should have more views.

[@Gorlung]

what does happen when you add a new and empty node?

[@Guitarman01]

Good Presentation, however I do have question. Since the file is split to other nodes, doesn't replication also take places so that if a node does go down, then you can retrieve. Node 3 went down, but I would have figured I could get it from another node. Does master save a copy of all files as well? I didn't see how that works on the video.

[@MSDlublin]

Very good work for begginers - THANKS A LOT!

[@alaayari6391]

thanks for the explanation

[@manojprabhakar5522]

Awesome, Thank you for the explanation, Could you please make videos of Spark with Yarn and how the communication is handled?

[@nirupamaj6140]

very informative, thank you

[@mrdhksan]

Excellent, thank you. A serious question: What would happen if two of the four nodes crash?

[@ulrikkallblad6698]

Very nice video! Only one question: If node 3 is down, how can the data from node 3 be moved to the other nodes?

[@forbin80]

@@brianbitchballs3902 thanks for the great explanation BrianBitchBalls

[@elwyndude]

If a node goes down, why does it need 3 replications to pull the data, could it not just read from the existing two?

[@kausaralam2605]

Great explanation!

[@abhimanyukarkara4218]

Question: when we have to read from let's say the red file. Would all three nodes be processing simultaneously different data (chunks) and give us an combined output or would only one node process the complete the data processing alone?

[@joseenrique6723]

For the red file, are EACH of the replicas still 64 mb in size?

[@barefeg]

What if hulk smashes the naming node?

[@nocontentnoname5922]

Did we find who broke node 3 yet?

[@BabtaOfEinGedi]

Perfect! Thanks so much

[@AbhinavSingh-oq7dk]

if a data node malfunctions, then name node instructs remaining data nodes to create replicas of files that malfunctioned data node held. why create another replica when there are two others already? I mean they are there for the backup, right? Do correct if I am missing something. Thanks.

[@jesusoliveros9950]

Amazing !!! great Job

[@wow376]

feel like buying Legos already!

[@arisweedler4703]

Great explanation! I assume that another benefit of the HDFS is that reading large files will be quicker, because you would be able to effectively "BitTorrent" from your cluster. Does HDFS do that?

[@Irresponsibleful]

did you get a AC by now ?

[@buzz-uk]

Hi, While setting up pseudo or full cluster, do we need to format data node with HDFS file system or we only have to format namenode. I am asking this because, I have read this on many blogs that, HDFS stores the data in sequential order on the hard disk and it is an abstract layer which stores data on big blocks rather than default block size storage provided host file system. If we are not formatting datanode than the powerful feature of HDFS will come to toss.

[@myeverymusic]

What will happen once Data Node 3 is alive again? will the Name Node asks other nodes to copy some data to Data Node 3?

[@mahdiamrollahi8456]

Hello, nice job, I have a question, as a file system how hadoop can manage a database file(like mssql or mysql) file? how can it devide an .MDF file to other separeted files to store them on different machines? Because such files, have meta data and overhead and they are not like a basic txt file. Does hadoop have special system to treat each file type differently? Regards.

[@rimchatti3807]

Nice job, it is helping getting familiar with HDFS. I'm new to Data Engineering and so on.. Could you please explain to me what is a cluster. Thanks;

[@JackyA123]

yoou haveabsolutely no need to be nervous! Doing a great job here

The fact that hbase write on a node as you say is the reason why it corrupts the HDFS filesystem so easily?

[@CosmeJunior]

Nice Job. Brazil thanks you!

Interesting, it's very similar to how Elasticsearch works

[@TzGiwrgos15]

Brilliant!

[@marflem12]

Thank You

[@shyland20]

why s3 service streaming with embedded link is slow (get stock every 2-3 second) when embedded on wp site? after understanding what you saying how can i improve the speed? i read something about the hdfs but i don't understand how it's related to s3 if at all. thanks in advance

[@taharhalloub8721]

Thanks a lot

[@satwindersetia4367]

Very creative, indeed.

[@stivstivsti]

thanx!

[@GiacomoMilazzo]

I don't understand. If each set of blocks is "one" file (red, yellow, blue) why he says that blocks are replicated? He should say "distributed", not replicated! Replicating involve data resiliency, erasure code and so on. Is it? Then he put the case that one of the cluster's node crash. So in this case replications come on play. And, of course, he should not call the set of blocks "one" file. But it should say there's one file composed of "n" chunks that are replicated among nodes of cluster.

[@draganglumac]

The way I understood it, = . I suppose (if my understanding is correct of course) the confusion then comes from the fact that at the beginning of the video he said that a = . He really should have started with just one row of lego bricks for each file, and just explained that a data node sends a copy of the block it just wrote to one its data peers as directed by the control node.

[@malesamuel7736]

Cool

[@amni5tianone263]

tnx

[@KYC_life]

Now I like Legos :)

[@danielleu.877]

SUPER informative, but also i hear "Hadoop" and just think "Hadooken" just me? yeah okay hahaha

[@喃來呗往]

狡兔三窟说的就是这个意思

[@samiulsaeef2076]

play in 1.25x

[@李龙-h9v]

英语不太好，但是觉得很棒

[@cafecapes]

Why do Americans call Lego bricks Legos? Lego is a company name and small building bricks is what they make, they don't make Legos! You can't implicitly type Lego bricks as Legos it sounds silly.

[@lucaborzani56]

we do the same in Europe. Where are you from?

[@cafecapes]

I've been thinking deeply about this and decided I'm the worst person to be dictating English. Briton mate.

[@bugs181]

cafecapes Every nation has it's own way of speaking. There's a very elaborate section on the Stack Exchange website that goes into great depth on the difference in languages, pronunciation, and word usage. If this is a serious inquiry, I'd suggest you go there. It's a very informative place to learn anything you want - and if the question hasn't already been asked, you can ask it yourself. One example is that there is a topic on how Americans pronounce the word solder as "sodder" while other countries pronounce it as "sold-er" and where this distinction came from. You might be surprised to know that the language variations have a lot to do with heritage dating way way back. Every language and dialect, regardless of what it is has become bastardized - and it's just a part of how languages evolve. For what it's worth, I used to pronounce it as "sold-er" until I got tired of being corrected - and I have no indication of where I learned this from. I now colloquially pronounce it as "sodder" just because of tradition and geographical linguistics. Also, pronunciations and accent vary widely in the United States from coast to coast also. Apologize for the long comment.

[@guille.p]

It started off pretty well but then it got very confusing. He didn't seem so sure of what he was saying. It didn't work for me. Thank you, anyway.

[@marcelscherzer8385]

its lego, not legos... but nice vid.

[@viewerone]

It's been quite a challenge to hear this video. Headphones are in but it doesn't seem to help.

[@FredroStarr12]

audio volume is fine to me, must be an issue on your machine

[@viewerone]

Freddy yes, that’s what it was. Guess my Mac needed a reboot. Worked fine afterwards.

[@JM-fp3gf]

Why is he so sweaty?

[@jessetanderson]

Yeah, it was the lighting. We tried moving the lights around, but I didn't have any makeup on which mitigates the lights.

[@musasall5740]

You should not answer this moron. u doing a good job for free

[@vishnurajbhar007]

Looks like you are a moron!

[@vishnurajbhar007]

He replied so kindly and genuinely, you are calling him moron.

[@stonemysterioserusss]

Pretty sure he was referring to the initial commenter, not Jesse. Rude remark nevertheless.

[@ravatmehul3906]

Nican

[@ahmedaj2000]

Thanks 😊