The $1,000,000 Computer is Broken

Get $25 off all pairs of Vessi Footwear with offer code LinusTechTips at www.Vessi.com/LinusTechTips
Purchase your new G.SKILL DDR5 RAM today:
www.newegg.com/p/pl?N=1000076...
www.amazon.com/s?k=g.skill+%2...
The $1,000,000 PC has been having some serious issues which is why it's taken so long for this video to come out... but today we fix it!
Discuss on the forum: linustechtips.com/topic/14618...
Check out the Kioxia CD6 Drives: lmg.gg/rkDFz
Check out AMD Epyc 32 Core Processors: lmg.gg/5vQAv
Check out AMD Epyc 64 Core Processors: lmg.gg/pN6lh
Check out the Supermicro Storage 1124US-TNRP: lmg.gg/6Qqcu
Check out the Supermicro GPU 4124GO-NART: lmg.gg/bgJiM
Check out Samsung DDR4 ECC Memory: lmg.gg/GAN0z
Check out the NVIDIA MSN3700-VS2F Switch: lmg.gg/0SKZE
Check out the NVIDIA ConnectX-6 200GbE Cards: lmg.gg/rfSEo
Buy Infinite Cables Fibre Cables: lmg.gg/HGrgD
Buy Infinite Cables Custom Cables: lmg.gg/K4JK8
Buy FS.com 100GbE Transceivers: bit.ly/3IX9qpr
Check out RackStuds! lmg.gg/ze86d
Purchases made through some store links may provide some compensation to Linus Media Group.
► GET MERCH: lttstore.com
► SUPPORT US ON FLOATPLANE: www.floatplane.com/ltt
► AFFILIATES, SPONSORS & REFERRALS: lmg.gg/sponsors
► PODCAST GEAR: lmg.gg/podcastgear
FOLLOW US
---------------------------------------------------
Twitter: / linustech
Facebook: / linustech
Instagram: / linustech
TikTok: / linustech
Twitch: / linustech
MUSIC CREDIT
---------------------------------------------------
Intro: Laszlo - Supernova
Video Link: • [Electro] - Laszlo - S...
iTunes Download Link: itunes.apple.com/us/album/sup...
Artist Link: / laszlomusic
Outro: Approaching Nirvana - Sugar High
Video Link: • Sugar High - Approachi...
Listen on Spotify: spoti.fi/UxWkUw
Artist Link: / approachingnirvana
Intro animation by MBarek Abdelwassaa / mbarek_abdel
Monitor And Keyboard by vadimmihalkevich / CC BY 4.0 geni.us/PgGWp
Mechanical RGB Keyboard by BigBrotherECE / CC BY 4.0 geni.us/mj6pHk4
Mouse Gamer free Model By Oscar Creativo / CC BY 4.0 geni.us/Ps3XfE
CHAPTERS
---------------------------------------------------
0:00 Intro
1:26 The problem
3:18 Fixing it! :D
4:46 Booting it up & the specs
7:58 Starting the WekaFS array
9:55 Array speed test
11:38 NASA demo

[@ParadoxalDream]

I like how "thermal paste is expensive" is where Jake draws the line for a $1M storage server.

[@stevejones69420]

Hey man you don't get a $1M storage server by throwing thermal paste around willy nilly

[@bumpyshow745]

​@@stevejones69420 In fact if you begin tô have better thermal management you'll be well on your way towards tha t goal

[@Cylonknight]

@@thelightsilent that is simply not the case, amd had given companies more support in their server market than intel in the last 15 years.

[@giacomosmiderle9717]

@@stevejones69420 he got a point

[@ids1024]

Basic finances. If you just save a bit of thermal paste here and there and don't waste money on avocado toast, you too can afford a house and a million dollar server. Of course!

[@Bane_Diesel]

I once purchased $10k worth of bicycle car racks for my store and only when we started to assemble them did we notice they sent us the wrong products and I needed to send them ALL back. By that point the young guys who were told to clean the store had recycled all the boxes. When I saw Linus's face saying the boxes were gone I felt that.

[@legeaux1]

Down bad 😢

[@wom_Bat]

Oh dang. Did you have to buy a tone of boxes or did they hose you on the refund?

[@Bane_Diesel]

@@wom_Bat we spent weeks making new boxes out of bike boxes one by one as we assembled enough bikes for the racks

[@kjeldschouten-lebbing6260]

​@@Bane_Diesel How about "your problem, you fix or no pay". Works absolute miracles.

[@jonsyoutube6456]

@@kjeldschouten-lebbing6260 Please don't ever employ anyone

[@MaxHaydenChiz]

back in the first episode, wasn't there a guy from the mother board vendor who said it was hard to seat the cpu correctly and that Linus should just let them do it? I bet that guy feels very proud of himself about now.

[@ebaystars]

these guys are like kids in school. I wouldnt hire them or let them have anything more complex than a Nokia phone.....

[@MENACE450]

@@ebaystars lol jealousy is strong in this one

[@dean5224]

@@ebaystars huh??

[@gameguy1337]

@@ebaystars They wouldn't want you to hire them either, they are richer and more successful than you lol.

[@clyde3013]

Except the server with the problem wasn't the one they worked on lol

[@vishinator]

As an HPC engineer, this hurts me yet amuses me watching Jake and Linus's pain. Our entire DDN 7990 array of 10PB was wonky because of one cable causing controller issues, go figure. Love the video :D

[@delquandrecolter3477]

Not as exciting

[@dbattleaxe]

At least they're dealing with commercial hardware. Once you get into truly top of the line hardware, especially proprietary HPC interconnects, there are also so many driver/missing feature bugs to deal with. Some things I've heard about, I'm not even sure how they managed to correctly diagnose the problem. Like on one of the supercomputers I've worked on, around 10 years ago, there was an infamous hardware-level failure that only occurred with one application and only when it was doing a full system run of over 10k nodes, using over a PB of RAM. It turns out that memory was being laid out and accessed just right to cause rowhammer faults. Even scheduling the debugging sessions in the queue for such huge full-system jobs must have been a nightmare. Plus, it's not good to have a several hundred million dollar computer consuming megawatts of power offline and not doing useful work.

[@freddiedelchamps5910]

Be honest, did you drop it?

[@Randomussrs]

yes

[@Loboid]

😂😂😂

[@drlnkspiller]

Lmfaoooooo

[@eaglegaming2019]

😂😂😂

[@StarkillerZ4258]

How is he gonna dropp half a ton

[@TheQuickSilver101]

The best lesson in electronics I ever had was from my father who was an electrical engineer before he retired. He told me "the weakest point and your greatest source of issues is always the connection and connectors. Check those first." Checking all the connections, even if they seem unrelated has saved me so many headaches over the years.

[@phyde1885]

I'm also a retired EE. Your Dad is 100% correct! That is Electronics 101,point of 90% of most failures is to look at the connections 1rst. They are the weakest link in ANY situation of interest and most likely to fail 1rst.Just because it LOOKS and acts TIGHT,DOESN'T mean it's a GOOD Connection ! i.e. good example your car battery ?!? NASTY $H!TZ ! 😎

[@batt3ryac1d]

It sounds so unnatural to think of the cpu as a being plugged into connector to me though even though obviously it is I can see how they'd struggle to troubleshoot it.

[@OlaMagnusLie]

…And your weakest point in troubleshooting is the wetware not being able to process the input.

[@themonkeyman2547]

My dad told me something similar, “it’s always a mechanical failure”. The electricity is never going to fail to flow, but the path can be broken

[@fitybux4664]

@@themonkeyman2547 depends if you are lumping mechanical into the chemical realm as well...

[@penatur]

Reseating a CPU would have never occurred to me to be a troubleshooting step for anything other than CPU errors. It's amazing just how many small things can cause issues that one would not even consider.

[@jayhollowayii2]

i love how the server sounds like its ready to take off

[@tonyspencer9350]

I started building computers nearly 30 years ago, 286, 386, 486, P90. Today's tech blows my mind with its speed and power. I remember building the first SCSI-II based system in Oregon for a logging company to do virtual logging using aerial photos. Noboday had heard of, thought of or could comprehend a GPU back then. OMG, the power of this beast.

[@markkoops2611]

Same, I've used everything from a C64, and despite that I'm still wrapping my head around this... One thing I don't quite get... If the drives pcie is direct to the CPU, how the heck does directstorage go straight to the gpu?

[@sisamusudroka3000]

@@markkoops2611 simple, through technosorcery of course!

[@giornikitop5373]

@@markkoops2611 it's simple, it doesn't. the cpu is still in full control of the bus. the point of DS is, the cpu (apart from accepting the transfer and "releasing" the bus) is not participating in the data transfer at all. the dma controllers do all the work so the cpu is almost at idle. same for network rdma.

[@giornikitop5373]

gpu designs were a thing since the late amiga era but technology was not yet at a point that it was doable,

[@donc-m4900]

They were still putting the math co-processor on the cpu. (SX/DX)

[@ariakingstrom6212]

Using that NASA data for the demo is such a good idea. It really helps contextualize the sheer amount of computation packed into that server

[@paulbecker7039]

I still don't get why this is so huge, should't it be a pretty simple computation?

[@DarkSession6208]

@@paulbecker7039 Guess its not that simple of a computation if it even runs like ass on a $1M Dollar System

[@zachsmart8663]

@@paulbecker7039 every single frame of that demo is running millions/billions of characters worth of formulae trying to decipher the perfect way to land that craft. Air pressure/Fluid Dynamics/Delta V/etc are all being ON THE FLY computed and changed every single time that frame updates. Trying to make a computer do a fluid dynamics computation or render of any sort is already taxing, now do it with an atmosphere that humans have never physically touched. Those calculations alone make every single frame change worth 14GB of data to the render. Meaning that 5FPS is ~70GB/s of OVER NETWORK processing power. That's the same amount of data as The Lord of the Rings: Fellowship of the Ring extended directors cut on 4K blu-ray. Every. Single. Second.

[@ProjectPhysX]

@@zachsmart8663 the amounts of data are impressive, yet already behind the state-of-the-art. I can already _run_ simulations 10x as large, with 120GB volumetric data/frame, on such a visualization server with 8 GPUs in a matter of hours with my software. In CFD per grid point the set of equations is rather simple, but the number of grid points makes file size so gigantic.

[@zachsmart8663]

@@ProjectPhysX really cool to see another layer deep on how this stuff actually works. I know Linus wouldn't probably have his hands on something cutting edge (especially given the layer of dust he scraped from the top at the start), I was simply giving a response as to why the render size was so large, and how 5FPS for that specific project was still pretty incredible. Thanks for coming in and giving even more insight, I'll have to dig a little deeper on some of the topics you brought up

[@Kadranos]

That visualization is really cool. When I did my CCNA class in 2006, I said someone could make a network management program that gives a visual map of your network that looked a lot like that (in my head... it was a bit sloppier when I tried drawing it). With the layer 2 traffic Cisco has flying around, you could even have it update in real time and show and log new devices connected even for a short time without asking for an IP. I was told that that wouldn't make sense because its just too much trouble to bother with an app or GUI and the CLI does everything you need better than any GUI ever could. I think some things like that came out a few years later (not that I would have known how to make it).

[@Adroit1911]

It reminds me of the original Jurassic Park. "It's Unix"

[@WeirdTippy]

That part at 3:55 where Linus almost dropped that cpu almost gave me a heart attack. Whether he was faking that drop or not, I could'a died

[@robertdascoli949]

"The solution was shockingly simple, and I bet it's something you've seen before." Did you turn it off and then turn it back on Linus?

[@DeathProductions200]

I see you work in IT

[@andrewamann2821]

@@DeathProductions200 first steps are first steps for a reason, my dude.

[@leonro]

that was literally the case lol

[@TheMetroidblade]

I mean that’s essentially it. Take it out and put it back in

[@therodyman700]

No, Turn it off, take all non-necessary components out. turn on, does it work? yes? put in 1 part, does it still work? yes? next part. Rinse repeat until it stops working again. Found your broken component. replace, test, show customer it works, clock time, next assignment. Sorry for telling all your secrets HPE

[@Ivan-pi6ur]

Finally, a computer and me have a same thing in common, I'm also broke

[@EpikoXailia]

The difference is, you probably aren't worth $1mil

[@estebanguerrero682]

MA MAN

[@darkmann12]

lol, the irony

[@Lunatic5306]

Financially or emotionally?

[@MattyEngland]

Sell your granny for medical research

[@jacobl2179]

This video and the presentation almost made me cry. I come from a academic research background and we look at the ear and how certain diseases can affect hearing. I can say that the machine that Ltt just showcased here is going up on my next presentation on how to scale up hardware for or next big paper.

[@ScottJWaldron]

Props to whoever did the sound editing on this video! You somehow managed to deal with that intensely loud server.

[@djsaekrakem3608]

IPMI CONTROLS FAN SPEEDS

[@VIN-7567]

The NASA example was a great touch to show how tech like this is actually used. Shout out to the LTT team for getting that done.

[@and_I_am_Life_the_fixer_of_all]

Major respect for the LTT devs, good shit.

[@Maxam-Media]

Giving real world examples of technology like this really helps not only understand the need for this tech, but also how it can help the STEM field in the future. The NASA demo was really cool to see in a video like this where when I started watching I didn’t really see the need/purpose for a server like this besides storing files quickly.

[@ddognine]

You obviously don't have much experience with video production. I am not sure what workflows LTT uses, but I know exactly why they need this. They use RED cameras and probably shoot in 8k 90% of the time (those cut ins aren't zooms), or maybe they don't but would like to and can't because they don't have this server up. That may not be a heavy lift if you are one dude at a workstation, but when you have a whole team of editors hitting the same server day in and day out, it can really make scrubbing, editing, NR, grading, rendering, etc. a serious pita. People don't realize just how storage intensive video production is these days. That's one reason LTT goes around helping KZbinrs install their own servers because you don't have to be computer geek to know when you are spending more time watching an hour glass spin than cranking out video. And this doesn't even include compositing which really pushes the envelope and I've seen LTT use, although sparingly.

[@gb76231]

@@ddognine yeh

[@aSaltedPeanut]

@@ddognine LTT isn’t keeping this, it was on loan

[@Jehty21]

@@ddognine no. Watch LTTs other videos where they actually build their own servers for their video production. This server is waaaay over kill for video production.

[@zachsmart8663]

@@aSaltedPeanut yeah it was on loan, LTT already has a 2.5 petabyte server of their own.

[@GodlikeIridium]

12:35 "From 12000 mph to, you know, the ground" Well that's easy 😂 The hard and important thing is to get to the ground at 0 mph 👌😅

[@BlokeOnAMotorbike]

that isn't difficult, the hard bit is making the deceleration *survivable*.

[@BAoxymoron]

As someone who works on computers, I thoroughly enjoyed watching the satisfaction on your face after you finally figured out the issue. It can be pretty painful trying to find that super small issue that is completely breaking everything.

[@Sonix.2016]

I fought a fringer scanner for two days tried any and every driver. Turn out to ne the fn usb cable. I work for the feds so this was really funny.

[@iAmVonexX]

yep. i recently had a loose connection in a fuse INTERNALLY. damn that took some time to figure out. of course it always connected when i tested the fuse...

[@BagelTwist]

My bet is that Linus dropped it 😂

[@gabbythegamer79]

Here before it blows up also agreed

[@Pixlator]

linus drop tips

[@dabagelgd]

lol

[@Hellocrafting]

I expect nothing else

[@frunzadaniel7495]

🤣🤣🤣💀

[@johnmarianhoffman]

I know the target market for these particular LTT videos might be a little smaller, but I LOVE to see y’all work through these systems that really are at the absolute peak of computing performance. By far my favorite of all the different LTT sub-genres!

[@why_tho_]

I second this.

[@daenemark_mark]

So do I.

[@championxxlNL]

Well I mean it's not like I'm gonna build the budget pc so I find this more interesting to watch

[@Carnage8]

Insane. I am so glad ltt had the opportunity to share this technology. I loved this series. The omega of your storage videos for sure.

[@frankcross2297]

Nice fix. Found similar issues on boards that needed one more cleaning in the ultrasonic which cleared up drive issues.

[@kedharsairam33]

Jake: Did I tell you they threw away the boxes. Linus: sorry what? OH MY GOD. (I love to see how Linus reacts to this situation, hehe).

[@DuyNguyen-yx2vd]

Between Jake "I threw away the boxes" and Linus pretending to drop the obscenely expensive CPU... I think they're even.

[@davidd.234]

@@DuyNguyen-yx2vd except their not even because if the cpu actually got damaged it more then like comes off linus's business bottom line.

[@tannerhughes6274]

Still really excited for the LTT screw driver to arrive that I ordered! Going through a aircraft mechanic college program right now and excited to use the LTT screw driver during my training and career. Yes, I’m training to be a aircraft A&P mechanic, but I’m also a nerd and have build the two computers that I have!

[@Neadrik]

Oh my god, the number of times this happened when I was assembling servers for Citrix. Always falling out on the stress test with incosistent errors. The first time it happened it took a month to figure out that we just needed to reseat the CPUs 😭

[@aswa121235]

You know the funny part is back in April Jake was super confident about installing CPUs in the first video when the Supermicro's representative raised his concerns. lol

[@LucasOhaiFilgueiras]

You didn't pay attention, the server wich they worked on is not the one with problems on the cpus, so the fault here is on the person who made the server before it got on Jake or Linus hands.

[@SivaKanthSharma]

I cannot believe that the demo they gave you was actually impressive even to a layperson like me. Mind blown

[@ProjectPhysX]

It's already outdated though. My software can do simulations with 10x that resolution on half a day on a single 8 GPU server node instead of an entire supercomputer :D

[@lonelypotato3001]

@@ProjectPhysX Can your server do it live like this one can?

[@ProjectPhysX]

@@lonelypotato3001 it can compute+visualize the entire thing in a matter of hours, and while the simulation is paused, it can also visualize it interactively in real time!

[@cts006]

@@ProjectPhysX But this was an 8GPU server node. Just with a wicked fast storage cluster tacked on.

[@ijmad]

Build it and they come! Doesn't take people long to come up with problems to solve on hardware like this does it? Count on NASA to have some incredibly big problems to solve that fits right in...

[@jarrymay1]

1. Loved it. 2. Some data from the nvidia profiler would be sick. If you could upload to the forums or float plane to see the gpu memory activity in real time.

[@2_Idiots_Shooting]

I work at the NASA facility in Fairmont Wv and so it makes me so curious as to where that server is going to end up. It would be so cool to see it in person after you guys reviewed it, knowing its intended function.

[@alexander1899]

I love buying a PC worth $1,000,000 and it breaks

[@falsemcnuggethope]

The title was just a clickbait

[@Nadirisim]

aww i broke it good job Linus

[@hariranormal5584]

They anyways got that given to them for "free" :v

[@S31Syntax]

Had a family friend acquire this HUGE multi-motherboard special linux server like 20 years ago, he was SO EXCITED. Killed it with static discharge within 10 minutes of opening the side panel. It sits next to his desk as an end table to this day.

[@hariranormal5584]

@@S31Syntax Jeeeeez.

[@assafweiss8078]

The NASA bit looks like real time rendering, not real time simulation solve which are 2 different things. The solve process takes a week on a cluster super computer, and then you need another less "super" (but still a magnitude of order more powerful than a desktop) computer to render the results (its less rendering, more like displaying the physical results which comprise of tens of millions of cells in this kind of model). This kind of "rendering" is much more storage speed dependent and much less compute dependent.

[@waggy401]

I could be wrong, but didn't they show a screen on which they could change variables? That would imply simulation.

[@nesyboi9421]

Thank you for your insight and knowledge.

[@assafweiss8078]

@@waggy401 I highly doubt this, since this kind of simulation takes a few days to run on the largest supercomputers in the world. I guess whats happening is they changed the variable being rendered, but the results being used for the rendering are pre-solved.

[@waggy401]

@@im.thatoneguy Yeah, after I posted this I saw that the actual calcs were done with 72,000 GPUs or something like that. Crazy.

[@engineeringvision9507]

There is no need for fancy graphics when solving a systems model like that. Definitely a render type setup. They might have a very high fidelity model for testing but you can easily simulate such a vehicle with a decent desktop pc using a reduced model and it will still be valid.

[@abs_official]

0:58 Linus seriously outdid himself on this "sponsor segway". Absolutely brilliant!

[@pakkazull8370]

segue

[@the_reletubby4466]

@@pakkazull8370 yeah some people need to really educate themselves bro

[@FusionC6]

This has solved a lot of issues with my old retro battle stations... took out it of storage after a few years and it wouldn't post, gave the cpu heat sink a wiggle and she's now purring along for hours.

[@sry4head]

Jake and Linus have became such a good team. I remember some video from the beginning, where this wasnt the case. But this is still the best Duo so far.

[@Fusion05]

I just don’t like Jake. He seems like a jerk to me.

[@Fusion05]

I’m sure he’s a decent guy, but he just gives me the wrong vibe

[@PixelSheep]

@@Fusion05 this is one of the best responses I have read in a while tbc I personally like him but this comment gives me hope that there are still people out there in the internet saying their opinion while also seeing that it's just them personally I hope anyone gets what I am saying - and again - it's not my opinion - I personally like Jake :D But Jesus Christ does this comment give me hope

[@williamhansen9456]

@@PixelSheep I used to not like Jake, but he grew on me.

[@Fusion05]

@@PixelSheep yeah I don’t like acting that my opinion is better than everyone else’s.

[@wulfgarpl]

Power grid operators in Linus's city: Oh, no. We need to upgrade again.

[@karaloop9544]

3:56 I like how Jake feels with the hardware on Linus' drop joke, even if it has zero consequences for him personally should anything go wrong. That's true love. :)

[@hennesreekers9106]

Funny they even mentioned the cost of thermal paste and not the tens of dollars they just spent on raw electricity for the amount of power the server used during this video..

[@yankumarrah]

Can't wait the 3-5 years it takes before I find storage like this available on the cheap!

[@flameraker6824]

Until they flip this and make one out of soldered in ram chips

[@rpgamedeveloper5034]

Lol, yeah.

[@mrbanana6464]

more like twenty years

[@legeaux1]

50 years😊

[@pistol0grip0pump]

Goodwill gonna be kickass 😂

[@Tech-Tempest]

This video may have inadvertently solved an issue I've been having with my M2_1 slot. The NvME drive would just randomly disappear when its reading or writing data and I need to restart for it to come back. Will report back!

[@zachsimsphoto5344]

Commenting so I hear the outcome.

[@Tech-Tempest]

Well, no dice for me after reseating my 5900x. Guess it really is a bad M2_1 slot on my X570 Tomahawk. Tried it with a new drive as well

[@MasterCode86]

@@Tech-Tempest shame to hear that bro.

[@ShotGunner5609]

Glad you found the problem at least!

[@akirafan28]

@@Tech-Tempest Sorry to hear. Is still within the time to claim warranty?

[@callmemongo]

Awesome Sauce🤯 I'm having flashbacks to my 1989-1991 college years working HYPERCUBE THEORY and the complexities of designing some of the 1ST GEN HYPERCUBE system models with early Pentium and Motorola 68000 processors in arrays of n³

[@MdoubleJay]

As a mechanic, I totally get the intermittent now that I found the issue it makes so much sense thing. Half my job is staring at I wiring diagram and going, oh…. I bet it’s here. Then Finding something like burn marks on the tiny pins in the connector or a harness that rubbed on something and is shorting out but only over big bumps. It can be so simple, after you figure it out.

[@mustdobetter6748]

Good job finding that! It’s so satisfying when you find the answer to a really weird problem.

[@sehrverwunderlich95]

No, I did not forget about this. And I was getting kinda sad that you seemed to have dropped the project. Would have loved to see more of the building & troubleshooting, this is a fantastic piece of tech, very impressive.

[@jackpowell9276]

yeah i thought it had been canned. Great to see it come back!

[@heavyarms55]

Mad respect for trouble shooting that. I've recently come to the conclusion that I am not cut of for this kind of work. I swear that 90% of tech issues is just FINDING the thing that isn't working right.

[@mocko69]

Supersonic retro propulsion is actually used to shield a craft from reentry heat too, not only landing. Falcon 9 uses it

[@BakersTuts]

_But you know who will never be broken? Our sponsor!_

[@derteater]

Yes

[@KingLarbear]

Lol

[@eshxplorn]

I've seen this with WAN circuits in metal boxes in afternoon sun, for example. Very sketchy connector would lose the connection in thermal expansion and the circuit would drop. Sun goes down, circuits comes up. That was a fun one.

[@ackkipfer]

Omg this segway was not expected this time! Wonderfull!

[@CptVein]

I say it everytime but god DAMN I LOVE Linus and Jake's chemistry. The fact that Linus is Jake's boss but Jake acts like the boss because that way, Linus can just have fun and not worry (too much) is magical. Jake truly is Yvonne's husband boyfriend.

[@lilkittygirl]

You mean Linus' husband/boyfriend. Work husbands! haha

[@triopical6884]

@@lilkittygirl Jake is perfectly thick 🤤🤤

[@zeroumus]

golden things to consider when you have a computer that misbehaves depending on if its been on for or while or not. if it stops working after warming up, check the thermal interfaces, if works after warming up, assume your dealing with a solder or contact problem.

[@kian1259]

I see that you guy's have made the intro more fast-paced. Very good move for retention. As always, keep it up the great work :)

[@markstott6689]

I enjoyed this so much that I thought there was something wrong with the video. It felt like I'd been watching for 5 or 6 minutes when it ended not 16 minutes. Obviously it's the power of Linus and Jake together.

[@ericcrear1359]

I've been watching your stuff for a while. You've had some nerdy shit before. But you were definitely bringing it up a notch on this one. Freaking out about wattage and transfer rates and some visualized representation of the hard drive array. Amazing.

[@88porpoise]

12:50 So, you are saying that NASA made an HD remake of Lunar Lander?

[@ExoKiller4]

Bad socket contact can cause truly bizarre issues and if you don't have a reason to expect it being an issue it's the absolute last thing you expect.

[@ArturoTabera]

The last thing you expect is the Spanish Inquisition.

[@memediatek]

@@ArturoTabera no one expects the Spanish Inquisition

[@CL4K.]

@@memediatek the Spanish Inquisition?!?!?

[@tactileslut]

With pin counts around a thousand, expect it more.

[@DeinonychusCowboy]

3:40 I love LTT and love that they made the screwdriver, but here it is, the thing I was most worried about with their existing driver sets and approach to bits in general. Never have that problem with ifixit driver kits. (I'll probably still buy a screwdriver at some point, but I hope they come up with a more diverse bit set than the current ones at some point, and/or a better solution for storing more bits)

[@Jack-yl7cc]

They are using standard size bits that are essentially cut in half and there is already a source for those kind of bits. Check out the VIM Tools VHC77 Half Cut Bit Set for a set that contains just about every kind of bit you could want. I would also recommend getting the VIM Tools VIS112 Master Ball Set for dealing with off-camber access issues; although those are full height bits.

[@GodlikeIridium]

So it very probably isn't drop damage 😄👌 Pins not having perfect contact is probably the common problem with those EPYC CPUs. Their retention system really isn't perfect. I think the next socket with the cooler pressing the CPUs down, instead of the socket mechanism like now, will improve contact. And it's not obvious to diagnose, but if you know it, it's easy.

[@jonatansteuernagel1264]

Finally! I have been wondering when this project would finally be continued. Crazy tech.

[@gfdggdfgdgf]

LTT: how to build a budget gaming PC. Also LTT: yeah our 1 million dollar PC broke.

[@coreyh3994]

14:36 you guys should try and get Wren from Corridor to visualize all that data per second for you! he'd love it.

[@MichaelMichael-kv4gp]

I wish Amd or Intel would come out with a piece of software to show which pins are detected and which ones are not. I have a old p76 motherboard where a few ping on the motherboard i know are bad and not touching the CPU and in turn 1 of my dimm slots does not work. It would cool to get a visual and see if all the pins are in contact through a GUI or heck the bios worst case. I don't see how this would be tough to do as the motherboard and cpu talk to each other and they know which ones are on and off... just display the info??

[@lost1110]

The need for something monstrous like this made sense when you talked about the implementation of it. Amazing!!

[@gobbel2000]

I find it great that they got an actual workload with very high resource demand as a demo.

[@handlemonium]

Honesty they're gonna need that 7,000-10,000 watts of computing power if 7-channels' worth of real-time editing and rendering is gonna be done without any workflow hiccups.

[@aortsashah]

i work with these type of servers and let me tell you, what linus say is absolutely true about the noise. it's literally ear deafening when im at the racks, and the heat from the back is insanely hot.

[@ColonelSixPack]

I want to see this server used for running a complex DCS World mission. I would bet the most complex simulations run flawlessly on it. A flight sim would be a great way to load test what the server can actually do in a "practical" application too.

[@BigColton]

Big sad. Was hoping they got to keep the big server :( I can imagine some cool experiments could have happened

[@BlueKnight87]

If that was the blade that they took apart to assemble for us on camera, we all know SuperMicro gonne be like "we told you so!"

[@jesilvas0813]

I love that Jake still chuckles at sponsor segues.

[@TheBurg229]

We need a million dollar server to land on Mars when we landed on the moon with a pocket calculator and a slide rule.

[@KubaEkaj]

We also crashed a *couple* of rockets while trying to do that...

[@lunakoala5053]

"Only 75 Apollo Guidance computers were ever made, and on average, they each cost the United States around $200,000 (equivalent to $1.5 million today)."

[@arjunyg4655]

The moon landing used plenty of computers lol…

[@ringstar3316]

I’ll give them 2 weeks before it’s broken and they go for something else again that’s somehow even better before repeating the cycle.

[@ShiroKage009]

This is an insane piece of gear. Holy crap. I can't believe I forgot about it until now.

[@Kirmo13]

11:52 the off-frame side kick really got me here

[@Jack-yl7cc]

I can attest that anytime you receive a server with components pre-installed(especially if it is delivered in a rack) you will run into intermittent issues that can be resolved by reseating components. I have seen this issue with every major server manufactory. Most common issue I've seen is DIMMs that aren't fully seated followed by PCIe devices.

[@Vazzdom]

At 4:07 Really shouldn't reuse thermal paste when it has already been used and heated up and left for a long time. Creates air pockets since the thermal paste has already cured before (Might be okay if you just did it that day and had to readjust the CPU in the socket or CPU heatsink to touch the IHS properly if it wasnt making proper contact, but Ive always put new paste because ive seen it cause issues in the past. I.E Overheating on one or two cores because of bad thermal transfer or just bad temps in general)

[@benwu7980]

I would be sure they did it properly off screen, cpu was already identified as the root of the issue before making the video so they just made a bit of fun content about the fix. ( not that thermal paste was the precise issue)

[@Vazzdom]

@@benwu7980 you didnt get my point at all. Had nothing to do with what was wrong with it in the first place so ill re-explain it to you and I highly doubt they repasted it if they didn’t put it in. All I was saying was basically to dumb it down for ya: Reusing cpu paste that has already cured will create air pockets creating bad heat transfer. Over time it will get worse. (Built enough computers pentium 2 and onwards to the newest gen stuff that it will affect proformace and thermals of the cpu. Nothing that I said had to do with the video problem, just another one they are creating for theselves in the future.)

[@benwu7980]

@@Vazzdom Oh, I got your point, and I fully agree. I would never refit anything like a cpu and cooler without a clean and a fresh application of paste. All I meant was that I'm fairly sure they did do that, but just went with the more entertaining route of Jake complaining about the price of thermal paste, on a 1m$ computer :)

[@That_Guy_Named_Epyk]

I love seeing a ltt upload and watching within the first 10 minutes keep it up!!

[@sherrilltechnology]

That is an awesome server or rather servers!! Great video Linus!

[@theaterlightman89]

It'd be cool to see this rig handle some computation for Bret from ufd and what he's trying to get done

[@dylandreisbach1986]

As much as it sucks for expensive things to break, I bet there's an excitement in being able to make a video on it.

[@falsemcnuggethope]

Especially when it's actually not broken but you just make a clickbait out of it

[@iantaakalla8180]

On the other hand, troubleshooting a really persistent problem would itself be a good video. Clickbaits make a 50/50 of a video.

[@thedarkdade]

At the rate this young kid is learning this stuff. And i am speaking from 20+ years in INFOSEC and IT. Pretty sure hes close to if not ready for an enterprise career should he choose that. Which at least in USA here are xxx,xxx six figure roles. Keep on pushing Jake! As long as ur happy! -Dade

[@benwu7980]

Not sure what paper credentials he's got, but I'm sure he'd easily get any of the basic ones. Having LTT just as a reference alone wouldn't get a lot of enterprise jobs above entry level.

[@thedarkdade]

@@benwu7980 The market has shifted in united states. HS/College/Certs dont matter. They want hands on knowledge of systems and basic to above basic knowledge of security and networking. Obviously it depends on size of company and role. CIA/Homeland/DOD ur gonna need a ged or diploma and have security clearance. ANY Consulting FIrm or MSP would hire him in a hearbeat

[@benwu7980]

@@thedarkdade I would hope so. Thinking outside the box, being eager to learn, and passionate about a subject.. says a lot more to me than paper certs alone. But, at the 6 figure starting salary, it's not about being a Jake of all trades, it's about some mastery of one or more, whether ccnp/ccie level in networks, comptia a+ or s+, or whatever the other types are.

[@gingaming_gg]

I’m sure he’s making plenty of money at LTT. Also can confirm that college/certs don’t matter. What does matter is people who understand the tech a company is looking to use and being able to teach technically illiterate people how to use it. I almost doubled my salary teaching people how to use OneDrive and Citix 💪

[@shaung2965]

That's insane and one I will keep in the memory banks! Thanks for sharing Linus.

[@seankkg]

4:13 Chimps with typewriters comes to mind, but I wouldn't have it another way with Linus.

[@harry2b2t]

I love your work, keep it up!!!

[@Qwarzz]

Even Perseverance (or Curiosity before it) did not land with just a parachute but needed rockets as well.

[@TheByQQ]

Aren't they doing aerobreaking to lose most of the speed though?

[@bigpod]

@@TheByQQ its combination of both retro propulsion and aerobreaking/parachute braking. most aerobreaking with body of probe happens in upper atmosphere during the time heatshield is used then after heatshield is dropped parachute opens to slowdown and at the end retro rockets are fired to slowdown to standstill and fly upper part of sky crane away(in case of perseverance and curiosity)

[@Qwarzz]

@@bigpod Also lowered down while the retro rockets hover the whole things 70ft up.

[@liamashdown7955]

proof that forms of direct storage to gpu connections are super important for the future of computing. really awesome showcase

[@SpaceOddity4214]

I remember It use to take an hour to transfer one gig of data from one hard drive to another, back in the pentium 2 era And that was considered fast!.. Progress is amazing

[@Guillermo_7_XD]

It always amazed me with how genius Jake is, freaking prodigy

[@fitybux4664]

Wait, you think he solved the thermal expansion issue? LOL. Likely a combination of Googling and contacting the company that sent this to them solved this issue.

[@8point6]

@@fitybux4664 genius is "patient" in his native language.. I agree, he probably spent 3 weeks working with the vendor over the phone directly crossing the "t's" and dotting the proverbial "i's" top to bottom, for the sheer purpose of running the NASA demo..

[@djlowtek]

You guys were trying really hard to express how cool that lander simulation is... I think a lot of your audience understands and appreciates it.

[@Trithis2077]

I was so confued back after the last $1000000 computer video because I was trying to figure out if I had missed something or just why the finale never got posted. It all makes sense now.

[@Ckoudous]

People may luagh at the lack of boxes, but boxes for a rig like this can cost upwards of 5k$. I learned this the very hard way

[@aindatenhoconta]

"-Eight GPUs, That's a lot of GPUs. 3.2Kw, that's a lot of kilowatts" Ethereum miners: "heh."

[@AJman14]

I can just imagine how dated this is going to look in 20-30 years. "You needed a full tower for that??" I have a tiny card in my phone that holds 100x that much data!

[@TheRitchieRicher]

Linus, you can use a flathead screw bit on Torx providing it is roughly the right size.

[@yubi-kun958]

I cannot be the only one who anticipated this coming.

[@Focus_Fearless]

Lmao

[@mr702s]

LMG, gotta say, toward the end, this has been one of your most impressive videos for capabilities of a system. To think you could recompute a NASA model like that in real time.... holy crap 😳

[@FluLu]

I thought they were only rendering the data…

[@ProjectPhysX]

@@FluLu yes they are only rendering, compute was a week on 27000 GPUs. Seems very inefficient to me though. My software can compute simulations at 10x that resolution on such an 8 GPU visualization server node in half a day :D

[@pearce05]

@@ProjectPhysX With the same number of parameters?

[@ProjectPhysX]

@@pearce05 only 4 variables per grid point instead of 7, but 10x more grid points, and about 40000x faster lol

[@dipi623]

@@ProjectPhysX Yeah but the NASA code solves a full model of the chemistry of martian atmosphere, plus radiative heating...a whole different ballpark

[@tubaterry]

I switched over to the development side of the house in the last decade or so, but man for some reason I miss these kinds of troubleshooting moments.

[@evifnoskcaj]

Wow, honestly, I would have never guessed reseating the CPU and probably would have tested and replaced everything else and followed traces and checked components, updated bios, replaced cables, backplane, etc before I would have thought to just reseat the CPU. Such a finnicky issue but glad you determined the issue and got it working! $1Mil worth of equipment is not worth wasting.

[@Henk14789]

But why would that error affect the same drives when put in a different slot?

[@ademiravdic]

i dont think they actually said that same drive was affected every time

[@bullettube9863]

It doesn't matter if it's Christmas lights or a computer, when something doesn't work, start at the beginning and check to make sure everything is tight and your connectors all connect properly.