George Hotz on his $15,000 deep learning machine | Lex Fridman Podcast Clips

[@LexClips]

Full podcast episode: kzbin.info/www/bejne/mn_VhaWuaZd3fbM Lex Fridman podcast channel: kzbin.info Guest bio: George Hotz is a programmer, hacker, and the founder of comma-ai and tiny corp.

[@PatNeedhamUSA]

Thumbnail made me think it was a DIY deep ocean exploration machine

[@lule-ahmed]

😂😂

[@0xsupersane920]

This guy is very entertaining.

[@davidcahan]

Which one? George is a little know it all churlish brat. Lex is great!

[@Murlur]

Hotz: I want it to be quiet. I want the Tinybox to sit comfortably in your room. Lex: This is really going towards the girlfriend thing…

[@Martinko_Pcik]

6:36 make it fit the standard central air duct close to furnace. In the winter it will contribute to heating air and in summer the a/c will keep cooling it with the house.

[@Hashartnft]

Great idea actually.

[@johnlocke3481]

I love how terribly that computer will age in 10 years.

[@atillacodesstuff1223]

explain yourself

[@aidilmubarock5394]

​@@atillacodesstuff1223Moore's law

[@aloktripathi27]

"Gravebox"

[@patrickdoherty9116]

@@aidilmubarock5394 Perhaps he should be using a 2034 GPU instead

[@siguc]

I don't understand the pricing model for the tinybox. 1.5KW of power, which is the box's power budget, translates into two 40-series nVidia GPUs + standard PC hardware (CPU, motherboard, RAM, RAID, etc), and even such specs are stretching the power budget [so, in reality, 1.5KW is enough for only one 4090 GPU or 2x 3090 GPUs]. Those 40-gen nVidia GPUs cost under $2K each. So, the components of such a box cost about $7K, if we somehow manage to use 2x 4090s in the box while staying under 1.5KW. Hence we're talking about buying a no-brand garage-built box filled with off-the-shelf components sold at a 50% markup. What am I missing?

[@siguc]

@@comedicsketches Thanks for the info! Would you mind sharing a link to your setup? Also, have you tried to build a 4090-based multi-GPU box? Any info on the power draw and cooling?

[@siguc]

@@comedicsketches Yes, I fully agree re:mods. It's either i) liquid-cool every GPU, while converting those into a single or dual-slot board or ii) employ PCI raisers. Both options a pain to build in a garage. Once you factor the costs of those mods and their maintenance (not sure about the warranty too), it seems to turn out cheaper to build two dual-GPU off-the-shelf boxes, with no mods. Also the dual-box approach is more robust: if the critical component like MB smokes, only one box goes out of service.

[@siguc]

@@comedicsketches Would be great to see a blog post about your build. Given that cloud is rather cost-inefficient for small shops requiring heavy compute, I would find an open source hardware architecture of a cost-efficient PC for GPU-heavy compute way more valuable than yet another ML/AI infrastructure toolkit.

[@thepupil2]

I'm getting a strong Galvin Belson vibe on this box thing.

[@DouglasRenwick]

Gavin B Signature edition

[@johnkost2514]

The PCIe bus will be saturated from memory transfers across GPU(s) on the larger language models. Gaming cards rarely have NVLink or Infinity Fabric. There is a reason the DGX Server/Workstation costs so much more.

[@michaelnurse9089]

AMD need this to be a success so they can catch team green for training. They will provide the right hardware.

[@honor9lite1337]

Make sense.

[@AI-xi4jk]

Epyc has tons of PCIe lanes

[@johnkost2514]

@@AI-xi4jk but it still becomes a bottleneck when you shard large language models across 6 cards. Those 128 PCIe lanes will get saturated at some point and limit the peak performance of the cluster. Just sayin'

[@thomasgrimm1664]

I'm not planning to buy a 15k pc, but that no arousing suspicion part is awesome.

[@Phasma6969]

Bro giggling when he thinks about all that floating point surging through the GPUs

[@Mustachioed_Mollusk]

My man hitting Lex with the dead inside smiles

[@patolobos8266]

It will be great to see a review of the since already is on sale for the last 2 months

[@drewgoodman7932]

Mount about 6 powerwalls to the tiny-box and charge it with solar.

[@PatrickOMara]

i am thinking 65b with specialized tuning for each scenario that you need can go a long long way.

[@__J____ff]

you ll never make it understand as accurately as openAIs GPT

[@Harris08912]

What is lama?

[@andywatts]

Meta’s model

[@Harris08912]

@@andywatts can you explain models to me please?

[@NisseOhlsen]

Llama. Large Language model.

[@Harris08912]

@@NisseOhlsen ahhh I'm with you, thank you

[@bengsynthmusic]

LLaMA is an AI model made by Meta. It is a competitor to GPT or PaLM. These models are the backbone of an AI and give the AI unique properties.

[@goldstandardpersonalgrowth]

But can it run crysis

[@johannesdolch]

738 is "almost 1000". That's some hillbilly rounding right there.

[@ggzzbb]

15k for a computer, you could just store it somewhere in your house, your server room/laundry room/garage/etc. and just use it through terminals on your other computers and devices.

[@Fonzleberry]

I'm wondering how would a $12k mac pro with 192GB of unified memory match up? Able to run the same stufff just at a much slower rate?

[@__J____ff]

unified architectures have an extremely low memory bandwidth (per second) compared to dedicates GPUs, ok for training but not good for inference

[@xlagunaa]

​@@__J____ffIts the contrary

[@__J____ff]

@@xlagunaa you're right, I was stoned off my tits when I wrote that, it's supposed to be exactly the opposite of what I said

[@Venga7]

If I had the money I would invest $1,000,000 into building my own personal LLM

[@honor9lite1337]

Clearly it's not enough.

[@iGhostr]

Nicht genug

[@aquireeverything9382]

Why? Stanford alpaca cost a few hundred dollars to train.

[@philtrem]

I wouldn't. I'd just wait 2 years and get something a thousand time better for free.

[@e.c-machiavelli9659]

Remember GPT 3.5 actually have 175B Paramaters so nearly 3x more than open source also they are very struggled for data to getting this point. Maybe setting up lama and then training with GPT4 for spesific topic as you want achieve that thing.(Of course GPT4 data for training will cost like 8 10k$)

[@lets-evolve]

Everything is moving to cloud, many interested in this might already have the computing power and setup. I'm guessing the market for these is likely small, though there is a market for them. AI will also likely improve enough to make these not needed in a few years perhaps. Maybe other technologies might open a market for them though.

[@NisseOhlsen]

is that so ? To my knowledge, compute is much cheaper per flop if you do your training locally ? It's in the order of 3 months before you have recouped the investment in, say, a RTX 4090 based local system ?

[@AlgebraicContinuation]

@@NisseOhlsen Correct, this guy above has no idea what he's talking about. Deep learning is going to require increasingly more compute over time too, "AI will likely improve to make these not needed" is an absurd statement.

[@sean_haz]

I've always hated working with cloud servers. It gets messy.

[@lets-evolve]

I'll stick to my opinion and we'll see who's right in a year or two 😉

[@Jake_Hamlin]

Wait till 2030 when anyone who isnt a left wing looney wont be serviced by the cloud. I hope George looks into utilising Mythic AMP tech

[@nathanpotter1334]

Just for the future, Thanks George.

[@bugmanuk]

This man has zero idea. Like he has a few, but not any of them in a row. Love this podcast

[@biesman5]

Wrong

[@liminal27]

TinyBox is the Altair 8800

[@DeanZylman]

Really interested in buying a tiny box sounds awesome

[@woolfel]

at 15K for tinybox, it's still more expensive than Mac Studio M2Ultra.

[@anyadatzaklatszjutub]

M2 Ultra has ~50 TFlops of compute, tinybox would have ~1000 TFlops, so about 20X more for 3X the price, not bad

[@iGhostr]

What’s your point?

[@AI-xi4jk]

Maybe Apple fanboy

[@aidilmubarock5394]

Not even the same product

[@janklaas6885]

📍4:41

[@NathanHaney-gj3gl]

Uh…. my electric company says AI is able to see every appliance and identify it on my bill. They know it’s not a fudging car charger sir 🤣

[@FractalPrism.]

on/off stats alone can do that, no a.i. needed

[@vaishnoo1168]

This guy is a confident version of Robert from silicon Valley

[@aga5109]

This guy seems funny😅 and full of himself, which is too funny.

[@Unpluggedx89]

Dude I'm an engineer in one of the MAANG companies and I will tell you there are people like him everywhere and it at first is really funny but it gets soooo old very fast and it's so annoying.

[@Jake_Hamlin]

Hes kinda got a reason to be so full of himself...

[@othername2428]

​@@Jake_Hamlinwhy?

[@othername2428]

oh u mean cuz he's gay?

[@lorisvds]

@@othername2428 😅🤣 what a flat joke

[@xFlRSTx]

ubuntu with kde is best

[@jasonbuksh2958]

Why couldn't you just rent that in the cloud ?

[@jasonbuksh2958]

Even if they do the hardaware, the software OS needs a lot of work and will need to be constantly updated

[@htownzx21]

This boy geekin' lol.

[@AI-xi4jk]

Just for context single Nvidia A100 PCIE card was about as much as this whole machine.

[@2mj0lk]

it also costs 20-30k for one gpu, h100 is north of 30k

[@TheChidmas]

He’s making server case what is special about this?

[@bengsynthmusic]

0:48

[@TheChidmas]

@@bengsynthmusic that isn't special anyone can spec that it's just a matter of using a motherboard which has enough pcie lanes and enough pcie slots. Supermicro having a number of them but there's others. Either an epyc or Xeon CPU with the necessary pcie lanes and then a bunch of GPUs with the graphics memory plus ram and some ssd. Not one thing he's mentioned is special it's common in data centres. Give me the money and I could definitely build you one. Even the sound issue that can be solved unless he's building a custom solution it's all off the shelf parts including PSU.

[@jbinmd]

Smarmy. Weird. Self-absorbed. But has a certain clarity that comes from being unconstrained by norms.

[@mwaikul]

To quote Charlie Munger he would rather “hire a person with an IQ of 130 but who thinks it's 120, as opposed to someone who is 150 and thinks he has IQ of 170". He made a lot of claims in his first interview with Lex of how Comma was going to dominate Self Driving. With that at least would have expected comma to be a significant player by now. He is smart but all over the place. If he wants to succeed in business he needs a smart business manager to whom he will listen. I wish him well he is young and can course correct. He will do well if drops the Tech bro ethos.

[@sgares6741]

@@mwaikul hes funnier/more interesting like this. But usually gets better as you age though.

[@dixztube]

They gonna cancel him for saying that 😂 ugh

[@symtexxd]

Ubuntu mate is also my favorite. If I met Lex we would become friends for sure.

[@joseperez-ig5yu]

You have to lower the price of the Tiny Box in order to get more customers to try it.

[@AI-xi4jk]

I’m flabbergasted by how many people who watch this and comment, as if they know better, have no idea of how things are in this space. All that knowledge Hotz is dropping goes beyond their heads. Please educate yourself people or just shut up and listen. Maybe you’ll learn something. There is obviously a huge gap between the audience and high level of content but Lex please don’t dumb it down.

[@TheChidmas]

You're clearly not knowledgeable about hardware because he's talking about off-the-shelf parts. Its more of a sign of people's lack of knowledge if you think he's dropping knoweldge

[@Nolimitss6]

This guy doesn't feel like a real human being

[@Nolimitss6]

@@sadface7457 I've no idea who George Hotz is to be honest. But yeah, odd character

[@mrtienphysics666]

this guy is a genius

[@shrimpflea]

This guy is forgetting that men look for men too. Huge market.

[@alexanderevans9262]

Somebody tell this guy no for once

[@PsychiCorey]

Wtf I'm glad theres people tending that lane of information

[@proximo08]

I bet tiny box will be outdated as soon as next year when Apple announces M3 Max on Mac Studio

[@AI-xi4jk]

Apple hasn’t been doing great on large GPUs for ML. I doubt they can beat Nvidia or AMD just like that with their general consumer thingy.

[@aidilmubarock5394]

Not the same product

[@ericperez2067]

7900xtx x6 is 6k this guy is a grifter

[@LynneHand]

😂😂😂 $15,000 😂😂😂

[@ChrisAthanas]

Status is easy to fake for a little while

[@aga5109]

Looks too 😂

[@ChaoticSatire]

What a human meme

[@sirruffalot]

I have CSGO skins worth more than that.

[@zengrz]

could've bought m2 ultra with better specs and have money to spare

[@Jirizo1]

but then you'd be overpaying for an apple product

[@bengsynthmusic]

His computer is 1 petaflop while M2 is only 3.6 teraflop (0.0036 petaflop). And has 6 GPUs.

[@zengrz]

It is not like he made his own chip from silicon. Training modern LLM is more about having ram than speed. M2 architecture combines ram for cpu and gpu, so it has more ram and can do things his gig cannot do. Apple products are overpriced in general, but in this case it has better value fs.

[@theindubitable]

Grok

[@leoarzeno]

❤

[@kuboteusz]

This guy is a little bit out of touch

[@NisseOhlsen]

.. with Political Correctness, you mean ? on the other hand, he knows what he's talking about. Are you building an AI setup ?

[@kuboteusz]

@@NisseOhlsen with emotional intelligence

[@NisseOhlsen]

@@kuboteusz you mean lying ?

[@kuboteusz]

@@NisseOhlsen what? No, I mean social skills. He doesn't strike me like a guy you can have a casual conversation with. Lex is very good at adjusting himself to a person he's talking to and here he looks like he's struggling to get in touch with the guy

[@TalkingDonkeyz]

@@kuboteuszsome people don’t enjoy small talk

[@LuthandoMaqondo]

Ubuntu is phonetic, It’s pronounced something like: uuh -bu -ntu

[@paulm3969]

I reakon both of these dudes would be better off going outside, maybe getting a girlfriend or something.

[@Max-zr7hr]

No

[@behnam2778]

Is he gay?

[@hi-gf5yl]

Who?

[@NisseOhlsen]

who cares ?

[@behnam2778]

Just curious. George.

[@hi-gf5yl]

@@behnam2778 curious george? The monkey?

[@hi-gf5yl]

@@sadface7457 source?

[@Wiz-xm7mv]

It's interesting how someone who's so verbally exact with descriptions of his tiny box's capabilities, then calls the electricity it uses "power"

[@lazyraccoon1526]

en.wikipedia.org/wiki/Power_(physics)

[@subashchandra9557]

Did you never learn in highschool that Watts are a unit of power?

[@dooflydetailguuy4349]

I'm confused about why the word power offends you bud

[@superagnitio]

You aren’t the brightest bud

[@valleruh]

#linustechtips please Co develop the Tiny Box in the ltt Lab!!

[@MrChaosi]

yeeaa--- i am gonna call BS on this guy

[@LuthandoMaqondo]

Ubuntu is phonetic, It’s pronounced something like: uuh -bu -ntu