Microsoft CTO Kevin Scott on How Far Scaling Laws Will Extend | Training Data

The current LLM era is the result of scaling the size of models in successive waves (and the compute to train them). It is also the result of better-than-Moore’s-Law price vs performance ratios in each new generation of Nvidia GPUs. The largest platform companies are continuing to invest in scaling as the prime driver of AI innovation.
Are they right, or will marginal returns level off soon, leaving hyperscalers with too much hardware and too few customer use cases? To find out, we talk to Microsoft CTO Kevin Scott who has led their AI strategy for the past seven years. Scott describes himself as a “short-term pessimist, long-term optimist” and he sees the scaling trend as durable for the industry and critical for the establishment of Microsoft’s AI platform.
Scott believes there will be a shift across the compute ecosystem from training to inference as the frontier models continue to improve, serving wider and more reliable use cases. He also discusses the coming business models for training data, and even what ad units might look like for autonomous agents.
Hosted by: Pat Grady and Bill Coughran, Sequoia Capital
00:00 - Introduction
01:20 - Kevin's backstory
06:56 - The role of PhDs in AI engineering
09:56 - Microsoft's AI strategy
12:40 - Highlights and lowlights
16:28 - Accelerating investments
18:38 - The OpenAI partnership
22:46 - Soon inference will dwarf training
27:56 - Will the demand/supply balance change?
30:51 - Business models for data
36:54 - The value function
39:58 - Copilots
44:47 - The 98/2 rule
49:34 - Solving zero-sum games
57:13 - Lightning round

[@shrek22]

Kevin is incredible in this interview. i appreciate him being so open. thank you!!!!

[@eirikgg]

Great interview. Happy to be a partner to Microsoft with this guy as CTO

[@ToonamiAftermath]

49:17 Gary Marcus is fuming

[@squamish4244]

Gary Marcus is always fuming. It's like Gary Marcus has a Gary Marcus agenda to push as a Gary Marcus cognitive scientist. If Gary Marcus could shut the *fck up for one week about how much AI sucks, I would be impressed - but Gary Marcus is incapable of doing that. Gary Marcus! Gary. Marcus.

[@SiyaMaliChannel]

29:15 the frontier model analogy versus inference as an equivalent of building out your own phone and OS to build an app.😂

[@jabowery]

Speaking of Solomonoff, it would be really interesting to find out the details of why Microsoft didn't underwrite the Hutter Prize back in 2006.

[@ShowRisk]

this feels such a genuine interview

[@tahir2443]

so incredibly based

[@ianglenn2821]

this is very good but watch at 2.5x speed

[@wonseoklee80]

This is the most honest explanation for whole AI situation at the moment: better frontier models are coming. They might not be AGI, but still can have a huge impact on the industry - cheaper, better reasoning, less hallucination and so on. Anyone who architects the product assuming the frontier model will be stuck at today's capabilities, won't be in a good situation.

[@squamish4244]

People went from being blown away at GPT-4 to assuming AI will be stuck at it in the space of literally one year. Even Claude 3.5, which is better, didn't do much to change that perception, because it wasn't astoundingly better. It is the strangest and least justified assumption to make about a rapidly advancing technology in living memory. It's how we are wired as humans, that we are almost impossible to impress, but even so, rationally, it is nonsensical. I don't care if it takes one, two or even three more years for the next major leap in frontier models - if GPT-5 is to GPT-4 what GPT-4 was to GPT-3, it will be a staggering improvement, and we will be entering territory that if it isn't AGI, will be so capable as to render the term meaningless. AGI is such a vague term with so many definitions now that the term is already functionally meaningless - yeah yeah, AGI or not, what can this model DO? AlphaFold 2 has been cited over 20,000 times in papers since it became open-source exactly three years ago, turning medicine on its head. But human trials by their very nature take time. And now AlphaFold 3, which is substantially simpler and better, is out, and who knows what it will do, along with all the other AI models being integrated into medicine right now. But because we don't already have the cure for cancer or whatever, and these are medical models and not sexy chatbots, nobody notices outside of those of us who take a particular interest in such things.

[@GNARGNARHEAD]

slow start but that really got going 😎

[@BrayanVZ-of8zl]

This channel is a go-to for information. It's the best!

[@bola986]

1.5x

[@ParnianMotamedi]

is the sequoi digital currency? it listed in pancake swap? Please answer me soon i'm in hurry

[@shinkurt]

Great interview

[@squamish4244]

This tells me GPT-4 was the top of the last sigmoid curve, and it will take about a year from now to scale the next generation of models for release. And no, he is not some hack, he DOES know what he is talking about. Check out his Wiki bio.

[@nuclearenergy919]

A year is also very fast if you take into account the change in capabilities of gpt 3 vs gpt 4, gpt 5 would be crazy. And If that be the case I will start believing in this theory of scaling laws

[@squamish4244]

@@nuclearenergy919 Indeed. If it be the case. But really, twice as long as that would still be insanely fast too, again considering the jump between 3 and 4. Even if scaling stopped dead tomorrow, what we already have is enough to massively change society as it is integrated into all kinds of systems over the next decade, and we know that the models are just going to keep getting better in that timeframe anyway, whether it be due to scaling or other methods. There's lots of gas in the tank, scaling is just one tool. Interview was grounded and hype-free. Dude is based af, as others here have said.

[@ishaankoratkar9242]

holy based

[@profit_sales_arch]

Is there a Spotify podcast channel of the same?

[@sequoiacapital]

Training Data is available wherever you listen to podcasts!

[@420_gunna]

35:50 dark

[@JazevoAudiosurf]

just let this man talk, nothing else to do

[@catshaman7098]

Not Scott gu?

[@Tubernameu123]

co piolt is too hyper filtered. microsoft is a corporation that does not care about out privacy but hes a really neat guy. Lets not forget with their openai partnership and all the sketch behind openai since day one..... then the nsa on the board thats scary. still hes a neat guy.

[@chriswiles87]

dumb

[@Ikbeneengeit]

FYI your wall is broken

[@zoner__]

He could retire now and pursue life 2 but if he's concerned about impact stay. My son works there and feels like a cog at times. He feels like he wants to fix things right but of course there is always pressure to just get it done.

[@captain_crunk]

I was at LinkedIn during the msft acquisition, and stayed another 5 years after that. Imho Microsoft is amongst the very best tech companies to work for. Feeling like a cog is obviously par for the course in capitalism across the board, but in my experience it's less prevalent at Microsoft than other places. YMMV, but I prefer being a cog to being a nail, which is fairly common at many startups :)

[@umuti5ik]

Getting it done and in the hands of customers quickly is the way to do it right. :)

[@AIForHumansShow]

every time we hear Kevin Scott get interviewed the more we like him

[@Copa20777]

I remember him from the whale presentation with Sam Altman.. 🎉🎉🎉 4rmZambia 🇿🇲 and thanks Microsoft🎉🎉🎉

[@dadsonworldwide3238]

The problem I have above all else is rooted even before 80 years into the transitor age. The whole nation is banking on this, and most arr uneducated to how to triangulate free will inertia in a frame of reference. It has dangerous loopholes to fear for future generations and how the world beliefs won't fit or worse. It puts so much on this means of production beast of burden textualism methodology objectivism = shining capital on the hill metamorphosis detoured by 1900s structuralism just to end back up where our classheritage was. 100s of millions of Americans know more than everything physicalism ease of access that everything doesn't start in greece. In many ways the puritan pilgrim movement is just now picking backup fighting for 3 lines of measure pragmatic common sense objectivism but it's surrounded by and through very messed up irrational informational models and maths to use .

[@LevelofClarity]

Can you elaborate?

[@dadsonworldwide3238]

@@LevelofClarityeveryone is equally broke and America has closed its private sector individual liberal power of Entrepreneurship until the more recent decade. in short 1945s Smith_mundt act, everything physicalism deterministic simplicity while pushing infinite sums of complexity upon all we held dear . Flipped our 3 line of measure = truest True known standard into a 3 body problem ,censorship of all things pragmatic common sense objectivism proper.. We granted liberal power to the state & higher education in a call to arms that has since been followed by one excuse after another, rebuild Europe, tranfer wealth to and for liberating all common sense marginalized groups leaving only criminals or extremists. Industrialized all new nations now crying that it's killing the planet after we successfully get eveyone on their feet. Lol

[@dadsonworldwide3238]

It Is a more nostalgic ancestral one as well,picking up where the puritan & pilgrim left off. Since we sold the soul of the nation, after getting so rudly interrupted, it is a more ancestral quest for eqaulibrium, Technically the shining capital on the hill ( metaphor) is metamorphosis In reverse textualism methodology objectivism = technological advancements in concert with free flow of information education.

[@dadsonworldwide3238]

@LevelofClarity we've not only held back and censored our psychology we've been disciplining and attacking our mind thru sanctions, fines,suspensions,monopolization, chasing material dope or physically carving up our bodu tissue to try and fix problems in our lives. We really need this streamlining to help free parents up and allow us to better incentives the family cells pay them back for all that been charged upon passing down countless generations of burden

[@brian-jv1nw]

It’s too monotone for my ears 😞

[@JumpDiffusion]

Old people talking slowly…

[@squamish4244]

@@JumpDiffusion Dude is 52. He's only "old" to a teenager, for whom everyone over 30 is old.

[@gasfeesofficial3557]

funny, this guy doesn't seem to know anything lol or maybe it's just a poker face... can't tell.

[@squamish4244]

You don't become Microsoft Chief Technical Officer by knowing nothing.

[@couldntfindafreename]

44:00 Devin mentioned. Stopped watching. No sub.

[@KAustinHopper]

This guy's voice does not match his face.

[@GggatsGggss]

From $37K to $45K that's the minimum range of profit return every week I thinks it's not a bad one for me, now I have enough to pay bills and take care of my family.

[@VssgGsggsd]

Excuse me for real?,how is that possible I have struggling financially, how was that possible?

[@GggatsGggss]

Thanks to Elizabeth Marie Hawley.

[@GggatsGggss]

Though I started with as low as $15,000 actually because it was my first time and it was successful, She's is a great personality in the states

[@Shanefrancis-vk8ks]

YES!!! That's exactly her name (Elizabeth Maria Hawley) so many people have recommended highly about her and am just starting with her 😊 from Brisbane Australia🇦🇺

[@FavourMamahdhe]

I have heard a lot of wonderful things about Elizabeth Maria Hawley on the news but didn't believe it until now. I'm definitely trying her out

[@MrEmbrance]

dude doesn't know anything ai, he has to pretend because he's cto

[@catshaman7098]

Yeah, a couple minutes in I realized he’s just spitting words. Can’t complain when Microsoft axed their own AI team for OAI lol

[@alessiograncini9564]

this feels such a genuine interview