The State of Silicon and the GPU Poors - with Dylan Patel of SemiAnalysis

If Charles Dickens was alive in 2024, A Tale of Two Cities might be the divide between the “GPU poor” and the “GPU rich”.
We mentioned these terms in some of our previous episodes; they were originally coined by Dylan Patel of SemiAnalysis in his “Gemini Eats the World” post, put on blast by Sam Altman. SemiAnalysis are one of the most in depth research and consulting firms in the semis world, and have a unique insight into the design, production, and supply chain of GPUs based on their ground presence in Asia.
In this episode we break down the State of Silicon: when are more GPUs coming? Are there real GPU alternatives on the way? Should Microsoft buy AMD chips just to scare Jensen? Is there a “GPU poor is beautiful” manifesto?
Full show notes: www.latent.space/p/semianalysis
0:00 - Introductions
4:31 - Importance of infrastructure and hardware for AI progress
10:53 - GPU-rich vs GPU-poor companies and competing in AI
14:22 - Optimizing hardware and software for AI workloads
17:00 - Metrics for model training vs inference optimization
21:38 - Networking challenges for distributed AI training
23:19 - Google’s partnership with Broadcom for TPU networking
28:04 - What GPU-poor companies/researchers should focus on
34:47 - Innovation in AI beyond just model scale
38:03 - AI hardware startups and challenges they face
46:15 - Manufacturing constraints for cutting edge hardware
50:36 - Apple and AI
54:18 - AI safety considerations with scaling AI capabilities
57:37 - Complexity of rebuilding semiconductor supply chain
1:00:08 - Recommended readings to understand this space
1:04:31 - Dylan’s process for writing viral blog posts
1:07:27 - Dylan’s “magic genie” question

[@videowatching9576]

A real joy to hear Dylan Patel talk through this! Appreciate his passion and insights!

[@videowatching9576]

Awesome overlay at the top of video to indicate what topic you’re on.

[@videowatching9576]

49:52 sequence including anticipating OpenAI after GPT-5 then maybe OpenAI goes for $100 B raise at $500 B valuation to build $100 B supercomputer, as the path that we’re on, is really fascinating. So interesting about really internalizing that notion of the path that we are on - including that notion of scaling laws and what is implied about how much bigger and more powerful of a computer is going to be that you would need. Great example of how listening to an interview is vastly better than just reading a transcript. Dylan has a unique energy. Wonder if that includes the notion that maybe OpenAI will really need to physically control their own infrastructure, rather than simply pay money that ends up driving the growth of Microsoft’s own infrastructure.

[@LatentSpaceTV]

wow true to your name you really do watch the video haha. thank you! yes openai is rumored to be building their own datacenters. we've talked about them becoming the 4th major big cloud, the AI cloud

[@skamalad]

What’s the mention at 35:05. It also goes silent

[@videowatching9576]

Look up the author’s papers: Transformers paper “Attention is All You Need” and more. Profound work and contributions to AI.