Where $17 billion has gone in AI

[@TechTechPotato]

Slides: drive.google.com/file/d/14y5WFvUHtkjI815MhzqyMQCYctSqv3w8/view?usp=sharing If you're very familiar with the current types of hardware in the market from my previous presentations, skip to 22:33 or 28:50 This was a very abridged presentation that could have gone into extensive detail on each part. If that's something that interests you, let me know!

[@jannegrey]

This was honestly great. The part that I thought was abridged "too much" (I didn't understand it beyond concept) was Analog Chips. I understood limitations of conversion too Digital, but in a way I didn't get how they work - except that they have theoretically "infinite" precision (not counting the fact that Quantum Mechanics doesn't allow for infinite). Saying that they work "because Physics" is a bit too short for me to infer how do they work ;) And it would also be nice if they could make transformer models (unlike Mythic AI which decided not to). Also - I did hear that Blackwell GPU has a tiny RISC-V CPU in it. High Yield made a video about GB202 (? 5090), but while photo was okay, we would need higher resolution to spot it. Interesting to see - given that NVIDIA works closely with ARM, AMD uses ARM in Ryzen chips (or at least used promontory chip which used ARM), but NVIDIA GPU has RISC-V...

[@jonford1988]

Thanks for the educative video…. As new investor, it's important to remember that investing and trading require more than just technical analysis skills. Discipline and emotional maturity play a significant role in achieving success. It's wise to keep in mind the adage of "time in the market vs. timing the market," as this mentality can help you weather market volatility. With insights of Kerrie Farrell and my commitment to learning and growth, I've been increasing my earnings in just a few months. Keep up the good work...

[@jonford1988]

Kerrie Farrell program is widely available online..

[@jhonatanascimento364]

As a beginner, it's essential for you to have a mentor to keep you accountable.

[@teriusofficial6405]

I appreciate the professionalism and dedication of the team behind Kerrie’s trade signal service..

[@LeightonCorrigan]

Investing with an expert is the best strategy for beginners and busy investors, as most failures and losses in investment usually happen when you invest without proper guidance. I'm speaking from experience.

[@CatherineOliver-c7d]

Great skills and knowledge about the market. I enjoy full profits and easy withdrawal with no complains,.

[@mukkeshmckenzie7386]

Ian, idk if you'll read this comment, but your work is amazing and l hope you continue to keep doing this.

[@Zeee530]

Same

[@SixFootShort]

Just the fact you are given access to the powerpoint is massive. Thank you.

[@davidgunther8428]

Nice overview, especially the intro connecting older ideas and where they went or didn't go.

[@wolpumba4099]

*Dissecting the AI Silicon Landscape: Key Trends and Players* * *0:00:14** Introduction:* The speaker highlights key trends in the AI silicon market, covering major public companies and startups. * *0:00:21** Definitions and Market Overview:* The presentation starts with definitions of AI, its functionality, and market structure. * *0:00:48** AI Hardware Landscape:* A review of current AI hardware, focusing on CPUs, GPUs, FPGAs, and ASICs, emphasizing the latest generations and their roles in AI. * *0:01:54** CPUs:* x86 dominates, with Intel leading in AI inference due to integrated machine learning instructions. The shift to chiplets for managing high-bandwidth data is noted. * *0:03:33** AMD:* Similar chiplet strategy, using TSMC for manufacturing, achieving 30% data center market share. * *0:04:41** Arm CPUs:* Growing presence, notably with Nvidia Grace for AI factories. * *0:05:53** GPUs:* Nvidia dominates 90% of the training revenue, with AMD favored by OpenAI for inference due to cost. * *0:07:43** Intel's AI Product Line:* Introduction of Intel GDI (Habana acquisition) and HPC chip Ponte Vecchio. * *0:08:28** FPGAs:* Niche but dedicated role in AI, with AMD (Xilinx) and Intel (Altera) integrating AI engines into their products. * *0:10:26** ASICs:* Highlight on smart NICs and specialized chips like DE Shaw's Anton for molecular dynamics. * *0:12:55** Non-Von Neumann Architectures:* Discussion on analog, neuromorphic, quantum, and optical computing as emerging technologies. * *0:13:13** Analog Computing:* Offers low power and latency but faces manufacturing and scaling issues. * *0:14:36** Neuromorphic Computing:* Mimics brain function with spiking neural networks, with Intel's Loihi 2 as a notable example. * *0:16:30** Quantum Computing:* Still in development, but showing promise in physics, chemistry, biology, and specific mathematical problems. * *0:18:26** Optical Computing:* Potentially faster and power-efficient, but currently challenged by scalability and manufacturing issues. * *0:19:40** Reduced Precision:* Explanation of reduced precision (FP16, FP8) in AI computations, emphasizing power efficiency and performance gains. * *0:22:44** AI Silicon Market Segmentation:* The market is divided into four battlegrounds: data center training, data center inference, localized inference, and edge inference. * *0:23:05** Data Center Training:* Large clusters for foundation model building. * *0:23:57** Data Center Inference:* Large-scale deployment for real-world applications. * *0:25:15** Localized Inference:* Edge servers for on-premise AI processing. * *0:26:08** Edge Inference:* Small, specialized chips for devices like smartphones and IoT. * *0:26:55** Companies and Chips:* Overview of key players in each segment, including startups and their funding. * *0:28:55** Startup Funding and Market Trends:* Analysis of startup funding, mergers, acquisitions, and potential IPOs. * *0:32:37** Hot Companies:* * *0:32:43** Tenstorrent:* AI and RISC-V IP, chiplets, and servers, with significant funding and partnerships. * *0:35:57** Groq:* Focus on language processing with a unique chip, but facing challenges with memory and power consumption. * *0:37:11** Cerebras:* Wafer-scale chip for AI, offering high performance and unique capabilities. * *0:39:06** Failures:* * *0:39:22** Graphcore:* Initially successful but struggled with the shift to transformer-based models. * *0:39:58** Mythic AI:* Analog computing startup that faced manufacturing and scaling issues. * *0:42:09** Key Industry Trends:* * *0:42:09** Advanced Packaging:* Stacking chips to overcome physical limitations and improve performance. * *0:45:10** Connectivity:* Importance of high-speed interconnects within and between servers. * *0:46:32** Co-Packaged Optics:* Emerging technology for faster and more efficient chip-to-chip communication. * *0:54:08** Conclusion:* The AI silicon market is rapidly evolving, with significant investments in new architectures and technologies. Optics and advanced packaging are poised to play crucial roles in the future of AI hardware. * *0:54:11* Ian's plug for his KZbin channel. I used gemini-1.5-pro-exp-0827 on rocketrecap dot com to summarize the transcript. Cost (if I didn't use the free tier): $0.09 Input tokens: 66461 Output tokens: 1125

[@pauljones9150]

Thx

[@Teste-gp7bm]

This is an excellent presentation. Thank you.

[@ArunRamakrishnan]

Ian wonderful as usual. Started my HPC career with your articles and still continuing to enjoy your balanced focus on all these minute aspects of systems engineering.

[@opius1199]

Super good presentation Ian, thx!

[@TMS-EE]

Perfect timing to give a "State of Play" report. The electronics that enables the advances in AI has been overlooked by those who namedrop TSMC and ASML but have no idea what it takes to move technology forward. I respect that you look inside imec and attend Nvidia GTC. I value your insights from the events where the top engineers in chip design or HPC gather. I found Dylan's 5 hour interview with Lex a good compliment to your channel. It was heartening to hear Lex express how impressed he was with the breath of knowledge Dylan (and you) have of the entire semi and AI supply chain and ecosystem. The money involved is incredible and yet it is still risky. Keep up the great work sharing knowledge with techies who are going to make the next few years the most unpredictable ever.

[@TechTechPotato]

I watched the Lex/Dylan interview and agreed, it was very good overall. I thought the third guy did a great job for the first 20 minutes explaining the environment.

[@yoonsikp]

Can't wait for optical to come

[@briancase6180]

Dude, Google TPU is used for data center training. I think Google (and others such as Apple) will be surprised to hear they aren't playing in that segment. Also, you left out Amazon tranium and inferentia. I think they and anthropic will be quite surprised to hear they aren't playing in both segments. Etc.

[@solidreactor]

I kinda want to see Cerebras do chiplet packaging with their wafer sized chips, mainly for the reason "just because".

[@4989656]

Thank you, Appreciate your work. I would love if you can do more interviews with Engineers from all these chip companies, I find those extremely interesting.

[@accursedshrek]

Thanks!

[@floroos]

It would be interesting to see your take on Alphawave Semi. Not exactly a startup anymore, but with very interesting developments in high-speed optical interconnects and chiplet-based architectures.

[@egalanos]

That NVidia slide showing the direction with stacked memory would be insane capacity if they intend to use HBM4: 24 chips would allow for a single GPU to have 1.5 TB RAM 🤯

[@jh4684]

Great work!

[@francescorossi7582]

Slide 27: Google TPU is also used for training

[@tringuyen7519]

Actually, no. Google buys NVDA & AMD GPUs for training. Once training is completed, Google moves AI over to TPUs for inferencing.

[@SubclavianStandards]

One disadvantage of optics that really needs work is the reliability. Optical connections have 10-100x worse MTBF and can suffer from flapping too. All of this can be hand-waved as okay until you are trying to run training jobs on huge clusters and your blast radius is large. All the frontier labs and hyperscalers have complained about reliability, and this is a big contributor. If your optics are now packaged with your expensive chips, do you have to replace your beefy GPUs instead of the pluggable?

[@gabrielpi314]

44:01 How much of TSMCs capacity crunch comes from ASML's capacity to deliver EUV machines vs other factors (factory space, workforce, raw materials, etc)?

[@SaltyKimchi]

At 21:15 the name is actually a bit misleading Tensorfloat-32(TF32) is actually 18bit FP format and not 32bit.

[@couldntfindafreename]

When can we get affordable unmanaged 10Gbps Ethernet switches for small office and home use? We've been stuck at 1Gbps for 10+ years by now, while the industry as a whole is going from 400Gbps to 1.6Tbps soon.

[@Bguha1]

Brilliant talk. What about in-memory computing ? Has that trend subsided ?

[@TechTechPotato]

Lots of companies still doing it - the question is if you're talking about compute-in-memory or memory-in-compute. Everyone has the same argument but a different idea

[@neiljamessloan]

Tell us about the ‘lot of companies and the lot of startups’ you’ve worked with. Pls.

[@TechTechPotato]

They're all in the description of every video I publish. Current long-term contracts are AMD, Intel, IBM, Qualcomm, Tenstorrent, NextSilicon, SiTime, Synopsys, Ayar Labs. I have project work with a half-dozen others.

[@ethanwelner1230]

Fix every bridge and road in the country OR boil the oceans so that two marketing bots pretending to be humans can spam eachother on a dead social media website while recording fake click-throughs. Our society is so cooked.

[@dhavalsharma4656]

Interesting no mention of Rivos!

[@TechTechPotato]

I would mention them, IF THEY EVER SPOKE TO ME. Rivos has been silent for 24-30 months really. If you know someone there, please get them to reach out.

[@Jensenr8]

A couple of years ago, I heard something about analog computing for AI research. Is that something that you have seen or know about how it's coming along

[@jacksonmatysik8007]

Can I get the source for OAI preferring AMD for inference?

[@registeredblindgamer4350]

$2 a second for quantum computing time in the cloud is insane. $7,200 an hour. I'm in the wrong business aha. . .

[@siddharthkolte3659]

Why does Wave Computing have the dead symbol after them?

[@TechTechPotato]

Wave as a business died - went through chapter 11, and rebranded as MIPS (one of its owned brand IPs) making RISC-V cores.

[@lukevassallo3808]

Whats the source of information on slide 23 (time ~ 21:50) ?

[@TechTechPotato]

ISSCC 2023 presentation

[@paul.1337]

When will my terminators be ready? I need my robot legs.

[@BellJH]

Ian, do you know what the issue is with Intel Foveros Direct 3D and Clearwater Forest? I’m told that’s why it’s delayed to H1 2026

[@JGaffney9000]

The public face is big conglomerate spending billions on training bigger LLM's models based on data they should have paid for. The end result is some better chatbots, a coding assistant and an evolution of digital assistants used to help sell smartphones/competition for google or wikipedia. It's boring and a lot of money is going down the drain because I don't see any LLM product that's going to generate or save >$10bn per year in revenue. The big revenue source right now is selling licences for other companies to use your model to peddle the AI hype, or sell subscriptions to curious people. The interesting part is much of the purpose build silicon which could be used to make scientific research more accessible, quicker or better.

[@tringuyen7519]

Nope. Corporations are spend $ billion in AI in hopes of replacing as much of their headcount with AI as possible. Labor is the most expensive cost for companies!

[@jonnypanteloni]

Dear Ian! Wikichip is down?

[@TechTechPotato]

Yeah we noticed that a couple weeks ago as well. David Schor wasn't at the usual Dec conferences as far as I recall, and he's usually really difficult to get hold of. I'm hoping he comes to the February conference

[@ChrisJackson-js8rd]

what proportion of businesses in this space are actually generating operating revenue today? and/or have contracted to do so in the next 12 months?

[@tringuyen7519]

MSFT, ORCL, & AMZN are all making $ through subscription into their AI productivity helpers for corporations & government agencies.

[@ChrisJackson-js8rd]

@@tringuyen7519 yes and I think that is probably where the smart money goes. As interesting as the hw is. And even more fundamentally I would think that the real benefit of LLM's is to the software vendor. Cerebras and Tenstorrent are the two hw companies that seem likely to have long-term potential. I almost think moreso than the obvious picks like AMD and NVidia, though I could well be wrong about that lol

[@TechTechPotato]

Most are making some money. Blaize recently did a SPAC and said they had 11m in revenue. Only 48k was hardware, rest was design services.

[@ChrisJackson-js8rd]

@@TechTechPotato ahh ok yes design services does seem like a much better business model in this space. Though it still strikes me that investing in AWS, Oracle, and Microsoft has both less risk and at least to date better returns than investing venture capital directly in a start-up. With a few very notable exceptions.

[@rupertsmith6097]

Graphcore lost a large deal with Microsoft. The same Microsoft that was one of the early founders of the company...

[@The_Conspiracy_Analyst]

They should immerse the rackmounted machines in fluorinert or something and use that as a working fluid for a closed rankine cycle or sterling heat recovery engine. They always say it's "not economical" but that's just intellectual sloth and lack of initiative on the part of their flunky engineers. If they hired bright lads like me they'd be well on their way to this brave new future instead of sitting around moping and daydreaming.

[@TechTechPotato]

I've been seeing companies built around supplying immersed systems for years at conferences. Nobody wants them - aside from the upfront cost, the repair and maintenance is wildly inconvenient.

[@AstrogatorX]

Does anyone have an idea why AMD couldn't make a true chiplet GPU (multiple CGDs) using TSMCs CoWoS and Apple managed to do it using the same technology in the M series of chips?

[@TechTechPotato]

What would be the end-cost and who would buy it

[@tringuyen7519]

AMD could have made a RDNA4 chiplet GPU to take on the RTX 5090. Due to complexity, time, & business constraints, AMD chose to keep RDNA4 monolithic & aim for 5080 instead.

[@acasccseea4434]

The latency probably will break most compute instructions. It's already quite bad for chiplet CPU. Parallel compute data offloading is lumpy, making the bus width expensive to make, and latency still too high

[@AstrogatorX]

Probably all those people sleeping in tents outside the Microcenter so they can buy a 5090 for $2000+. The M Ultra which is made with CoWos technology is expensive, but probably no more expensive than the GB202. I asked because Sam Naffziger, a Radeon engineer in an interview with Steve from Gamer Nexus claimed that building a multi-chiplet gpu required 10000s of connections between them. Moments later, Apple boasted the first multi-chiplet GPU in the form of the M1 Ultra, which has 10000s of connections between chiplets, produced using CoWoS - the same technology AMD uses So what is going on? BTW keep up the good work!

[@PhilipWong55]

NVIDIA, AMD, Qualcomm, Broadcom, Apple, and Google do not own or operate any manufacturing facilities. It relies on a network of external partners in Asia to fabricate, assemble, and test its products. These tech companies do not have the ability or resources to build or operate any manufacturing facility anywhere in the world. Silicon Valley no longer has much to do with Silicon. These US companies do not manufacture silicon chips or physical products and are not in the "Semiconductor Chip Industry." They are chip design and marketing companies. When their products are imported into the US from Asia, they will be subject to the proposed blanket tariffs.

[@TechTechPotato]

If you watch this channel, we cover manufacturing extensively.

[@dieselphiend]

Doesn't anyone ever wonder who's buying all of the leading process nodes (beyond 5nm) at all of the major fabs? Intel 18a is already in limited production but only for defense contractors, and when it comes to leading process nodes, that is the norm. They always go to the defense sector. For what?!? What could they possibly need these nodes for? Trying to figure out who's producing what, and for what, is impossible for all leading nodes. It's highly secretive, and guarded information, and as far as I'm concerned, a crime against humanity. We should be pressuring Congress to pass laws to make all information pertaining to high end silicon production 100% public. Yeah, I know- the fact that these companies are able to pay out the nose for these leading processes benefits us all but I do not care. It's far too dangerous for the rest of us to allow private companies to access the world's leading silicon fabrication processes in secret. This will become very apparent in the coming years.

[@SupaSupaKewl]

Apple mostly and other high end smartphone chipmakers like Qualcomm. Only smartphone makers really need leading edge stuff due to power constraints, so theyd need and pay for leading edge at the highest prices. The M4 chips and A18 chips in new macs and iphones take up most of the 3nm manufacturing. Apple also probably has the highest volume of orders for leading edge nodes, so that why they usually get first dibs.

[@dieselphiend]

@@SupaSupaKewl Ignore Apple, that's mostly at TSMC, which I might add is already making 2nm but for who, and for what? There's many different fabs working with EUV, and in limited production.

[@Cybot_2419]

A lot of the HBM generations on slides 8 and 9 seem to be wrong. A100 and MI250X use 2e not 3, MI300 uses 3 not 3e

[@sky0kast0]

I really wish I wasn't the future technology but then again AI could grow to be a friend same time AI grows into an enemy

[@joehopfield]

Most of those "uses" sound completely dystopian. Nobody is asking for more surveillance in retail, nor from appliances or watches. Maybe Leon's coup de etat while he desperately pours someone else's money into his fascist LLMs is coloring my view?

[@patrickcarpenter6258]

It went in to the researchers pockets.

[@motionthings]

"LLM's are nothing but autocomplete on steroids" - Linus Thorvalds

[@anasevi9456]

True, but Linus Thorvalds has had his brain long scrambled by alcohol, social media; and now more recently being on the losing side of a geopolitical dickwagging. Him brushing something off as a nothingburger these days is like a geriatric cockroach blowing off an approaching steamroller.

[@kelownatechkid]

Commenting for the algorithm, but based on the title I'd say it 'went in the toilet' based on what I can see in real life lol. None of this investment has improved anyone's life so far, it has only made things really bad sadly :( I expected ML work to go into things like helping to cure disease, but all the investment is for chatbots that are mostly used to generate spam.

[@TechTechPotato]

IBM has done extensive work in drug design using ML.

[@Radium3D]

Less AI chips please, we need the silicon to go to GPUs.

[@tringuyen7519]

Essentially, yes! Co-Package Optics is being pushed by GPU companies bc it’s needed to reach ASI.

[@patrickcarpenter6258]

AI is nothing more than what the cloud was a few years ago. It's just a term and tech for VC's.

[@rogerc7960]

Smuggling Nvidia High-End Chips; Deepseek Core Member Worked at Nvidia kzbin.info/www/bejne/f4PYkp2ohpilbMU

[@tringuyen7519]

It’s hilarious that the Chinese engineer is flaunting that he can acquire an H200 easily. I never supported the GPU ban on China. Depriving Chinese gamers of 4090 & 5090 is silly!