"Compute is the New Oil", Leaving Google, Founding Groq, Agents, Bias/Control (Jonathan Ross)

Jonathan Ross, the inventor of the TPU at Google and founder/CEO of Groq, the company behind the LPU and the most insane inference speeds on the market, joins me for an in-depth interview on topics including the founding story, where the value will be created in AI, bias in models, and so much more.
Join My Newsletter for Regular AI Updates 👇🏼
www.matthewberman.com
Need AI Consulting? ✅
forwardfuture.ai/
My Links 🔗
👉🏻 Subscribe: / @matthew_berman
👉🏻 Twitter: / matthewberman
👉🏻 Discord: / discord
👉🏻 Patreon: / matthewberman
Rent a GPU (MassedCompute) 🚀
bit.ly/matthew-berman-youtube
USE CODE "MatthewBerman" for 50% discount
Media/Sponsorship Inquiries 📈
bit.ly/44TC45V
Links:
Previous Groq Team Interview: • LPUs, NVIDIA Competiti...
Chapters:
0:00 - Intro
0:38 - Founding Story
3:20 - Groq Chip Memory
6:24 - Chips vs. Cloud
9:28 - Future of AI
11:04 - Where is the Value in AI?
13:45 - Agents & Inference Speed
17:18 - Optimizing Models for Groq
19:32 - Fears and Hope for the Future
22:24 - Bias and Algo Control

[@yonatan09]

Amazing Guest! You're big time Matthew.

[@aanthanyj]

For real!... You have come a long way in the past year ... Isn't it fantastic when our passions become famous?... Thank you for sharing your knowledge, passions, and doing so in a way that I can understand... I really enjoy your videos and have gone back to watch the ones that I have not seen. It is interesting to watch the ones from a year ago and see how far you've come!

[@executivelifehacks6747]

Scoop! Well done Matt Berman! For your next trick, exlusive 1hr interview with Ilya Sutskever please. 🙏

[@gremlinsaregold8890]

Matt would need to also be James Bond to break Ilya out of the dungeon under the OpenAI building. Might also need to poke Ilya with a stick to get him talking....cos nobody has seen hide nor hair of Ilya since the whole Altman outer last year.

[@Copa20777]

We need Ilya out here for real

[@rodrigovianna470]

MORE OF THIS CONTENT PLEEEEASE!!!!!!

[@torarinvik4920]

+1 for this!!!!

[@DrW1ne]

i love the fact that you get to meet so many important people in this industry!

[@joe_limon]

Speed is like candy. It gives that instant gratification yes. But if given the choice between owning my data and hardware and simply waiting longer. I start to care a lot less about speed. Even with agentic systems. When we get to smarter more capable agentic systems, I wouldn't mind letting it run 24/7 to say handle the equivalent of a small company worth of intellectual power.

[@Alu404]

Agreed.

[@babbagebrassworks4278]

I am testing LLM, SD XL etc on a Raspberry Pi5, default SD takes 40-60minutes to make image, Onnxstream optimised version takes 3minutes. Code can be improved so GPU card is not needed. For home use who needs 60 images per minute? I can just run my Pi5 overnight and get 400 images. Once we get Agentic systems for local home use things are going to get much faster.

[@REASONvsRANDOM]

That’s too sustainable.

[@joe_limon]

@@wes555 All analogies can be torn apart if you start comparing aspects that weren't mentioned.

[@fxlltxtsearch]

The issue is that at scale it doesnt work like this. The compute compounds. Run your machines for 3 years. The speed aspect will compound the results you get whether it be training models or scraping websites. The speed actually matters, but also, you’re right the actual data is what companies will buy off of you.

[@jim7060]

For me this was probably one of the best videos you have put out. John put everything so simplistic that I can now understand a whole lot more. My fears have now weakened. Thank you Matt for contacting this man and being able to sit down with him in front of everyone and answer a lot of questions in a short amount of time. 💥

[@user-xj1bc8hk8o]

Awesome interview, thank you!

[@liberty-matrix]

This represents the greatest transformation of humanity ever. In 10 years, everyone's best friend will be an AI Agent. Will we talk to it, ask it questions and think to ourselves, - what did we ever do without AI?

[@jackflash6377]

I'll go a step further. In 10 years we'll have Neuralink connected to our phone which will be able to access any of the AI providers. Engineering problem? Just look at it and ask AI to solve it using your thoughts. You will have access to more IQ than you actually have.

[@rRobertSmith]

Yes we will ask ourselves where did all our jobs go, and what can I possibly retrain into.

[@privatecitizen4001]

Ppl said the same retarded stuff when the internet was invented. If you look at the traffic, we mainly use it for porn and Amazon.

[@Sephaos]

10 years? Think next year.

[@finbarrmcgrath1686]

You know they said the same think about the steam engine…..

[@TillmanZ]

Excellent interview! Great questions and really insightful answers! Thanks a lot Matt!

[@middleman-theory]

Man, what a clear and concise interview. Jonathan has such a simplistic-yet-informative way of conveying ideas and points, I'm learning so many new and interesting things in this discussion. This interview was like eating a well-balance meal with a side of cherry pie at the end. I'm full. I vote for more content like this as well. Keep it coming, my friend. I shared your video.

[@kenchang3456]

Thanks Matt for a great interview. Loved the Galileo analogy.

[@BoNaha]

Nice interview with a great person. 24 min of distilled knowledge ❤

[@VDMQuickView]

Great video, Matt. You should do more interviews like this. Excellent!

[@Taurus_Skyglaive]

We work on a system where npc agents decide in seconds what to do next and where a world reality is constructed for them at every other ocasion. Speed will let us do a lot more generation events and henceforth more decisions in the game were building. Speed is super important.

[@whiteycat615]

I didn't think much about it when clicking the video, but man, this was an excellent discussion. Thank you!

[@hotbit7327]

Same!

[@billcollins6894]

I have been consulting for Silicon Valley startups since 2014. I "retired" last year but I still help out a few startups for free. It is really nice to only work with projects I believe in because the money no longer matters. I do believe the vast majority of AI startups will fail, more so than usual. Capabilities are just changing too fast to have a solid business model particularly with predicting what competitors will do.

[@IlllIlllIlllIlll]

Interested in hearing your take why most AI startups will fail and very few will rocket

[@billcollins6894]

@@IlllIlllIlllIlll In general most startups fail. They usually fail because they don't understand one of a few key critical items. Typically it is overestimating the value of the product to consumers and underestimating the path to market challenges. The issue with AI is that as a startup gets going, whatever capability they are going after will emerge in another platform. I believe strongly that AI has a huge future. But carving out unique space will be difficult when everyone is jumping on board and the platforms inherently are self improving. Maintaining an advantage will be very difficult.

[@JeremyRabbit]

I love Jonathan’s supposition about and commitment to support AI models broadening people’s minds and understanding of each other and the universe we live in. Imagine how beneficial to society it would be if that was the focus and the result of people getting access to this technology.

[@minimal3734]

Since most people will be communicating with AI on a regular basis, I am convinced that this will happen. I have followed some of the conversations with Anthropic Claude 3. Regular interaction with an entity of such understanding and integrity must have a positive effect on the human mind.

[@jimg8296]

WoW, great interview. Thank you.

[@DailyTuna]

Congrats Matt for securing this interview! It’s cutting edge!

[@GeorgeMonsour]

This conversation is not to be overlooked and should be revisited. Mr. Ross is the responsible adult in the room. Please get insight from him regularly.

[@GetzAI]

Awesome interview Matt!!

[@maj373]

Just brilliant. The wisdom in this interview is extremely valuable

[@kitersrefuge7353]

Amazing interview. Groq has it licked. Whereas others are trying brute-force with GPU's he and his team have found how to leverage software on their hardware to do it more efficiently. Groq is going to win imo. I am switching to it tomorrow from chatGPT.

[@klaymoon1]

Amazing interview!!

[@TreeYogaSchool]

Wow! This was a great interview! Thank you for doing this, and you got me on Groq!

[@AtomicDreamLabs]

My favorite A.I. channel by far!!!

[@BradCordovaAI]

Great questions! And also great answers. This was an ideal, information packed low fluff interview. Thanks for having this. Very informative and entertaining.

[@Maisonier]

Amazing interview! Thanks for sharing it.

[@MeinDeutschkurs]

Great, Matt! Amazing opportunity. I wish there could be a company like groq, but also thinking about those, who want to run model XYZ at home. Like a Raspberry Pi, but for AI. (Raspberry AI?) There are so many who are not allowed to push data to whoever over the internet. No, APIs are not the way for personal/sensitive data.

[@jirikadlec7796]

You need to wait couple of years for that, or be really rich...

[@MeinDeutschkurs]

@@jirikadlec7796 , unfortunately. MacStudio with 192gb RAM is a good start, and already some kind of „Raspberry AI“, I bought and use. But it‘s way too slow. In comparison with A100 or H-series it‘s cheap. I really hope that with M6 Chip, Apple increases mem-bandwidth and the ML-cores dramatically.

[@babbagebrassworks4278]

I use Ollama to run LLM's and Onnxstream to run Stable Diffusion on my Pi5. They work fine and are only going to get better/faster as code get optimised.

[@babbagebrassworks4278]

@@jirikadlec7796 Or just find the right method now.

[@hitchslap8254]

I know almost nothing about coding but I'm pretty sure AI is going to eat my (financial services) lunch within 5 years. I'm trying to speed run understanding AI and this is one of the most informative interviews I've heard so far. Bravo.

[@Techonsapevole]

great interview, thanks!

[@adtiamzon3663]

Excellent questions, Matt! 🌞 Comprehensible explanations from Jonathan of Groq. Relatable, indeed. Good one. 👏👏😍 More from Groq Team. 🤔💐

[@NicolasEmbleton]

Really good interview. Jonathan really gets it! Impressive.

[@HybridLizard_com]

When "blazingly fast" is not an overstatement. Nice interview.

[@chrisn7847]

Great questions and interview, Matt. First time listener. First time caller here. Subscribed and will be back!

[@FabiCriativa]

Thanks for this amazing chat guys!

[@anthonykougkas5309]

@Jonathan Ross is an awesome dude. I love and share his point of view. Thank you for providing great examples (telescope, car factory, decision making story, etc). Thank you @Mathew for organizing and running this interview. I noticed you really wanted to pick Jonathan's brain about the future use cases and outcomes of the recent AI craze. He resisted but you insisted! Sign of a good journalistic mind! Your channel is dope! :>)

[@user-xt3vf9xp3f]

Very powerful last few words in the interview. Feels safer to hear these than the other interview.

[@muslimahmood]

Thank you, @Matt, for sharing. I wish you all the best of luck, @JonRoss, in your endeavours. #Groq

[@isaklytting5795]

20:06 Wow, I so liked his take on this. A lot of wisdom. I too think there is a lot of potential for making people more capable of understanding things, and of heightening the level of communication in terms of depth, seriousness.

[@torarinvik4920]

Amazing. More of these expert interviews!!!! Jonathan is a very charismatic speaker, he knows how to explain this stuff to us noobs.

[@scotlandcorpnaics2385]

Amazing & Intriguing discussion!

[@motarski]

Wow, what a content. Thank you Matthew.

[@ExecutiveZombie]

So informative. Thank you! 🙏🏽☀️🎧

[@jamesyoungerdds7901]

What a great interview, thanks Matt! Also, for the inference speed, in the area of agents, it would certainly be a lot faster because the agents communicating with each other don't have to read and interpret at human speeds, so you get a bunch of agents communicating at that token speeds and you're off to the races!

[@robertheinrich2994]

that was fascinating. and I'm really looking forward to how this works out.

[@tonykipkemboi]

Awesome interview, Matt!👏

[@alexjensen990]

That was awesome! I really enjoyed hearing his perspective and words of wisdom for those of us trying to find a place in this space.

[@arinco3817]

Wow superstar guest! Great interview you asked good questions. He didn't seem mega interested in agents tho?

[@kai_s1985]

Maybe he is not aware how important agents will be 🤔

[@jeffdude8713]

Do more of these. You’re an excellent interviewer.

[@timtim8011]

Super good stuff. Love Gen Gen!

[@GPT-zodiacReadings]

I've switched to Groq now and I'm really enjoying writing code with it due tro the speed. Thanks for this!

[@designthinkingwithgian]

Extremely articulate man. Thanks!

[@luisortega7028]

You are a gret interviewer. Ross was awesome too. Matthew, consider expanding your portfolio of interviews.

[@HMexperience]

Great interview. Have him back when they have an update for their chip launching. I believe it is made on 16 nm node so could be much faster on a more modern node.

[@pierretetreau7497]

Very impressive, loved it, all of it

[@r34ct4]

Man you are starting to get some incredible interviews

[@TairaKirkland]

Amazing content as always. I learn so much here, thank you for doing all the research for us! I’d never be able to keep up on my own.

[@Alice8000]

TOTAL GROQ-STAR⭐

[@dletendr]

Jon and product market fit = 💯

[@middleman-theory]

10/10 interview!

[@goddess_of_Kratos]

Love how speed subconsciously guided preference

[@maslaxali8826]

great interview mann

[@jordig3412]

great interview ;-)

[@PIOT23]

Great content!

[@ssygon2]

Smart guy, and love the metaphors used to describe the ideas and processes

[@richweborg3753]

This is great!

[@Shri-gk1ti]

Easy to understand analogies!

[@billcollins6894]

Given the extreme inefficiency of GPUs for AI, dedicated silicon/other materials are definitely the future.

[@fangeming1]

Thank you for the good work. I still lack basic information to understand the value of groq: if I want to run at home a 70 bn model quantised to 8 bits using grok hardware and reach a speed of 100 token per second, how many of their cards do I need to buy, what would be the costs, and what would be the power consumption? This is the only way for me to begin any kind of comparison.

[@mwissel]

Very interesting!

[@blackgptinfo]

He did a phenomenal job communicating complex tech in laymens terms

[@gremlinsaregold8890]

So... Starting to pull in the interviews with movers and shakers Matt. Well done and I think this would be a great direction for your channel too.

[@carloscampo9119]

This man is brilliant and it shows

[@YoungMoneyFuture]

Matt with consistent quality conent, great job! Jonathan said HE NEED SOME MILK!😂

[@goddess_of_Kratos]

I really hope our gov is supporting you

[@igorcosta]

Jonathan is a nice person and great engineer.

[@jonyfrany1319]

Fascinating 🧐

[@TubelatorAI]

0:00 1. Introduction 🌟 Meet Jonathan Ross, founder of Groq 0:34 2. Founding Groq 🚀 Jonathan's journey from Google to founding Groq 1:36 3. Innovation Decision 🤔 Reasons behind leaving Google for entrepreneurship 2:24 4. Groc Architecture 💡 The inception of Groq's unique chip architecture 3:08 5. Compiler Development 🖥 Early stages of Groq's compiler development 3:21 6. Inference Speed 🚄 Exploring Groq's impressive inference speed capabilities 3:57 7. Memory Optimization 🧠 Understanding the strategy behind Groq's memory design 5:19 8. The Efficiency of Assembly Line 🚗 Comparing car production to chip manufacturing efficiency. 5:55 9. GPU vs. Groq Chip Speed ⏱ Explanation of why GPUs are slow at producing tokens. 6:50 10. Introduction to Groq Cloud ☁ Details on starting with Groq Cloud and its benefits. 7:46 11. GROC Hardware Business Model 💼 Discussion on the potential business model for Groq hardware. 9:28 12. Compute as the New Oil 💻 Insights on the future of compute and its importance. 10:11 13. The Evolution of Technology 🌐 From the internet to generative AI. 11:05 14. Advice for AI Entrepreneurs 💡 Choosing the right focus in AI startups. 12:05 15. Challenges in Building AI Chips 💻 Obstacles faced by AI chip startups. 12:60 16. Predicting Success in AI 📈 Challenges in predicting AI model success. 13:46 17. The Future of AI Agents 🤖 Exploring the potential of AI agents. 14:25 18. Inference Speed and Agents 🚀 Impact of inference speed on AI agents. 20:26 19. Understanding the Impact of Media 📺 Exploring the influence of TV and social media on emotions and curiosity. 21:17 20. Embracing Generative AI 🤖 Discussing the positive impact of generative AI on curiosity and nuanced perspectives. 22:25 21. Controlling Algorithms and Bias 🧠 Addressing concerns about controlling algorithms and the potential biases in AI models. 22:54 22. Preserving Human Agency in AI Age 🛡 Exploring Groq's mission to ensure human control in the era of AI. 23:32 23. Empowering Decision-Making 🤔 Focusing on enabling individuals to make informed decisions with AI assistance. 24:03 24. Curating Information Challenges 📚 Discussing the importance and complexities of curating AI models and information. Generated with Tubelator AI Chrome Extension!

[@luismoreno1405]

Tubelator?

[@TubelatorAI]

@@luismoreno1405 It's a chrome extension for KZbin.

[@weredragon1447]

Awesome interview! Just one caveat. The challenge isn't keeping AI from making decisions for people. Controlling AI isn't the issue. The issue is keeping people from wanting it to make their decisions for them. Groq may not want AI to make people's decisions for them, but other people in power will. And too many people will just abdicate their power because it's easier than making their own decisions.

[@RC-uk8qs]

Great interview! Now interview Cerebras please!!!!!

[@CAMILOH]

Excellent 👍🏼👍🏼👍🏼

[@lucabarone2910]

I'm loving this conversation! The last question about the fear around AI really resonated with me. I'm both excited and scared for the progress of AI, and it's nice to know I'm not alone in feeling that way. But what really hit me hard was when Jonathan Ross talked about the analogy of Galileo inventing the telescope. It's wild to think that as we're expanding our understanding of intelligence, we're also realizing just how vast and capable it is. It's like, whoa, we're not even scratching the surface! And let's not forget about that new term for the new generation - "GenGen". Basically a new generation growing up with AI like we did with smartphones! -My creation, AI's elevation.

[@user-cw3jg9jq6d]

Hi Matthew, thank you for the content. Does anyone know which companies are top ten AI chip makers? I also would apprecciate any content that expalins why we need to build chips for AI specifically please.

[@vsanden]

It was an amazing guest and I loved listing to him. I tested the model and asked groq several riddles but it couldn't solve it yet and convincingly gave the wrong answer. I would prefer to hear that it didn't know. It were not very complicated questions like: A box contains two coins. One coin is heads on both sides and the other is heads on one side and tails on the other. One coin is selected from the box at random and the face of one side is observed. If the face is heads what is the probability that the other side is heads?

[@lancemarchetti8673]

Brilliant

[@colmxbyrne]

The brilliance: "Here's why compute is the new oil: The internet age (what we're leaving now) was where we made copies of data with High Fidelity and distribute it but that's also what the printing press was; they're effectively the same type of Technology just at a different scale. Now we are in the Gen AI age because we're not making copies of something - we're making it new new in the moment. The difference is when you're making something new in in the moment you need compute to do that. "

[@michaelphillips4197]

I feel like Jonathan didn't really get the point you were making about how increased inference speed will make Agents much more effective. I think this is a great point, I've seen improvements in my use of gpt-pilot by using the groq api endpoint (although token/minute limitations are annoying).

[@jackflash6377]

Excellent interview and it brings up something I was wondering about when I heard MS is going to build a $100 billion data center. Why spend that much money when the technology is growing so fast? It will take years to make such a big center, by the time it's finished it will be out dated. I didn't get enough answer when you were talking about agents. For complex problems it seems as if agents will be the solution and as you pointed out, inference speed will be big.

[@jeffg4686]

Love their business model. That's the one that works. Let you use it free and maybe create something really useful. If you make something that is popular, everyone makes money.

[@Davorge]

You are growing to the next level Matt! Soon to reach 1M+ subs

[@ivanmytube]

23:15 “convenience” for dictators to take control and make decisions.

[@mshonle]

Here’s a question: what about protein folding computations? This seems to be something with huge societal benefit that we aren’t talking about enough. Any idle compute should be incentivized to run protein simulations.

[@YoutubeWatcher264]

I think there are opensource versions of LLM that you can setup. No idea where to get the data to train it with.

[@paulsaulpaul]

Real quantum computers are best suited for this. No AI necessary other than to perhaps narrow down the possibilities. But there is also the issue of storage density, and time crystals will need to be advanced more for that.

[@JT-Works]

It is odd how he didn't address your agent question. You are correct that is the killer app. The fast tokens allow the agent to have super fast internal dialog to think about the answer before responding.

[@user-fy8gm1nc9g]

groq is incredible performance

[@nasrawi11]

Mattew, easy on us.

[@xraylife]

When is there a LPU PCI card coming for the PC ?

[@fangeming1]

This is my understanding: to run an llm of 70bn parameters, I will need to buy 200 Groq cards, which would cost 3,6millions USD and consume 50 kilowatts of power. I can run the same model, much slower I concede, with an nvidia hardware costing 2 thousand USD and consuming 60 watts of power. So the speed increase has a huge cost, limiting the use cases to very few specific situations. This calculation is based in the amount of memory built in the Groq cards and the assumption that the full llm model needs to be loaded and distributed to these card's memory.

[@PalimpsestProd]

I realize that groq is a language AI but this model of business gives me hope for custom after-market FSD, for cars with radar and sonar, to add or extend features.

[@dho449]

So groq can generate 500-600 tokens per second. In the video Jonathan mentioned groq requiring 1/10th the power of nvidia chips. Is that running at 500 tokens/second? Or is the comparison of power usage done while keeping the same number of tokens per second as openai. Another way of asking it is, while groq is generating 500-600 tokens per second is it 1/10th the power required for openai to run at its 50-100 tokens per second?