Stanford Webinar - Large Language Models Get the Hype, but Compound Systems Are the Future of AI

[@labsanta]

00:14 - The future of AI lies in compound systems, despite the hype around large language models. 02:27 - AI's future relies on integrated systems, not just standalone models. 06:50 - Focus on entire systems rather than just individual model components. 09:08 - Exploring diverse methods for model output generation beyond basic token selection. 13:22 - Sampling and prompting are crucial for AI system behavior. 15:32 - GPT-3 showcases advanced in-context learning for various tasks. 19:36 - Model performance varies significantly with prompt framing. 21:23 - Understanding AI requires a systems thinking approach that integrates models and prompts. 25:14 - Optimizing language model prompts enhances flexibility and performance. 27:07 - Systematic thinking enhances language model performance via optimization strategies. 31:03 - Cost constraints necessitate efficient system design for AI models. 32:57 - Future AI will involve complex systems rather than just large models. 37:01 - Future AI advancements hinge on diverse scaling methods beyond unsupervised training. 39:00 - Future AI will focus on compound systems over standalone language models. 42:37 - Complexity of AI systems will increase, drawing parallels to evolving technologies like Google Search. 44:33 - Language models will evolve, impacting society in both positive and negative ways. 48:17 - AI systems require careful oversight to prevent unintended consequences. 50:10 - Navigating AI development requires clear goals and understanding risks involved. 53:59 - Starting with proper software systems avoids pitfalls of prompt templates. 55:49 - Focus on systems, not just models, for effective AI development.

[@dr.teerakiatkerdcharoen2338]

Thanks so much. 🥰🥰🥰

[@christian.adriano]

Saved me 58 minutes. Thanks!

[@rakeshd7131]

What model did you use to summarize the video? 😄

[@marcinwk]

Trying to get serious about LLMs and how to use them appropriately and this lecture just jolted me with zap of electricity! What a wonderful, thoughtful content and what clear and articulate delivery! I just might replay this lecure to learn how to give a talk about a topic, any topic... Thank you sir!

[@Andre-mi6fk]

What a great lecture this was!! Super important for anyone building AI Products with LLMs. Even if you think you know the material, it is good reinforce the best practices.

[@juanjoserojasconstain6561]

This was great! I finally get what was the interesting thing about in-context learning and emergent capabilities. Despite of being trained just to predict the next token, the model can learn perform NLP tasks (summarization, QA), without further training. Just from the right prompt. Before that, any model should be trained specifically for one of those tasks. 14:25 Reflecting on his idea of systemic thinking (7:23) is a must if you want to build applications with LLM, as he shows in 29:18. Using the same model (GPT-3.5), we can get a 20% performance boost just from the right prompt-optimization system 31:23. The questions were also very though provoking 32:05. I think almost all answers are clear: smaller models with good systems could be more powerful. Thank you very much, Prof. Potts.

[@saurabhsrivastava69]

Standford lectures never disappoints in content

[@potsy555]

in *context

[@youngzproduction7498]

Bros, this change my view on the model. Your clip unlocks many new ideas for me. 🎉 great work!!

[@alirizaerfan1969]

What a great and insightful lecture! Learned a lot. Thanks much.

[@aproperhooligan5950]

Fantastic discussion and reality check. Keep it coming, please.

[@danielnofal]

Completely agree that the model is just a part of it and we should be talking about compound systems. I would say that probably WE are a compound systems of neural networks competing for control pretty pretty pretty much like inside out movie.

[@dewinmoonl]

chris potts is always a good watch : )

[@null4624]

Learned a lot, thank you.

[@learnbydoingwithsteven]

However researchers advance in this field of language processing/inderstanding, one aspect is risk control on “random walk.” On system outputs, the other is on input interpretation. It seems to me that these engineering aspects could be incorporated in the models, with innovative designs in the future.

[@meisherenow]

Old-fashioned software: control, predictability, testability. LLLs: power, flexibility, generality. Together: controllable, testable, predictable, flexible, general power

[@shivibhatia1613]

Brilliant questions and explanations

[@learnbydoingwithsteven]

Very insightful.

[@diga4696]

Fascinating talk! While I agree that compound systems are critical, I wonder if the future of AI might involve a unification of models and systems, where the 'peripherals' evolve into integral modalities of the architecture itself. Couldn’t these compound systems eventually become emergent behaviors within a truly scalable architecture? That said, rather than focusing on system design as an external layer, wouldn’t it be more impactful to explore architectural innovations like active inference or test-time adaptations to improve generalization and scalability? For instance, refining pre/post-training processes could allow for more dynamic integration of tools and capabilities, effectively bridging the gap between model and system. In my view, attention-based architectures still hold a decisive edge over external system optimizations-but perhaps the two approaches are not mutually exclusive. Manning’s vision of coordination between smaller, specialized models and tools does suggest a fascinating synergy between attention mechanisms and compound systems.

[@Maximos80]

I like the way you think. Interested to discuss this further with you.

[@BitShifting-h3q]

thank you !! so happy to have found this

[@IsxaaqAcademy]

Great perspective

[@samyio4256]

Thank you so much! Got it :)

[@GrowStackAi]

Don’t call it a robot; it’s an intelligent assistant 🔥

[@NLPprompter]

22:03 is that prompt for RAG that's seems sucks prompt...

[@yumingliu7403]

This is a brand new architecture of the AI system that may be able to make large impact on people's life, we currently have many large language models, GPT, Gemini, LLaMA, etc., if they can be combined and interacted with each other, there might be a great chance to build more and more inteliggent AI system. Now the question is, how can we, as a developer for example, get start with developing a system like this, are there any resources, opensource project, tutorials or guidlines to follow, thanks.

[@JohanZahri]

There's gut brain and there's head brain; so far we have gut brain in gpt models; how do we translate the head brain mechanics in our system?

[@neoaistudios]

5:50 in my case is converged thinking, Im creating this system to create infinite diverse content/films, thought oromots

[@ionuchin]

Interesting... how about writing prompts in JSON format (not output, but input)? It gives some advantages for prompt generation.

[@VKjkd]

Oh wow. I didn’t think of this. Any examples of what/why this is useful? I can imagine it reduces issues in attention.

[@richardnunziata3221]

It would be nice if we started using blockchain and cryptography so access and be controlled to critical resources permissions and monitoring of excess accumulations of access keys in the blockchain by any one system

[@statebased]

Might the choice of presenting models as abstractions, and systems as "something else", be a communication strategy, just like the choice to emphasize models?

[@MegaStatis]

Enlighten talk.

[@ceilingfun2182]

It’s been known since GPT-2 that one prompt doesn’t work the same way on another model.

[@joe_hoeller_chicago]

Well, I built this 5y ago and got cancelled for it. 🤷‍♂️ I find it funny how Stanford vc’s are saying this now that they funded a company that just launched this. I’ve used this system my whole career since. This is nothing new.

[@mmasa1]

is compound AI system another word or way to describe AI Agents?

[@justwanderin847]

where are you on discord?

[@MrBox4soumendu]

...thank you SIR... got the key, its multiplayer systems we need to focus more... and the marketing side of wisdom will always be behind academic side, selling other's original contribution 🙏🙏👏👏👏👏

[@LatentSpaceD-g3p]

love the wheels on the f1 engine- they appear to be 2 skateboard wheels and 1 roller-blade wheel !!

[@ArgoCrawler]

I predict the winners in this new world will be those that most efficiently work with their ai tool(s). Obviously, redundancies should be removed for more efficency.

[@DistortedV12]

Define compound systems? Can someone help me skip to that point to save time

[@kasgol-zl9xo]

I am not quite sure this system approach will take us back a step into machine paradigm, and comparing it to a car might not be the right analogy.

[@viky2002]

he sounds like hulk

[@AppointmentWithJase]

AI is the worst thing humanity has invented since nuclear weapons.

[@justwanderin847]

We do NOT need government to regulate AI

[@TheLiteralist-j5h]

nor companies

[@dadsonworldwide3238]

You did Reagan thatcher traded & negotiated it far from American domestic courts jurisdiction and workers to tiawan for 50 yrs. Even before that people started world Wars to stop usa from texting this 2024 reply in concert with TV radio and we did 80+ yrs of ease of access teaching men and myth in stead .

[@dadsonworldwide3238]

The question is why did it have to be government held far away during the nuclear age and could we have not allowed those blocked out of echo chamber feilds miss aligned learning the hardway and instead allowed those who didn't need 1945s Smith_mundt act to get along the ability to live life out building there multi generational project where they left off puritanizing English and pilgrimage to confirm it common sense objectivism to program it with. We didn't have to wade through so many bad idealogy to now seemingly be left to fight back through it anyway for true optimization

[@dadsonworldwide3238]

If your American your very wall outlet y axis plugs +/- grounded by planetary nature dictating phase changes is just how fine tuned this is encoded every step of traceability has been planned for. Lol But here at the finish Line over since ww2 it's out of context problems are still present and not dealt accordingly

[@Crawdaddy_Ro]

Then how should it be regulated?

[@MohdAli-nz4yi]

Disappointing talk. Everyone and their grandma knew LLM's would just be a component of the system, whoop de doo. Also the bitter lesson.

[@SubhamKumar-eg1pw]

Thinking about this…

[@danielvalentine132]

Beyond the click bait, and the first 10 minutes, the rest of the video is a good demonstration and explanation of good systems thinking and architecture. There is a enough dogma around “prompt engineering” and “models”. Not only did he remind us that systems are definitely the way, but he also points us in the right direction. I thought this was a very rewarding talk if you listen to the whole thing.

[@MohdAli-nz4yi]

@@danielvalentine132 I watched the whole thing. I appreciate Stanford sharing this content, I am very grateful for that. But this talk truly did not contain anything new or interesting in my opinion. Especially the "some questions to mull over" section. That pissed me off a bit actually. "Which is more reliable? A giant LLM that embeds a snapshot of the entire web as of today or A tiny LLM working with an up to date web search engine" He asks leading questions, makes false dichotomies. Especially trying to take credit "for predicting" that LLM's are used as components to software systems. This sort of corporate fluff talk BS is not what I'd expect from a university professor.