Stanford CS25: V3 I Retrieval Augmented Language Models

December 5, 2023
Douwe Kiela, Contextual AI
Language models have led to amazing progress, but they also have important shortcomings. One solution for many of these shortcomings is retrieval augmentation. I will introduce the topic, survey recent literature on retrieval augmented language models and finish with some of the main open questions.
More about the course can be found here: web.stanford.e...
View the entire CS25 Transformers United playlist: • Stanford CS25 - Transf...

[@erniea5843]

I love that this content is freely accessible to everyone. Lots of helpful information being shared here

[@nintishia]

The amount of research work on retrieval augmented generation for large language models has exploded in recent times. Thanks to the speaker for directing attention to the most significant bits.

[@dongxu9013]

The best talk about RAG so far

[@morespinach9832]

How many have you heard, right here on KZbin? Lots of actual hands on info out there and in 1/3rd the time. This one had loads of intro forever before getting into specifics.

[@edwardmitchell6842]

@@morespinach9832 Can you recommend some sources? I'm compiling a list.

[@yangzju]

@@morespinach9832 can you recommend some videos?

[@comunedipadova1790]

@@morespinach9832 What lecture do you suggest for a more practical (code) view?

[@morespinach9832]

@@comunedipadova1790 plenty of them - search for these keywords.

[@capucinnolover]

Nicely explained in just right technical details. Thank you!

[@heisenberggiao9303]

Thank you for the great contents!

[@Arvolve]

So many great ideas here! Fantastic resource, thank you.

[@velociraptor75013]

Excellent content, thanks for the references!

[@loopaal]

this is what I needed, thank you sooooo much!!!!

Good lecture, offers a good summary of the literature on RAG.

[@ronitakhariya4094]

just the thing i wanted thank you so much.

[@kingki1953]

My lecturers in my university never explain those things, thanks for this free lecture

[@99BLACKLP]

Great insights and video!

[@NerdyX90]

Thank you.

[@johntanchongmin]

Pros: Gives a brief overview of many RAG methods Cons: No intuition given which is the key reason for why it works for the different methods Would have preferred more insights rather than just describing the papers, but overall thanks for the video!

[@reslleygabriel]

Awesome content, thanks for sharing!

[@deeplearningpartnership]

Really cool.

[@duongkstn]

Hi, about Atlas. You said that we can update the Retriever. At 42:00 is some retriever loss, but what about pair label (question, positive paragraph, negative paragraphs) like normal retrieval model - do they contribute to the retriever loss ? I read codebase of Atlas, and do not find that kind of loss

[@echofloripa]

Does these embeddings tools such as FAISS work with other languages other than English?

[@muhannadobeidat]

It is good to see the whole spectrum of options, but what would be a practical way to get started on this? His old colleagues in HF actually have an excellent book that goes through many different ideas including RAG in chapter 7 where they do an excellent job explaining context and giving you options for implementation.

[@蔡鉎驊]

3:34 What chatgpt was really about - fix the user interface to LM 12:00 Frozen RAG

[@samferrer]

Actually we can see a first sign of language models in Shannon's 1948 paper, A Mathematical Theory of Communication.

[@ssssssstssssssss]

Language models started to appear in the 1800s.

[@therainman7777]

Language models actually appeared sometime during the Cretaceous period, though scientists aren’t quite sure of the exact year. They think the stegasaurus might have had something to do with it; all these people who think OpenAI invented them are so wrong.

[@zhuolin730]

where can I find the slides?

[@trinityblood5622]

Could you please share the slides?

[@lambertch]

Shouldn’t it be Shannon, 1949?

[@ginogarcia8730]

WE ARE IN THE FUTUREEEEEE

[@ritikdua7225]

does anyone have list of research papers metioned in this video?

[@AIPoker-tj6lr]

What about scann by google ??

[@deconcoder]

THe bit about where this came from was funny, but your search just gave a less silly answer. Go reread Shannon's 1948 paper that invented Information Theory. Yes he did not talk about using neural nets (which did not exist) but he did talk about probability.

[@amansinghal5908]

Are there assignments for this?

[@vaibhavjoshi7961]

I did like the joke you made.

[@kingki1953]

why there's no framework to make the process was easy

[@Arthurlvaz]

It's not entirely clear what "frozen rag" and "retrieve rag" refer to.

[@user-wr4yl7tx3w]

there is too much review information

[@MedTn-i3h]

worth dozens of "shiny" "trendy" "tik-toky" videos over there, without asking you to subscribe and leave comments..

[@jstello]

I love how everyone tries to hide the fact that OpenAI is 100% the reason everyone is watching this video

[@pw7225]

No one is hiding anything

[@a3mia3mi82]

“Ignorance” is not equal to “facts”

[@therainman7777]

I know, isn’t it hilarious? The bitterness and jealousy is so transparent. It reminds me of the scene from The Social Network where they’re trying to claim credit for his creation and he says “You know it’s really not that complicated. If you guys were the inventors of Facebook, you’d have invented Facebook.” Yann LeCun likes to drone on on Twitter about how everything OpenAI is doing is “old” technology. Well if it’s so old, how come hundreds of different companies filled with smart people were trying for years to make a chatbot that was worth using but until OpenAI no one seemed to manage it? Yes, OpenAI didn’t invent the Transformer. We know. Who cares? They clearly solved dozens of incredibly difficult engineering problems that no one else had been able to solve, and gave the world a language-based AI that was actually _useful._ As evidenced by the fact that it was the fastest growing app in human history, by an extremely wide margin. And as soon as they do it all these idiots turn up basically saying “I could have done that 10 years ago, I just chose not to.” Yeah, ok. It’s so pathetic.

[@Gingnose]

Maybe because only you are the one that is obsessed by the idea of starting learning something new but merely due to its "novelty and coolness" is something to be ashamed of and you project that idea on others which make you think others are watching this video with the same reason but hinding that information (and trying to look smart) because it is emabrassing to declare and you think that way too. However I will remind you that trying to learn new things whether it is because it is fashion or cool is nothing to be ashamed of and a great excuse to start new things.

[@srh80]

Whats an Open Al?

[@starmountpictures]

meh

[@mahkhi7154]

Us Mandalorians CYBERMEN: Understand The Material - Given To You By Minions, better Than The Minions. Your Gods Gods. Ask Them.

[@therainman7777]

Could you try writing that again but having it not be nonsense this time?

[@mahkhi7154]

You're a P1G. Memorising and Repeating without Understanding. What is a Sentence? A Collection of Concepts. The Brain can Learn 10's of Thousands of concepts. It Has Hardware To Help it. Concept: "Doing" brings up pictures of Making a Cake or Assembling a Chair. Concept: "What" brings up Pictures of "Asking someone a Question" or Deciding Colour Show to Buy. A Computer Doesn't Work Like That. All it Understands is Numbers and Maths. All These "Concepts" have To Be Encoded in Numbers and Maths and Rules Applied To Them. The Computer Doesn't Know What The Character "B" is. All it Know That if it Finds The Number "66" in Memory, it Draws Something That Looks like "B". Spelling. It Doesn't Know What Spelling is. All it Knows is "66,65,84" Valid for word BAT. BAB 66,65,66 is an Invalid String of Numbers and Not in the Dictionary. 10,000 Concepts Need - Modelling and Rules Creation (100's of Thousands) - For the Computer To Understand. That isn't an Easy Thing To Do: Trillions of Lines of Java Code.

[@therainman7777]

Maybe if you were a computer maybe you would understand that you Don’t Need To Capitalize Every Word when writing English. And trillions of lines of code? You know nothing about coding; in all of human history everyone out together has not written a trillion lines of code, let alone in a single computer or a single program. P.S. Your comment was utter nonsense, it contributed nothing, and it meant nothing. Have a nice day.

[@dongmo6546]

Thank you