Aligning LLMs with Direct Preference Optimization

[@PritishYuvraj]

Excellent description between PPO and DPO! Kudos

[@eliporter3980]

I'm learning a lot from these talks, thank you for having them.

[@NitinPasumarthy]

The best content I have seen in a while. Enjoyed both the theory and the practical notes from both speakers! Huge thanks dlai for organizing this event

[@AurobindoTripathy]

cut to the chase - 3:30 questions on DPO - 27:37 practical deep-dive - 30:19 question - 53:32

[@vijaybhaskar5333]

Excellent topic. Well explained. One of the best videos on this subject I saw recently. Continue your goodwork😊

[@amortalbeing]

This was amazing thank you everyone. One thing though, if thats possible, it would be greatly appreciated if you could record in 1080p where the details/text on the slides are visible and easier to consume. Thanks a lot again

[@MatijaGrcic]

Check out notebooks and slides in the description.

[@amortalbeing]

@@MatijaGrcic Thanks a lot, downloaded the slides

[@katie-48]

Great presentation, thank you very much!

[@jeankunz5986]

great presentation. Congratulations.

[@PaulaLeonova]

At 29:40 Lewis mentions an algorithm that requires fewer training samples, what is the name of it? I heard "data", but don't think that is correct. If anyone knows, would you mind replying?

[@AurobindoTripathy]

Possibly, this paper, "Rethinking Data Selection for Supervised Fine-Tuning" arxiv.org/pdf/2402.06094.pdf

[@ralphabrooks]

I am also interested in hearing more about this "data" algorithm. Is there a link to a paper or blog on it?

[@austinmw89]

Curious if you compared SFT on all data vs. training on completions only?

[@TheRilwen]

I'm wondering why simple techniques, such as sample boosting, increasing errors for high ranked examples, or attention layer wouldn't work in place of RLHF. It seems like a very convoluted and inefficient way of doing a simple thing - which convinces me that I'm missing something :-)

[@dc33333]

muy bueno gracias

[@iseminamanim]

Interested

[@trisetra]

kzbin.info/www/bejne/h4m5dKSqdr90pJY The details in the Llama3 paper seem to validate the claim that DPO works better than RL at scale.

[@MacProUser99876]

How DPO works under the hood: kzbin.info/www/bejne/gKaQoXmAg8uCnLs