Direct Preference Optimization (DPO) explained: Bradley-Terry model, log probabilities, math

[@Patrick-wn6uj]

The legend returns, Always excited for your videos. I am an international student at Shanghai Jiao Tong daxue. Your videos have given me a very strong foundation of transformers. Much blessings your way

[@umarjamilai]

我们在领英联系吧，我有个微信小群，你可以参加

[@汪茶水]

​@@umarjamilai我也想加

[@汪茶水]

​@@umarjamilai我看到b站也有你的账号

[@EliRahimkhani]

same here from Canada! I can't thank you enough @umarjamilai

[@RudraPratapDhara]

Legend is back, the GOAT, if my guess is right next will be ORPO or Q*

[@umarjamilai]

Actually, the next video is going to be a totally new topic not related specifically to language models. Stay tuned!

[@olympus8903]

@@umarjamilai waiting

[@luxorska5143]

wow your explanation is so clear and complete... you are godsend, keep doing it. Sei un fenomeno

[@pietrogirotto6414]

Your explanations are on a whole another level, compared to whatever else you can find online. Keep up the amazing work and thank you!

[@cken27]

Thanks for making these videos. Concise and clear

[@mlloving]

Thank you! It's very clear explaination. It helps for reading the original paper. Looking forward to new topics.

[@amanattheedge9056]

Very clear explanations!! Please, continue making such good videos!

[@kmalhotra3096]

Amazing! Great job once again!

[@nwanted]

Thanks so much Umar, always learn a lot from your video!

[@vanmira]

These lectures are amazing. Thank you!

[@sauravrao234]

I humbly request you to make videos on how to build a career in machine learning and AI. I am a huge fan of your videos and i thank you for all the knowledge that you have shared

[@umarjamilai]

Hi! I will for sure make a video in the future about my personal journey. I hope that can help more people in navigating their own journeys. Have a nice day!

[@yinghaohu8784]

You explained very clearly. Thanks!

[@amankhurana2154]

Awesome, thank you so much for putting this out, super helpful!

[@alexm1815]

This is very, very good. Thank you!

[@lukeskywalker7029]

New video🎉 can't wait to watch. Although having used DPO in production for a while now!

[@SaiKiran-jc8yp]

Best explanation so far !!!!...

[@binjianxin7830]

I believe the most evident insight of DPO is to change a RL problem to an equivalent MLE, while the optimal reward model is guarranteed by the human input as definition. That's the meat. But the efficiency depends still on the human annotater's consistency.

[@k1tajfar714]

Awesome Video. please Continue.

[@mrsmurf911]

Love from India sir, you are a legend 😊😊

[@mahdisalmani6955]

Thank you very much for this video, please make ORPO as well.

[@olympus8903]

My Kind Request Please Increase volume little bit , just little bit. Otherwise your videos Outstanding . Best I can say.

[@tuanduc4892]

Thanks for your lecture. I wonder could you explain the vision language models

[@AndriiLomakin]

Thank you for the video ! Can you provide the video that explains AgentQ training in details ?

[@vardhan254]

love ur videos umar !!

[@abdullahalsaadi5991]

Amazing explanation. Would it be possible to make a video on the theory and implementation of automatic differentiation (autograd).

[@kevinscaria]

Brilliant!!!!!!

[@jak-zee]

Enjoyed the style in which the video is presented. Which video editor/tools do you use to make your videos? Thanks.

[@umarjamilai]

I use PowerPoint for the slides, Adobe Premiere for video editing

[@jak-zee]

@@umarjamilai What do you use to draw on your slides? I am assuming you connected an ipad to your screen.

[@AptCyborg]

Amazing Video! Please do one on SPIN (Self Play Fine-tuning) as well

[@sidward]

Thanks for the great video! Very intuitive explanation and particular thanks for the code examples. Question: at 37:41, how do we know that the solving the optimization problem will yield the pi_*? Is there a guaranteed unique solution?

[@umarjamilai]

Please check the paper I linked in the description for a complete derivation of the formula. It is also done in the DPO paper, but in my opinion the other paper is better suited for this particular derivation.

[@elieelezra2734]

Hello Umar, Great as usual, however why do you say at 46:11, that you need to sum log probabilities up? The objective function is the expectation of logarithm of the difference of two weighted log probabilities ratios. I don't get what do you want to sum up exactly? Thank you

[@plslokeshreddy]

Thanks for the video. Do you know any way on how we can create a dataset for DPO training. I currently have only question, answer pairs. Is it fine if i take y_w as answer and y_l as some random text(which would obviously have lower preference than answer) and then train it?

[@plslokeshreddy]

The potential problem that I think could happen is that having random text may decrease the loss and the policy may not even change much

[@tommysnowy3068]

Amazing video. Would it be possible for you to explain video-transformers or potential guesses at how Sora works? Another exciting idea is explaining GFlowNets

[@nguyenhuuuc2311]

Hi Umar, If I use LoRA for fine-tuning a chat model with DPO loss, what should I use as a reference model? - The chat model applied LoRA - Or the chat model itself without LoRA?

[@umarjamilai]

Considering LoRA is just a way to "store" fine-tuned weights with a smaller computation/memory footprint, the model WITHOUT LoRA should be used as the reference model.

[@nguyenhuuuc2311]

@@umarjamilai With my limited GPU, I can only fine-tune by combining a 4-bit-quantized model + LoRA. Surprisingly, using just the 4-bit model leads to NaN weight updates after one batch. But once LoRA is added, my loss updates smoothly without any problems.

[@nguyenhuuuc2311]

Thank you SO much for the quick answer and your excellent video. I did get the hang of DPO loss and be able to implement DPO loss + training loop with vanilla PyTorch code.

[@koiRitwikHai]

Great explanation but I have some doubts, please help 36:50 in Ldpo π* was replaced with π theta... why π theta is considered as optimal policy? 44:13 You said "each hidden state contains information about itself and all the tokens that comes before it", but this is applicable only to decoder part of the transformer. So this transformer layer is actually a decoder layer? like GPT

[@OGIMxGaMeR]

Thank you very much for the explanation. I had one questions. Are the dataset of preferences always made of two and only two answers?

[@umarjamilai]

According to the Hugging Face library, yes, looks like you need a dataset with prompt and two answers, one is called the "chosen" one and the other is the "rejected" one. I'm pretty sure there are ways to convert more than two preferences into a dataset of two preferences.

[@OGIMxGaMeR]

@@umarjamilai thank you! Yes of course. I am just wondering why it wouldn’t help to have more than 1 rejected for 1 accepted. I guess the formula does not consider this case but may add value.

[@TemporaryForstudy]

great video. love from india.

[@mohammadsarhangzadeh8820]

I love ur videos so much. please make a video about mamba or mamba vision

[@umarjamilai]

There's already a video about Mamba, check it out

[@ai.mlvprasad]

what is the ppt software you are using sir ?

[@ernestbeckham2921]

Thank you. can you make video about liquid neural network?

[@CarterKira-g9s]

great explaination, thanks. how about the recent work: KTO: Model Alignment as Prospect Theoretic Optimization? can you compare it with DPO?😁

[@Mortazaghafaripour]

Great 👍

[@samiloom8565]

I enjoy your videos umar on my phone while commuting or sitting in a coffe. Only the small fint on a phone is tiring me ..if you make them a bit bigger that will be better

[@umarjamilai]

Sorry for the trouble, I'll keep it in mind for the next videos!

[@trungquang1581]

thank you so much for your effort! could you make a video about tokenizers like BPE and sentencepiece from scratch? I would be very appreciate of it!

[@DiegoSilva-dv9uf]

Valeu!

[@kevon217]

“digital biscuits”, lol