Direct Preference Optimization (DPO) explained: Bradley-Terry model, log probabilities, math

[@Patrick-wn6uj]

The legend returns, Always excited for your videos. I am an international student at Shanghai Jiao Tong daxue. Your videos have given me a very strong foundation of transformers. Much blessings your way

[@umarjamilai]

我们在领英联系吧，我有个微信小群，你可以参加

[@汪茶水]

​@@umarjamilai我也想加

[@汪茶水]

​@@umarjamilai我看到b站也有你的账号

[@EliRahimkhani]

same here from Canada! I can't thank you enough @umarjamilai

[@RudraPratapDhara]

Legend is back, the GOAT, if my guess is right next will be ORPO or Q*

[@umarjamilai]

Actually, the next video is going to be a totally new topic not related specifically to language models. Stay tuned!

[@olympus8903]

@@umarjamilai waiting

[@luxorska5143]

wow your explanation is so clear and complete... you are godsend, keep doing it. Sei un fenomeno

[@pietrogirotto6414]

Your explanations are on a whole another level, compared to whatever else you can find online. Keep up the amazing work and thank you!

[@DiegoSilva-dv9uf]

Valeu!

[@xugefu]

Thanks!

[@cken27]

Thanks for making these videos. Concise and clear

[@mlloving]

Thank you! It's very clear explaination. It helps for reading the original paper. Looking forward to new topics.

[@amanattheedge9056]

Very clear explanations!! Please, continue making such good videos!

[@vanmira]

These lectures are amazing. Thank you!

[@nwanted]

Thanks so much Umar, always learn a lot from your video!

[@kmalhotra3096]

Amazing! Great job once again!

[@yinghaohu8784]

You explained very clearly. Thanks!

[@sauravrao234]

I humbly request you to make videos on how to build a career in machine learning and AI. I am a huge fan of your videos and i thank you for all the knowledge that you have shared

[@umarjamilai]

Hi! I will for sure make a video in the future about my personal journey. I hope that can help more people in navigating their own journeys. Have a nice day!

[@janigiovanni6075]

Great video, thank you very much for this!

[@amankhurana2154]

Awesome, thank you so much for putting this out, super helpful!

[@alexm1815]

This is very, very good. Thank you!

[@SaiKiran-jc8yp]

Best explanation so far !!!!...

[@mrsmurf911]

Love from India sir, you are a legend 😊😊

[@lukeskywalker7029]

New video🎉 can't wait to watch. Although having used DPO in production for a while now!

[@sidward]

Thanks for the great video! Very intuitive explanation and particular thanks for the code examples. Question: at 37:41, how do we know that the solving the optimization problem will yield the pi_*? Is there a guaranteed unique solution?

[@umarjamilai]

Please check the paper I linked in the description for a complete derivation of the formula. It is also done in the DPO paper, but in my opinion the other paper is better suited for this particular derivation.

[@k1tajfar714]

Awesome Video. please Continue.

[@binjianxin7830]

I believe the most evident insight of DPO is to change a RL problem to an equivalent MLE, while the optimal reward model is guarranteed by the human input as definition. That's the meat. But the efficiency depends still on the human annotater's consistency.

[@jak-zee]

Enjoyed the style in which the video is presented. Which video editor/tools do you use to make your videos? Thanks.

[@umarjamilai]

I use PowerPoint for the slides, Adobe Premiere for video editing

[@jak-zee]

@@umarjamilai What do you use to draw on your slides? I am assuming you connected an ipad to your screen.

[@elieelezra2734]

Hello Umar, Great as usual, however why do you say at 46:11, that you need to sum log probabilities up? The objective function is the expectation of logarithm of the difference of two weighted log probabilities ratios. I don't get what do you want to sum up exactly? Thank you

[@tuanduc4892]

Thanks for your lecture. I wonder could you explain the vision language models

[@koiRitwikHai]

Great explanation but I have some doubts, please help 36:50 in Ldpo π* was replaced with π theta... why π theta is considered as optimal policy? 44:13 You said "each hidden state contains information about itself and all the tokens that comes before it", but this is applicable only to decoder part of the transformer. So this transformer layer is actually a decoder layer? like GPT

[@AndriiLomakin]

Thank you for the video ! Can you provide the video that explains AgentQ training in details ?

[@abdullahalsaadi5991]

Amazing explanation. Would it be possible to make a video on the theory and implementation of automatic differentiation (autograd).

[@agnarCS]

thank you

[@kevinscaria]

Brilliant!!!!!!

[@olympus8903]

My Kind Request Please Increase volume little bit , just little bit. Otherwise your videos Outstanding . Best I can say.

[@mahdisalmani6955]

Thank you very much for this video, please make ORPO as well.

[@Mortazaghafaripour]

Great 👍

[@tommysnowy3068]

Amazing video. Would it be possible for you to explain video-transformers or potential guesses at how Sora works? Another exciting idea is explaining GFlowNets

[@ai.mlvprasad]

what is the ppt software you are using sir ?

[@vardhan254]

love ur videos umar !!

[@TemporaryForstudy]

great video. love from india.

[@AptCyborg]

Amazing Video! Please do one on SPIN (Self Play Fine-tuning) as well

[@CarterKira-g9s]

great explaination, thanks. how about the recent work: KTO: Model Alignment as Prospect Theoretic Optimization? can you compare it with DPO?😁

[@ernestbeckham2921]

Thank you. can you make video about liquid neural network?

[@mohammadsarhangzadeh8820]

I love ur videos so much. please make a video about mamba or mamba vision

[@umarjamilai]

There's already a video about Mamba, check it out

[@OGIMxGaMeR]

Thank you very much for the explanation. I had one questions. Are the dataset of preferences always made of two and only two answers?

[@umarjamilai]

According to the Hugging Face library, yes, looks like you need a dataset with prompt and two answers, one is called the "chosen" one and the other is the "rejected" one. I'm pretty sure there are ways to convert more than two preferences into a dataset of two preferences.

[@OGIMxGaMeR]

@@umarjamilai thank you! Yes of course. I am just wondering why it wouldn’t help to have more than 1 rejected for 1 accepted. I guess the formula does not consider this case but may add value.

[@plslokeshreddy]

Thanks for the video. Do you know any way on how we can create a dataset for DPO training. I currently have only question, answer pairs. Is it fine if i take y_w as answer and y_l as some random text(which would obviously have lower preference than answer) and then train it?

[@plslokeshreddy]

The potential problem that I think could happen is that having random text may decrease the loss and the policy may not even change much

[@nguyenhuuuc2311]

Hi Umar, If I use LoRA for fine-tuning a chat model with DPO loss, what should I use as a reference model? - The chat model applied LoRA - Or the chat model itself without LoRA?

[@umarjamilai]

Considering LoRA is just a way to "store" fine-tuned weights with a smaller computation/memory footprint, the model WITHOUT LoRA should be used as the reference model.

[@nguyenhuuuc2311]

@@umarjamilai With my limited GPU, I can only fine-tune by combining a 4-bit-quantized model + LoRA. Surprisingly, using just the 4-bit model leads to NaN weight updates after one batch. But once LoRA is added, my loss updates smoothly without any problems.

[@nguyenhuuuc2311]

Thank you SO much for the quick answer and your excellent video. I did get the hang of DPO loss and be able to implement DPO loss + training loop with vanilla PyTorch code.

[@trungquang1581]

thank you so much for your effort! could you make a video about tokenizers like BPE and sentencepiece from scratch? I would be very appreciate of it!

[@samiloom8565]

I enjoy your videos umar on my phone while commuting or sitting in a coffe. Only the small fint on a phone is tiring me ..if you make them a bit bigger that will be better

[@umarjamilai]

Sorry for the trouble, I'll keep it in mind for the next videos!

[@kevon217]

“digital biscuits”, lol