Multi-Armed Bandit : Data Science Concepts

Making decisions with limited information!

[@Sad-mm8tm]

I hope you will continue making videos forever. Your explanations are the best I've ever seen anywhere + the wide choice of topics gives me food for thought when dealing with my own optimization problems.

[@ritvikmath]

Thank you :) I'm happy to help

[@VahidOnTheMove]

If he makes videos forever, we'll get zero regrets.

[@abhinavbhatia5114]

​@@ritvikmathdon't let this channel die man

[@savasozturk00]

After watching 5 videos, finally I found the best lecture teller for this topic. The examples are great, Thanks.

[@khaalidmcmillan9260]

Well said, needed a refresher after not seeing this for a while and this nailed it. Hopefully you've gone into more advanced topics like MAB reinforcement learning

[@heteromodal]

Great video, and it's really nice listening to you! Thank you :)

[@simoneorlando2496]

It would be great if you made a whole playlist where you explain the statistics for machine learning by explaining the formulas in an intuitive way like you do (you make me understand them all). For example, explain the various distributions and their meaning, statistical tests (p-value), etc. Thank you so much for the work you do and the knowledge you share!

[@malice112]

What a great and easy to understand explanation of MAB - thank you for this!!!!

[@CaioCarneloz]

The way you explain is stunning, what a awesome lesson.

[@111dogger]

This is the best explanation I have come across so far for the Upper Bound Confidence concept. Thank you!

[@adanulabidin]

What an amazing explanation! Thank you so much. Keep making such videos.

[@marcelobeckmann9552]

Your explanations, didactics, and dynamism are amazing, way better than several university professors. Well done!

[@abdulsami5843]

A thing I absolutely like is how palatable you make these concepts, not too mathematical/theoratical and not overly simplified, just the right balance ( € - greedy is set right 😉)

[@vahidsohrabi94]

I'm grateful to you because of this great tutorial.

[@krittaprottangkittikun7740]

This is so clear to me. Thank you for making this video!

[@faadi4536]

What an amazing explanation. I am taking a Machine Learning Course and he tried to explain the concept using Bandits but couldn't quite really grasp it in detail. I understood what we are trying to figure out but wasn't quite their yet. You have made it so much easier. Kudos to You Brother.

[@gabrieldart9943]

This is so cool! Thanks for your clear explanation.

[@DarkNinja-24]

Wow, great example and amazing explanation!

[@bilalbayrakdar7100]

Bro I completed my CS degree with your help and now I got accepted for master and you are still here to help. You are a true man, thx mate

[@shahnazmalik6553]

Your teaching method is highly appreciated. Please make lectures on statistics and machine learning algorithms

[@raphaeldayan]

Amazing explanation, very clear, thank you Sr

[@A.n.a.n.d.k.r.]

Awesome cool technique just got hooked to this

[@Dr.RegulaSrilakshmi]

U r just awesome ,any person who doesn't have any knowledge of Reinforcement learning can understand,Keep up the spirit...cheers

[@rikki146]

i cannot thank you enough for makin this excellent vid!

[@traiancoza5214]

Perfectly explained. Genius.

[@softerseltzer]

Love your videos, the quality just keeps going up! PS. the name of the slot machine is "One-armed bandit", because of the long arm-like lever that you pull to play.

[@irishryano]

....And the bandit bc it’s the WORST odds in every casino

[@spicytuna08]

i guess the slot machine is a bandit cause it keeps robbing money from the players.

[@llescarini]

Subscribed since few days, your videos are more than excellent! Amazing skill for teaching, thanks a lot.

[@ritvikmath]

Awesome, thank you!

[@aryankr]

Thank you for a great explanation!!

[@yongnaguo8772]

Thanks! Very good explanation!

[@warreninganji7881]

crystal clear explanation worth a subscription for more👌

[@georgiakouvoutsaki7877]

This is amazing !

[@velocfudarks8488]

Thanks a lot! Really good representation!

[@NoNTr1v1aL]

Amazing video!

[@nintishia]

Very clear explanation. Thanks for this video.

[@rutgervanbasten2159]

really nice job! thank you

[@amirnouripour5501]

Thanks a lot. Very insightful!

[@soundcollective2240]

Thanks, it was quite useful, heading to your Thompson Sampling video :)

[@abogadorobot6094]

WOW! That's was brilliant! Thank you!

[@seowmingwei9426]

Well explained! Thank you!

[@jams0101]

awesome video ! thanks so much

[@avadheshkumar1488]

excellent explanation!!! thanks

[@fridmamedov270]

Simple and accurate. That is it. Thanks!!!

[@anaydongre1226]

Thanks so much for explaining this in detail !!

[@ritvikmath]

You are so welcome!

[@spicytuna08]

we need to a person like you to democratize these important concepts cannot express how grateful i am to understand these important concepts which i have struggled in the past.

[@jc_777]

enough exploration for good youtube lecture on ml. i should keep exploit this guy. 0 regret guaranteed :)

[@jinpark9871]

Thanks, your work is really awesome.

[@ritvikmath]

Thank you too!

[@dr.nalinfonseka7072]

Excellent explanation!

[@welidbenchouche]

This is more than enough for me

[@alirezasamadi5804]

You explained so good

[@user-bt5il9zq8p]

Best example ever!!!

[@josemuarnapoleon]

Nice explanation!

[@dr.kingschultz]

You are very good! Please explore more this topic. Also include the code and explain it

[@SDKIM0211]

Love your videos. To understand the average regret value for exploitation, which extra material should we refer to? Why not 604?

[@hameddadgour]

I just realized that I need to explore more to maximize my happiness. Thank you Multi-Amed Bandit :)

[@nassehk]

I am new to your channel. You have a talent in teaching my friend. I enjoy your content a lot. Thanks.

[@ritvikmath]

Thanks!

[@sabanburaknazlm1381]

Well explained!

[@abdulrahmankerim2377]

Thanks!

[@shahulrahman2516]

Great video

[@TheMuser]

Nicely explained!

[@davidkopfer3259]

Very nice explanation, thanks!

[@ritvikmath]

Glad it was helpful!

[@lukekim8304]

I love this vid! It would be great if you could also do more videos on online learning and regret minimization 😆😆😆

[@jroseme]

This was a useful supplement to my read of Reinforcement Learning by Sutton & Barto. Thanks.

[@ritvikmath]

Glad it was helpful!

[@snehotoshbanerjee1938]

Best explanation!!

[@ritvikmath]

Glad you think so!

[@rifatamanna7895]

It was awesome technique 👍👍 thanks

[@ritvikmath]

thanks for your words!

[@tariqrashid6748]

Great explanation

[@maxencelaisne4141]

Thank you so much, I passed my exam thanks to your explanation :)

[@ritvikmath]

Glad it helped!

[@nastya831]

thanks man, this is truly helpful! 6 min at 2x and I got it all

[@ritvikmath]

Great to hear!

[@bassamry]

very clear and simple explaination!

[@ritvikmath]

Glad it was helpful!

[@TheMuser]

I have explored and finally decided that I am going to exploit you! *Subscribed*

[@francisliubin]

Thanks for the great explanation. What is the essential difference between contextual bandit (CB) problem vs multi-arm bandit (MB) problem? How does the difference impact the strategy?

[@zahrashekarchi6139]

Thanks a lot for this video! Just one thing I would like to find out here is where we store the result of our learning? like some policy or parameter to be updated?

[@hypebeastuchiha9229]

My exam is in 2 days and I'm so close to graduating with the highest grades. Thanks for your help!

[@sau002]

Brilliant

[@stanislavezhevski2877]

Great explanation, can you leave a link to the code, which you used in simulations ?

[@ritvikmath]

Thanks! I have a follow up video on Multi-Armed Bandit coming out next week and the code will be linked in the description of that video. Stay tuned!

[@Status_Bleach]

Thanks for the vid boss. How exactly did you calculate the average rewards for the Exploit Only and Epsilon-Greedy strategies though?

[@thinkanime1]

Really good video

[@ritvikmath]

Thanks!

[@calinobrocea7502]

Hello, thank you for the awesome explanation, it really helped me a lot. But I want to ask you one additional question on this topic. Do you know some method of tuning the epsilon parameter? I tried searching on google, but I did not find anything helpful. Thank you!

[@PhilipKirkbride]

Related to regret, we never really know the true distributions (since we can only infer from taking samples). Would you basically just use your estimated distributions at the end of the 300 days as the basis for calculating regret?

[@yitongchen75]

Cool explanation. Can you also talk about Upper Confidence Bound Algorithm relating to this?

[@ritvikmath]

Good timing! I have a video scheduled about UCB for Multi-Armed Bandit. It will come out in about a week :)

[@annahuo6694]

Great videos ! Thanks for your clarification. It's much clearer for me now. But I just wonder how you calculate the 330 regret in the case of exploitation only ?

[@ritvikmath]

Good question. You can get that number by considering all possible cases of visiting each restaurant on the first three days. Something like, consider the probability that of the first three days of visits, what is the probability that restaurant 1 is best, vs. probability restaurant 2 is best, etc. You can do this via pencil and paper but I'd recommend writing a simple computer simulation instead.

[@annahuo6694]

@@ritvikmath Thank you for this prompt response. I think I get the idea from the epsilon greedy formula (option number 3 in the example). Thank you a lot, your video is really helpful :)

[@debashishbhattacharjee1112]

Hello Ritvik This was a very helpful video. You have explained a concept so simply. Hope you continue making such informative videos. Best wishes.

[@ritvikmath]

Thanks so much!

[@victorkreitton2268]

What ML books do you recommend or use?

[@jonathanarias2729]

Why 330 is the response in the explotation example? Should t be; 3000-2396=604??

[@TheFobJang]

Would you say exploit only strategy is the same as the eplore-then-commit strategy (also know as explore-then-exploit)?

[@bobo0612]

Hi! Thank you for your video. I have a question at 6:28. Why the roh is not simply 3000 - 2396?

[@senyksia]

2396 was the happiness for that specific case, where restaurant #2 was chosen to exploit. 330 is the (approximate) average regret for every case. So 3000 - 2396 would be correct if you were only talking about that unique case.

[@myoobies]

@@senyksia Hey, what do you mean by average regret for every case? I'm still having trouble wrapping my head around this step. Thanks!

[@madmax2442]

@Bolin WU I know it's 8 months already but I wanted to know whether you got the answer or not. I also have the same doubt.

[@wenzhang5879]

Could you explain the difference between the MAB problem and the ranking and selection problem? Thanks

[@l2edz]

Coolest prof ever! 😎

[@ritvikmath]

haha!

[@quanghoang3801]

Thanks! I really wish the RLBook authors could explain the k-armed bandit problem as clearly as you do, their writing is really confusing.

[@sampadmohanty8573]

I knew everything from the start. Ate at the same place for 299 days and got pretty bored. So watched youtube and found this video. Now I am stuck at this same restaurant on the 300th day to minimize my regret. Such a paradox. Just kidding. Amazing explanation and example.

[@yannelfersi3510]

can you share the calculation for the regret in case of exploitation only?

[@wenlouismao]

Any books on this topic you’d recommend? How do you come across this topic?

[@jimbocho660]

Try 'Bandit Algorithms for Website Optimization: Developing, Deploying, and Debugging' by John Myles White. It's short and sweet.

[@Akshay-fk8bu]

❤

[@shantanurouth6383]

I could not understand how it turned out to be 330, could you explain please?

[@manabsaha5336]

Sir, video on softmax approach.

[@Phil-oy2mr]

In the exploit only case, would there be a way to compute the regret mathematically without a simulation?

[@softerseltzer]

You could calculate the probability of picking one restaurant over the others and then sum over the expected rewards weighed with the aforementioned probabilities. So for example if one of the restaurants is clearly much better, you will most likely pick it in the initial one-shot exploration phase so it's probability will be close to 1. The probability of picking one restaurant over another could perhaps be derived using cumulative distribution functions of the initial reward distributions. One could imagine a simple example with discrete instead of continuous distributions. Say with any restaurant having only three options: a certain probability for a bad meal (reward 1), a mediocre meal (reward 2) and a good meal (reward 3).

[@user-sl6gn1ss8p]

*insert taleb nassim talking about the possibility that a meal kills you or changes you whole life* or something like that : p

[@Trucmuch]

Slot machines were not called bandit but one-arm bandit (they "stole" your money and the bulky box with one lever on its side kind of looked like a one-arm man. So the name of this problem is kind of a pun, a slot machine with more than one levers you can pull (here three) is a multi-armed bandit. ;-)

[@ritvikmath]

Wow I did not know that, thanks !!

[@sunIess]

Assuming a finite horizon (known beforehand), aren't you (in expectation) better off doing all the exploration before starting to exploit?

[@ritvikmath]

You've just made a very good point. One strategy I did not note is an epsilon-greedy strategy where the probability of explore in the beginning is very high and then it goes to 0 over time. This would likely be a good idea.

[@chyldstudios]

Why didn't you discuss the best strategy ... Bayesian Bandits with uniform priors coming from a Beta distribution

[@EugeneBos]

What if we will go in the worst restaurant 299 days and then we go to the best one? I think it is possible to get very high happiness(close to 3k for 1 visit) but regret may be as well high...

[@EugeneBos]

😂😂😂

[@shraddhashah1284]

In epsilon greedy method how comes 2907 kindly tell me calculate on priority basis.

[@geraldwu5753]

With only 3 options, wouldn't exploration naturally become exploitation?