Evaluation Measures for Search and Recommender Systems

[@aminghaderi1902]

Probably best explanation out there.

[@goelnikhils]

Hi James, I have a question on NDCG or any other ranking aware metrics. How does these metrics work where you have millions of products/items. What I mean is if we have millions of items, then it means we have to first label (manually) all the million items for relevance /rank. And then when our model predicts we use NDCG. Isn't this a big drawback of NDCG. Can you please suggest what is better approach to rank if we don't have relevance labeled data. Thanks in

[@preetimehta1247]

Hi , I have a query If I am working on a song recommendation project by using Spotify API data set, I have used models like cosine similarity, matrix factorization, knn , Latent Semantic Analysis (LSA) model, Correlation Distance method. Now I am confused about how should I approach for evaluation metric in this system.

[@goelnikhils]

Amazing Explanation. So clear. Very helpful

[@shrar837]

Your videos are impressive and very informative mate. 👌

[@jamesbriggs]

thanks!

[@anujlahoty8022]

What a video, hats off!

[@parsakhavarinejad]

Clearly explained. Thank you

[@Han-ve8uh]

1. I got confused at 18:29 when predicted is a nicely increasing sequence making me think are those ranks or item ids. I was also thinking whether the len of intersection act_set & pred_set could simply be len(act_set), then i realized this example here is a very special case where act_set is subset of pred_set. If act_set contains value 9, then we can't use len(act_set) alone and the formula in video is required. 2. Similar to question nikhil goel asked in comments section 2 weeks before this, where does 13:46 actual_relevant data come from? It looks manually labelled, and this labelling occurs per query making it super unscalable?. 3. Assuming we accept manual labelling how is the 0-4 range determined? I feel like drift is a problem, when todays 4 becomes tomorrows' 3 as value judgements change, does this mean relabelling all results again? 4. I noticed some metrics aggregate across queries and k, and some are only within 1 query across k, in what scenarios do we use each? 5. I didn't expect a *relk in AP@K formula, why do we ignore certain precision at certain k? Feels like artificially increasing metrics for the sake of it, which becomes ineffective if every query does it

[@sriks4003]

Very helpful, thank you!

[@sumantjha8392]

Super informative and great..thanks

[@miguelfsousa]

This video is great.

[@morannechushtan2101]

21:23 Statistically there is probably a cat in the box on image 3

[@Data_scientist_t3rmi]

IN MRR, when our search result doesnt inclued the result that we want, for your example if we want to search for cats and we find only dogs, how can we calculate MRR ? can we give it a big number for exemple rank 20 for all Not included results? 1/20

[@jamesbriggs]

yes as you said - or use another metric that better fits to your scenario

[@Data_scientist_t3rmi]

@@jamesbriggs Thank you for your answer

[@vishalwaghmare3130]

Very helpful ❣️

[@HazemAzim]

Super nice .. Thanks

[@Data_scientist_t3rmi]

Good video !

[@tarikkarakas587]

Biggest problem is labeling the product whether it is relevant or not. It is not possible to label each search. Meanless if you can't handle with that.

[@jamesbriggs]

Yeah data prep as usual with ML is the hard part, if you're interested in evaluation methods for IR *without* labeled data look into online metrics for eval (and training)

[@joyeetamallik5063]

Hi James! can u make some vedios of updating Models if we Keep on getting data(e.g Biweekly)

[@jamesbriggs]

cool idea! I'll add to the list :)

[@mattygrows7667]

love your videos but why do you always seem so sad

[@jamesbriggs]

thanks! idk I'm happy I promise lol