How to Design and Build a Recommendation System Pipeline in Python (Jill Cates)

Want to know how Spotify, Amazon, and Netflix generate recommendations for their users? This talk walks through the steps involved in building a recommendation pipeline, from data cleaning, hyperparameter tuning, model training and evaluation.
Personalized recommendation systems play an integral role in e-commerce platforms, with the goal of driving user engagement. While there is extensive literature on the theory behind recommendation systems, there is limited material that describes the underlying infrastructure of a recommendation system pipeline. In this talk, we will walk through the process of designing and building a recommendation system pipeline. We will specifically discuss techniques for data cleaning and normalization, hyperparameter tuning, model training and fitting, as well as quantitative and qualitative model evaluation. By the end of this talk, you will learn how to design your own recommendation system pipeline from scratch.
About the Author
Jill is a data scientist at BioSymetrics, where she applies machine learning techniques to biomedical data. Outside of work, Jill is working on an open-source toolkit for implicit feedback recommendation systems. She is a member of PyLadies and Women Who Code.
Presentation page -- 2018.pycon.ca/fr/talks/talk-P...
Author website -- topspinj.github.io/

[@88darnell]

I’ve watched many recommendation engine videos and this is by far the best I’ve seen! Fantastic expertise and thought leadership.

[@tony0731]

It's so beautiful how you include those content in merely 20 mins! Well explained!

[@jehan60188]

amazing amount of content in just 20 minutes! Also, thanks for covering train/test split- not everyone covers that with collaborative filtering.

[@Atlas-ck9vm]

You have made the different concepts really clear. Thank you.

[@ishfaq85]

Thank you so much for short and very informtive lecture. It helped me a lot to start my project on recommender system. :)

[@barbarasilveirafraga5692]

Excellent! Congratulation for your presentation!

[@HazemAzim]

Very nice and smooth introduction .. Thank you .. I hoped for a python code implementation as well

[@nehabalani7290]

very crisp but makes the point.. thanks

[@user-Sjskakendjsiwjd]

Nice presentation. In recommendation system, how do you define the relevancy for model evaluation hyperparameter tuning? Furthermore, how can you do this offline more accurately?

[@marekr.9339]

Great introduction!

[@lucianoinso]

Great talk for a general overview on recommendation systems! From there I could deepen in the subjects I found interesting or didn't know about, in my opinion it's a great video for people with a general knowledge of ML or maybe that have some knowledge in other applications but never touched Recommendation Systems. Just one thing that doesn't come clear to me at the pre-processing part: When she talks about normalization, she talks about applying mean normalization for the users ratings, which comes clear, but the slides show a formula with "user-item rating bias" which she skips explaining, can someone explain me on where does the formula come from and if it's something that you should need to subtract from every cell? The fact that there is a variable for "global average" and another for "item's average rating" kinda confuses me, does the global average regards the whole dataset of movies? Thanks!

[@BiranchiNarayanNayak]

Excellent talk on Recommender System

[@mirsahib596]

what is a good value for sparsity

[@jaggyjut]

How to deploy the model in a cloud platform and then consume in front end app like react. Thanks

[@emcemimotionandcontrol5554]

Excellent presentation. Thanks

[@buddhikas]

Nice talk !

[@charlesgormley9075]

Incredible presentation! I semi-disagree with precision and recall being good evaluation metrics for a recommendation system using a masking technique to evaluate model performance during the offline training phase. This is due to them demanding the output of the model to be binary, where as masked-prediction in this case would represent more of a regression problem leading RMSE to be a more valuable evaluation technique. Great presentation though, very clear explanations.

[@heena3553]

How do u make predictions bcz in knn for predictions we need train or test data by splitting but here we r using different approach for this so how gonna we make predictions for ds?

[@denniszenanywhere]

Any software I can use instead of building my own?

[@MagnusAnand]

Very clear. Thanks

[@appyviral8753]

How much u charge for making a video recommendation system for Android app?

[@intoeleven]

how do you update your k-latent factor matrix after a new user arrived? do you have to re-multiply the whole user-item matrix again?

[@NotfattyElaine]

I think we just need to optimize for new users only. stats.stackexchange.com/questions/320962/matrix-factorization-in-recommender-systems-adding-a-new-user

[@thankyouthankyou1172]

a good one, thanks

[@TauvicRitter]

Suddenly matrix factorisation comes up. Why? What are its benefits and limitations. Ok i never studied this but it looks to me that im very dumb or the speaker jumps over a lot of issues.

[@TusharDeb]

Hmm.. Jill Cates, sounds very much like Bill Gates.

[@joeljoseph26]

There is nothing new to learn from this presentation. Don’t waste time!

[@ronitsadhukhan7341]

Thank you, but for someone new this is very helpful, so maybe not a waste of time.

[@MrJoeDone]

nothing new for whom? Maybe not for you, but maybe then you are not the target audience. Then just don't bother to watch the Video and take your business elsewhere

[@joeljoseph26]

There are better videos out there who teach the same so don’t waste your time on this video. As simple as that. AutoML will kill data science jobs so I decided to move on to another technology. Personally , I would suggest learn blockchain because it’s plays a huge role in security and also ensuring any data exist inside a blockchain is tamper proof. ML + Blockchain = 💯💸🤑

[@MrJoeDone]

@@joeljoseph26 you should not waste your time writing this stuff, which has nothing to do with this video and others shouldn't waste their time reading it. Don't you waste anymore of other peoples time :)

[@simkort5799]

what an arrogant rude person. Nothing new to you does not mean that no one else would find this useful. This is a very good helicopter view for someone who is new to build recommender system using collaborative filterings. If you REALLY value your time that much, you should probably not write this time-wasting comment. So yeah, you are just bitter and not happy with your own life, and leasing out on an innocent target.