Design an ML Recommendation Engine | System Design

  Рет қаралды 17,388

Interview Pen

Interview Pen

Күн бұрын

Пікірлер: 20
@shreys3
@shreys3 8 ай бұрын
These are incredibly well made and well articulated videos. Thank you!
@interviewpen
@interviewpen 8 ай бұрын
Thanks!
@beautifulveneer
@beautifulveneer 3 ай бұрын
In the beginning the model output is the probability that a given post will be liked by a given user. One thing missing is how the worker jobs processing the new post queue scale to millions of users (or followers if only applying to posts by users that are followed) and what is cached. Another thing is resiliency around post to queue and write to database. A lot of dbs stream data so that is one potential solution. Good job though.
@estebanmurcia8451
@estebanmurcia8451 3 ай бұрын
7:49 i have a question here, if the inference server is distributed, would the cache still be a single "instance"? if not, how would the api know from what server should it get the corresponding data for a specific user?
@beautifulveneer
@beautifulveneer 3 ай бұрын
The inference service would be stateless and use a scalable distributed cache such as redis.
@interviewpen
@interviewpen 2 ай бұрын
A caching service such as Redis is shared, so data can be distributed across instances. We have a whole video on sharding at interviewpen.com :)
@alirezakhorami
@alirezakhorami 8 ай бұрын
Amazing job, I love your videos about data, keep rocking
@interviewpen
@interviewpen 8 ай бұрын
Thank you!
@chrisogonas
@chrisogonas 3 ай бұрын
That was incredibly useful. Thanks
@interviewpen
@interviewpen 3 ай бұрын
Glad it helped :)
@chrisogonas
@chrisogonas 3 ай бұрын
@@interviewpen 👍
@RiwenX
@RiwenX 8 ай бұрын
So you have to retrain the model for each new user?
@JohnSmith-op7ls
@JohnSmith-op7ls 8 ай бұрын
You can train a model to simply correlate various demographic attributes to liking certain things, then it can guess based on new user demographics. You can get more elaborate and first try to find if there are strong correlations between cohorts of certain baskets of demographics, then use that, alone or in combination with individual demographic attributes. Basically you have to run the numbers to see what produces the best recommendations. You wouldn’t have nearly enough data on new users to be statistically relevant in a data set large enough to itself be statistically relevant. Over time you’d want to,retrain the model as enough new data on new and existing users is added. Otherwise the model’s accuracy can drift.
@interviewpen
@interviewpen 8 ай бұрын
Not necessarily--the model can still make guesses based on the behaviors of other users along with whatever data you feed it during inference. However updating the model with a user's past behaviors will increase accuracy. Hope that helps!
@motbus3
@motbus3 8 ай бұрын
I see you used Airflow for generic approach, but Argo has easier integration with MLFlow
@interviewpen
@interviewpen 8 ай бұрын
Good to know, thanks!
@tanishkmahakalkar761
@tanishkmahakalkar761 8 ай бұрын
First!💯🔥
@semyaza555
@semyaza555 8 ай бұрын
SECOND 😈
@pieter5466
@pieter5466 8 ай бұрын
Last
@ALWALEEDALWABEL
@ALWALEEDALWABEL 8 ай бұрын
Why do you hide their names? Why is the teacher's name not put on the video? Is there something you are ashamed of?
Design a Data Warehouse | System Design
14:08
Interview Pen
Рет қаралды 31 М.
Design a Simple Authentication System | System Design Interview Prep
17:22
IL'HAN - Qalqam | Official Music Video
03:17
Ilhan Ihsanov
Рет қаралды 700 М.
So Cute 🥰 who is better?
00:15
dednahype
Рет қаралды 19 МЛН
Systems Design in an Hour
1:11:00
Jordan has no life
Рет қаралды 34 М.
When to Use Kafka or RabbitMQ | System Design
8:16
Interview Pen
Рет қаралды 149 М.
GraphRAG: The Marriage of Knowledge Graphs and RAG: Emil Eifrem
19:15
Scalability Simply Explained in 10 Minutes
9:20
ByteByteGo
Рет қаралды 51 М.
Elasticsearch Deep Dive w/ a Ex-Meta Senior Manager
44:03
Hello Interview - SWE Interview Preparation
Рет қаралды 31 М.
IL'HAN - Qalqam | Official Music Video
03:17
Ilhan Ihsanov
Рет қаралды 700 М.