Google Product Manager Execution Interview: YouTube Watch Time Root Cause Analysis

[@tryexponent]

Don't leave your product management career to chance. Sign up for Exponent's PM interview course today: bit.ly/3iT2D5u

[@umg4147]

My key take aways.. If faced with a new launch situation - 1. Think about what kind of feature was it (UX,UI, Backend,Algo) 2. Discuss and clarify launch criteria - what were the metrics that were decided 3. Discuss the tradeoffs that were decided with the launch Keep in mind the User Attention Span with UX(Social Media) platforms Also, ask how are the other related metrics trending. When it is a downtrend metric question - Breakdown the metric to - Geography, Technology, Time, Trend, Source, Category

[@manishkumar-qn6lx]

Hey would you please shed some more light on comment "Downtrend metric question".

[@abhi2010cse]

Her communication skill and clarity of thoughts are spots on. She never trembled. Very calm and confident approach.👏

[@NdubisiOnuora]

This is an excellent response, but it could have been made more concise. Bringing what she described in the beginning to the end made it seem like an awesome chess move. Definitely, she is an intuitive most likely an INTP. If I were a tired interviewer, I would zone out during the interview, because she took too long to get to the point.

[@anuradhatempe9033]

I wish I get to this level in terms of deep diving and confidence during my interviews, this was a great one to watch. Thanks for posting

[@sony.andreass]

definitely true

[@pranav8358]

so relatable.

[@amitkupadhyay]

One of the best and confident responses to the RCA kind of questions. Loved the way she started with pre-cursors to this launch rather than diving straight into the usual framework of asking clarifying questions. Always good to get the bigger picture first.

[@tryexponent]

Hey Amit! Glad you found this video useful! Watching others approach questions is a great way to expose yourself to different approaches or ways of thinking. Hope you'll find our other videos just as useful!

[@ThatTeluguGirl]

Have a Google interview in 4 hours. Wish me luck!

[@2210bubble]

All the best Ramya!

[@sumanto_pal]

Let us know here how it went

[@ThatTeluguGirl]

Went well! 2 down. 2 to go! Wish me luck! :)

[@sid4752]

Are you based in the Bay Area?

[@ThatTeluguGirl]

@@sid4752 No, I’m based out of India. Update: Done with final round this week. Up to the hiring committee now 🤞

[@Theartsygalslays]

Excellent trade-off and goal analysis! One thing that I didn't hear the analysis hit on as deeply till the very end, and which would be a base hypothesis from my perspective, is around comment QUALITY -- she did surface the concern about spam, but that isn't an only concern with comment integrity. One reason some videos have comments turned off completely is exactly this concern with quality: bullying in comments, trolling, etc. (none technically spam per se). Even valid comments can be dissuading: it's possible that users scroll through the comment content before even deciding to engage with the video, and they might leave / click away if discouraged (though again, we should be able to see some scroll activity through the comments, in that case, and the click path -- e.g., whether they click back to the search results or to one of the recommendations from there). We could possibly look at whether there's additional reduction in other activity: for example, are viewers doing less than before initial playback (first 10 s), skips forward, etc., because they already got what they needed out of comments and the video description?

[@proteinshake7731]

exactly, whole interview was just fixated on a single hypothesis that its a design flaw.

[@rajathslr]

Shes a PRO !

[@Theatriics]

Great video, I loved watching her go super in depth. I'd love to see her do a mock interview again for a non-Google product though, as I'm not sure how normal this would be for an interviewee with no prior working knowledge of the product they're interviewing for.

[@Theartsygalslays]

There is nothing particularly special about the metrics that she presented that a reasonably experienced PM could not deduce from perusing the product. :)

[@adorablecheetah2930]

@@Theartsygalslays You are right, just that its a good learning experience for younger PMs

[@aok1425]

@@Theartsygalslays I do think though that her familiarity with KZbin helped her come up with product ideas to mitigate the watch time decrease, like reducing the screen space for comments, changing the ranking algorithm, and checking for spam & abuse.

[@looppp]

I am impressed by how still the interviewer can be. I thought the screen froze for a second lol.

[@ShubhamGupta11]

🤣

[@niveditya]

Is it just me or did anyone feel that the conversation was very one sided ? I felt that there weren't enough check ins with the interviewer.

[@waattzup]

It's not one sided,she has suggested certain hypothesis which can be validated to deduce the root cause. This is how a RCA should be done ✅

[@rickhouten1622]

The title is misleading. This is not a RCA since the cause is already stated. “You shipped KZbin comments on mobile”. So this is a trade off question and next steps

[@DhruvGhulatiG]

I thought this might have been improved at the start. She tries to explain pre-launch criteria, saying "it depends on what the % metrics were on comments engagement vs. watch time, and we would just decide on that", I would have maybe done this framework: - Clarify the feature that was launched and what it looks like e.g. the UI - Understand the metrics and how they are calculated - What was the exact metric numbers and averages, any idea? - Clarify if the metric changes were global, on specific versions on mobile, specific geos, demographics etc Then, see why this might have happened: - External - did something external impact the launch? e.g. competitors launching? Is this across the entire KZbin e.g. watches going down overall not just due to the launch? Did we change the make-up of the videos to be shorter? Were there some other platforms that people find KZbin videos e.g. Facebook or Twitter impacted? - Internal - were there other products in the launch that launched in same time that caused watch time? Did the feature only launch on short videos by accident due to a bug? Did people engage with the comments more than the videos so naturally watch time went down? Did we change the UI in some way that cannibalised focus on the videos watched? Was engagement on comments more on short videos than long videos which affected the mix? - Out of control - was there something that caused the watch time to be decreased?

[@tryexponent]

Woah Dhruv! Thanks for engaging with the content and taking the time to share your improvements! 💪

[@nehadhr2]

Very good response from her. Got an insight on how actually product managers think on a daily basis when making a go/no-go decision or when diagnosing some issue. However, I felt that this interview could have been more interactive instead of her laying forward all scenarios and then making her own assumptions.

[@AakashMehta]

Great answer. Could have been even better if she had structured all of this on a Google doc while sharing her screen.. that would also force her to slow down and not have the interviewer lose focus

[@justinching9317]

I felt like she definitely rambled and could have been more concise

[@umangjain3592]

Some of the key takeaways for me atleast are: Consideration of KZbin Layout across both Web and Mobile App. Consideration of language semantics across countries in the comments section. She pulled this execution nicely. However, these days we see Ads just below the comments and then the recommended videos. But overall a great mock for an aspiring PM like me. Thank you Exponent ❤

[@jiahangli1906]

Very strong interviewee. I think she might can stop when she asks questions about geographic regional decline, gradual decline etc instead of making assumptions

[@NdubisiOnuora]

This is the best interview from Exponent.

[@sawmilljoseph]

I really liked how the interviewee started with questions, looked at evidence, then hypothesized a potential cause.

[@jessecuster8285]

Is this guy even a human? He hasn't blinked for like half an hour!!

[@varunsagar7522]

😀😂

[@timerwentoff]

Great opening on how she planned to address this concern....

[@namokar2001]

fab mock interview. she seems to be very learned and also seems to have a super nice work experience in product management. kudos to her.

[@ashishbarick]

gosh the interviewer is so mortifying! the poker face can get anyone nervous. great video btw.

[@CollectConnectDots]

This is one of the best interviews on metrics questions. The interviewee clearly demonstrates that the deep knowledge she has on each part of the KZbin features.

[@hamzakhalid8668]

bruh the guy be looking into our souls

[@iluvcakes19]

😂😂😂

[@vitrioling]

completely wowed by her clarity

[@arun553]

She's assuming that this dip in comments is happening at the launch of the comments feature, but i could easily see a case where comments was successfully launched w/ stable watch times in the past but in the present, watch time took a dive for some unrelated reason (not because of comments)

[@AbstractAbsorption]

That was not the question though.

[@poojasikdar3583]

I loved the insightful questions and answers she came up with during the discussion.

[@Freshmarketers]

10:18 Kevin watching like, he is in love. Great video, love the flow of the conversation 😃

[@nehakachewar4201]

Really appreciate her analytical skills

[@moonshine2468]

Great interview - I would thought that to begin with, i may clarify the reason why we need to ensure viewership while the comment starts to grow. In another word, what would be a healthy matrix of comment vs. viewing time , and what is the busienss reason behind it ( e.g. ads income)

[@japanboy31415]

this is pretty good. however, i feel that she doesnt really engage with her interviewer as much. might just be a personal thing. but it just seems like she is just rambling a lot.

[@rohittaneja091]

I would have also considered mentioning if ads are the possible cause but overall this was great interview 👍

[@HimanshuAgarwal101]

Maybe I'm missing something but didn't she just change the question in first 6 mins?

[@Theartsygalslays]

She merely listed her assumptions and thought process to narrow down the problem, create some context and triangulate possible root causes - just like in the real world where a high level question needs to be scrutinised. Not shocking for a PM’s role in this.

[@khushbookalyani2353]

Thank you for theses videos. They are really helpful. The interviewee took a well thought through approach to resolve the challenge

[@vcconnector605]

Here is my feedback. She used to be PM on KZbin. So she is familiar about the product and her interview should have been about other products like airline products something she is not familiar with.

[@Theartsygalslays]

Many companies, including Google and Facebook, ask questions about their own products, so this is a very typical prep video. Her being at Google ads credibility.

[@coolwaterz]

@@Theartsygalslays a ton of companies outside of FAANG ask case studies like this. So no, this is not a very helpful video to anyone who is objectively looking for a realistic answer. Companies do not cater to the product you're interviewing for. So everyone saying Oooo and Wowwww to an answer like this are kidding themselves when it comes time for their case study about a product they have rarely used. I dont understand the responses in most of these comments sections as though PMs are magically experts in every single application in the tech marketplace and should be expected to provide this level of insight.

[@videologs7278]

What PM level get this type of question? PM1 or 2, Senior PM, Group PM?

[@swahareddy8822]

In all the mock PM interviews, the candidate keeps asking questions which is great. But the interviewer's answer is always so predictable. "For now let's just take "/"Sure, that sounds good" If the interviewer would provide more meaningful input, it would be more encouraging to engage with them. Now it just sounds like something everyone has to do (to tick a box) that is not really adding value.

[@shivalishinde4327]

Random answer!! Should have asked clarifying questions instead of just saying own thoughts!

[@ramjiintrouble]

Not a PM but that was a very good analysis ! Just gives me more breadth

[@gashdashgash]

Ridiculously good articulation! And yes, internationalisation is a horrible word.

[@stevenspellberg]

Loved her approach

[@swahareddy8822]

Surprising that ranking of comments didn't come up (likes, user replies, creator replies, aggregate..). Could have brushed upon A/B testing for these. Especially when so much of the interview was about recommendations of Next Up videos

[@japanboy31415]

she gave good background. but i feel that she over indexed her time on that and did not dive quickly in on execution on an issue.

[@pranav8358]

Correct me if i'm wrong... So we don't actually have to reach to a root cause for a root cause analysis. Probable reasons are good enough.

[@Theartsygalslays]

It's a diagnostic exercise: you keep going and ruling out possible causes, and discussing how you might do that (e.g., looking at logs, looking at other metrics, etc.).

[@ShubhamGupta11]

The scoping question was actually a scoping speech

[@swahareddy8822]

Recommendations being generated but not being shown on screen. KZbin servers down which led to drop in viewership. Aren't these too far flung reasons to be discussing? You'd assume that Google would test new features before launch (recc not visible) and that you and the whole world would know about recent downtimes for Google.

[@wazooman123456]

this interview was not good. she just started listing out potential issues rather than talk with the interviewer and actually find the root cause. sure, the analytical skills are there but she didn't really get to the answer as you would be expected to do in such RCA interviews..

[@bitjuanx]

Amazing interview. Perfect.

[@sugandhadogra1781]

Just to add , maybe users are getting irrate due to multiple ads being displayed n dy get less engaged.

[@zareensubahaziz512]

One of the best interviews

[@ShubhamGupta11]

Loving the youtube comments all of a sudden

[@poojasinha5868]

Wouldn't the experiment setup be - 1. Control group - Already has comments rolled out to these users and with a decreased watch time 2. Treatment group 1 - 1 recommendation video above the comments section 3. Treatment group 2 - 3 recommendation video above the comments section 4. Treatment group 3 - User education prompts about recommendations below the fold 5. Treatment group 4 - A different tabbed layout that makes it easier for users to navigate between comments and recommended videos We can try different solutions, but the important point here is that the experimental setup would be against the current truth of the system which is where comments has already been shipped and the treatment groups should show an 'increase' in the watch time.

[@tarun8172]

This dude doesn't smile.

[@tommynguyen6399]

Just like most of interviewers :)

[@omaralmunawer]

Stop finding negative things about the host and the guest and appreciate what they are sharing for you to get value for free!

[@nitish-sh3iq]

@@omaralmunawer yes omar u r right 👍

[@jessecuster8285]

I feel like youtube actually considered here idea at 14:31. She mentioned to mention 1 comment and then showing recommended videos, which is the current implementation of the KZbin mobile.

[@tommynguyen6399]

What are you saying? She's THE PM at KZbin. Maybe she was already leading the team building it at the time :D

[@looppp]

Man, RCA is where my interviewing is the weakest in.

[@RaviKant-mw4ew]

Pre roll and post roll ads is another reason, introduction lot of ads

[@Theartsygalslays]

That would not be mobile specific

[@alexfriedman2175]

Cherrie is really good.

[@bedantamadhabgogoi]

What is a backstop metric?

[@Theartsygalslays]

A counter metric that you should be watching.

[@Theartsygalslays]

@@bedantamadhabgogoi sometimes also called guardrail metrics.

[@silv3rArrow]

8:15 hitting the nail right there. I wonder after all this time how google is managing this issue.

[@janmeppe]

Does she have a background in consulting? Very hypothesis-driven solution

[@vishnupanchal6215]

She is next level

[@mehulkapadia2029]

Got it!

[@lexixu7220]

OMG SHE IS GOOD

[@rhuitt]

My key take away is: I love her

[@umg4147]

Can't stop myself from asking this, is she reading out of a script?

[@Theartsygalslays]

To me it felt organic. But nothing wrong with having an outline up in front of you while interviewing, as well. It’s encouraged to structure your thoughts to avoid rambling and getting off tangents.

[@PrateekTade]

I just started watching this one and NGL it seems so because of her pauses and tone.

[@evonnechen7079]

She is amaing

[@peeyushmittal6368]

The interviewer is so still and not even blinking properly. I suspect there is an AI tool at play

[@shivalishinde4327]

Just because someone has confidence doesn't mean it is the right approach!

[@SahillKhn]

bro is staring inside me

[@ishanagashe2887]

the interviewe herself immersive like the interviewer,didnt like the approach taken at all

[@sudeepchadha]

looks like shes reading from something..

[@sudeepamitra]

Sweet

[@ShubhamGupta11]

Sweet at 1:45 was sweet xD

[@DrFunkBlast]

She gave an excellent response in spite of a terrible interviewer. Especially that last question about which testing arm to select was just terrible. She's a rockstar!

[@tommynguyen6399]

Why was the interview terrible?

[@debayandas7343]

Bla bla bla

[@mamayeye306]

I don't think it's very good.

[@mamayeye306]

Especially her answer to the A/B test, which shows her little understanding of it.

[@Theartsygalslays]

Can you provide some more constructive criticism on what you thought was wrong with it?

[@preetsarangmanifestationch5982]

I never seen anybody giving this much good PM interview

[@nikhilsingh3928]

Excellent Brainstorming!