PM Mock Interview (with Director of Product): Measure Success for Facebook Events [Execution]

[@tryexponent]

Don't leave your product management career to chance. Sign up for Exponent's PM interview course today: bit.ly/3L51F4k

[@niveditya]

Really liked the follow-up question around what the interviewee would do if they found their engineers launching a bunch of features that encourages smaller events. To address this, I think we should include a guardrail/counter metric on "Avg number of RSVPs per event". Also, I would challenge using RSVP rate (invitees who RSVP/Total Invitees) in the NSM. Many of these events on FB are organically discovered and not through invite. A better metric might be (RSVP/impressions or total number of users who viewed the event).

[@arnie411781]

Few comments: 1. I do not think competitor analysis was required for this question though I liked how he explained the competitive moat FB has over competitors by virtue its community on which the events are getting formed. 2. I did not understand the answers to the counter questions. Why would the interviewer need to do a test to understand the efficacy of higher v/s lower invitee events? 3. When asked how would be go about figuring out the RSVP threshold, he mentioned about going by category. I think that missed the point. I think we have to understand why is high RSVP important. Because it's a proxy for the success of the event and hence will drive repeat event creation. So, I would look into the data for correlation between higher RSVP v/s repeat event creation and draw the threshold cutoff accordingly.

[@JG-dy2hk]

Those are some good points you made. You need to do a mock interview for product sense and execution. Since Exponent dropped the ball on contacting you contact David Bhatt at Product Management HQ and setup a mock interview

[@HeresJonnie]

I agree. I think the Product Alliance approach I saw was much better than the employed here.

[@arunthomas]

Very well done. A follow-up question could be on specific engagement metrics related to events and how to establish a correlation/causation between engagement and event participation.

[@leisimin]

Your north star metrics should not be prone to misinterpretation/bias in your team, period. You can't 'experiment' on metrics to see if they should stay or go, in fact metrics themselves serve to inform if experiments work or not. I think he should have taken the cue to change the north star metric to something like 'total RSVPs in time period' so that solves the bias issue while aligning with overall objective

[@aok1425]

I think an even better metric than 'total RSVPs in time period' is '# events w/RSVP rate >5%'. It all depends on what the data science team finds out, but one way to avoid perversely incentivizing the team to build features for smaller events that have higher RSVP rate is to call even a low-ish RSVP rate a successful event. My thinking is that for events overall, it might be more impactful to get events from 5% RSVP rate. Then, for a subset of events, it would be impactful to get them to help get them to a higher RSVP rate. But if the DS team (or DS + UX Research) find out that many event creators are happy w/10% RSVP rate, then no need to set it as high as 60%.

[@mrwazo]

Yeah and why 60% surely that's ambiguous

[@BhavyaBansal-k2f]

Great mock! watched this right before my screening and took a couple of points into the interview. Helped me quiet a lot!

[@אחוקים-ח5ס]

great follow-up question on what you would do if the the eng team is now promoting smaller events that have by definition a higher attendance rate. 2 other solutions: 1) Segment that event by different types. Open to all friends, open to big communities (based on activity), events for smaller communities. 2) Even easier solution - just instead of using RSVP % as a/b use log(a)/log(b). Sometimes the magic is in the math :))

[@Claudius025]

can you explain why log(a)/log(b)

[@aok1425]

I was thinking that Vipul's revised metric, with a different threshold for each event segment, is too complicated. I can see a future where usage of the product changes, and those thresholds become out of date. A solution in my head is to set the threshold at the lower end of the thresholds from the different event segments (say like 5%). This would make the metric simpler, avoid the perverse incentive for Eng, and make the team focus on helping the

[@ashwinkumar675]

Amazing counter question between 17:00 - 18:00. Love the video! As a follow up, I'd love to know why Vipul rated this an "OK" interview. I thought it was great overall. Anything in particular he felt he missed?

[@experiencestreets]

The candidate repeated his points many times over throughout the discussion. Most clarifying questions were unnecessary. Interviewer didn't answer them anyway and asked him to take a call. He talked about competitors but didn't use those insights for defining the success metrics. He didn't really answer the follow up question about his approach to quantify 'high rsvp rate'. Having said all that, it seems to me that mock is setup to provide clear pointers on the framework for such questions, which could be useful to some.

[@saketpandey8897]

someone had to say this

[@IamMC939]

Good interview and thanks Vipul for logical way of going through the analysis and the flow. For NSM I would have picked the number of creators with events that have a high RSVP/attendee interest. Ultimately, FB as a platform should have a healthy number of successful creators, not just a few creators; Also, if there are only a few creators, there's a slippery slope of only niche events being created - which is not a healthy eco system for facebook over the long term. Again, as the interviewer asked the RSVP rate may depend on the type of event. Events could be segmented and ultimately FB needs to have both big and small events to align with its mission of bringing people closer together

[@tryexponent]

Hey IamMC939, thank you for the kind words and for sharing your insights!

[@HUSSEINHUSSEINAHMED]

Just like consultants, "they take your watch and tell you what time it is" - come on measuring events success is simply by how many people rsvpd assuming that was the outcome management wanted! didn't need 30 minutes to say that nor a $200K salary to say that.

[@jasper5016]

Well said, its all bullshit. When I attend directors meeting and I see what people talk about. Its just laugable that they getting paid $300K for throwing nonsense ( or simple comman sense)

[@boylitte]

Yea. that's one if the product is still in growing stage. If most ppl has already been using fb event to organize events, alternatively, the goal could be to leverage events to grow more user interactions/engagement on FB - eg. total # of comments in events page, # new friends formed from events attendees.

[@TheMrBayArea]

@@boylitte I agree with this assessment, but it's both. 1) The whole reason for anyone to create events and socialize them via FB is for increased participation (and audience building). In the absence of physically determining attendance, leading indicator of event success is # of RSVPs. As FB, I'd also measure depth of engagement - % of checkins from RSVP (for physical events), # comments in events page, # of new connections formed from RSVP list.

[@mrwazo]

It's good. But is an interview this long in 2024? I'm used to using star concisely with a specific example.

[@manosaieerabathini4724]

It was a bit frustrating that the interviewer did not push back or really give her perspective on rating the interview…so I’ll give my 2 cents. While TC is a structured thinker and intentional in navigating the question, I would rate them as a no hire for this interview. Communication style wasn’t great despite sharing his screen, and more importantly: defining the success metrics left a lot of room for improvement. Nuanced details like why it’s important to scale and keep the the supply side of events in this network relevant/interest or digging beyond a digital RSVP rate (which doesn’t ensure actual attendance) were completely overlooked. It’s a high-level okay answer but not at the bar I would expect from a director-level PM.

[@rahulsharma-li4io]

Additionally, I felt that the interviewee started rambling towards the end and was not making much sense. The classification of users was very basic and the metrics were not tying back to how it helps the related users. When talking about the hypothesis and experiment, a visual funnel and f(x) would have been great.

[@user-jp7ni5xv1r]

I agree. The structure was really lacking for a director level answer.

[@christinecc4858]

Basically been repeating the framework he memorized... mechanically

[@ozdynamics]

Great video to understand about how to measure success but can't follow the same style during an interview. I doubt the interviewer will give sufficient time to answer 1 question.

[@saketpandey8897]

precisely

[@Abdullah-zx6mp]

Great explanation but the answer could be more shorter. I felt it was dragged.

[@timerwentoff]

Great interview, combining questions on the metrics for success and issues with a product.

[@tryexponent]

Thanks for watching1

[@productmanagementbyutsav3069]

Thanks for this video

[@JayantaDash]

It would be great, if you please can you share the notebook with everyone.

[@alexandergloe3833]

This is an excellent strategy

[@AB-dw8vo]

very similar to the chatgpt response. and not quite to the point in my POV, RSVP Rate doesnt mean users will attend the event. kinda misses the point of hte events platform success.

[@esthergao4049]

any other ideas on how to answer the counter question between 17:00 - 18:00?

[@kielaagustin3600]

The product also has no influence and control over whether a user is actually available for events, there's too much external factors to make RSVP a good metric. Would be great if he thought more of the realistic uses of Events: there's personal/life events, social, and business. Creators would definitely spam invites for business type of events. On "joiners" segment: I would focus on number of users expressing interest to events (I'm interested CTA, or RSVP) which is a good indicator of good algo/matching and isn't dependent on invites. Next, probably whether users want to be notified of similar events or posts from those events. Counter metrics: I'd look at spam similar to any "bad content" in FB (reported events, low traffic events, "I'm not interested"). For cannibalization, I'd look into Events VS Groups because they're similar in nature except that Events are timebound. For the creator segment: I'd actually segment this further to type of creator whether personal or business/enterprise. Not much you can do to influence users to create personal/social events and there's only a very limited type of persona that would organize these, so I would focus on business sector because they'd be more motivated to reach new customers => then we get into monetization Although his process and structure makes a lot of sense, I don't agree with a lot of the metrics and direction

[@mayanksaxena3423]

Great insights. Will follow this strategy for my upcoming interviews. Thank you for sharing!

[@marcodivivo4751]

did it work ? :)

[@JG-dy2hk]

Great video

[@rhythmtv1753]

Informative

[@ananyasingh3376]

Super helpful! Great for newbies like me.

[@BlueStag]

What's a CTR?

[@iamak47h]

Click through rate

[@tryexponent]

Yep!

[@BlueStag]

Thank you

[@shivalishinde4327]

Slow interview! Too much repetition! Repeat users or first time event booker Hook strategy for first time users.

[@Deepaksie]

Communication wasn’t great

[@sumanguha2165]

It's probably a Weak Hire not a Hire if someone answer this

[@aayomide]

Why?

[@bankpo6099]

What is prd?? Plz

[@vinayreddyvaradhapally5190]

Product Requirements Document

[@nithishmonson6702]

This interview is soo bad, this would definitely be a no hire. You get rejected for much less. Exponent shouldn't post such bad interviews and give the wrong idea to candidates.

[@DarshanShah1994]

What’s bad about it nithish, more helpful to viewers if you can point out

[@arunbhargava2357]

You have no idea what you are talking about. This is a great interview. Don't mislead people.

[@hyperlemongt4890]

I agree that this wasn’t good. No strong reason for bring competitors in, no funnel in the metrics, jumping to conclusions on why the metric had dropped, without suggesting how to use metrics to find the route cause

[@markoshivapavlovic4976]

mrs