Product Metrics (15 questions)

1. What would be good metrics of success for an advertising-driven consumer product? (Buzzfeed, YouTube, Google Search, etc.) A service-driven consumer product? (Uber, Flickr, Venmo, etc.)

• advertising-driven: Pageviews and daily actives, CTR, CPC (cost per click)
  • click-ads
  • display-ads
• service-driven: number of purchases, conversion rate

  2. What would be good metrics of success for a productiv- ity tool? (Evernote, Asana, Google Docs, etc.) A MOOC? (edX, Coursera, Udacity, etc.)

• productivity tool: same as premium subscriptions
• MOOC: same as premium subscriptions, completion rate

  3. What would be good metrics of success for an e-commerce product? (Etsy, Groupon, Birchbox, etc.) A subscrip- tion product? (Net ix, Birchbox, Hulu, etc.) Premium subscriptions? (OKCupid, LinkedIn, Spotify, etc.)

• e-commerce: number of purchases, conversion rate, Hourly, daily, weekly, monthly, quarterly, and annual sales, Cost of goods sold, Inventory levels, Site traffic, Unique visitors versus returning visitors, Customer service phone call count, Average resolution time
• subscription
  • churn, CoCA, ARPU, MRR, LTV
• premium subscriptions:

4. What would be good metrics of success for a consumer product that relies heavily on engagement and interac- tion? (Snapchat, Pinterest, Facebook, etc.) A messaging product? (GroupMe, Hangouts, Snapchat, etc.)

• heavily on engagement and interaction: uses AU ratios, email summary by type, and push notification summary by type, resurrection ratio
• messaging product:

  5. What would be good metrics of success for a product that o ered in-app purchases? (Zynga, Angry Birds, other gaming apps)

• Average Revenue Per Paid User
• Average Revenue Per User

  6. A certain metric is violating your expectations by going down or up more than you expect. How would you try to identify the cause of the change?

• breakdown the KPI’s into what consists them and find where the change is
• then further breakdown that basic KPI by channel, user cluster, etc. and relate them with any campaigns, changes in user behaviors in that segment

  7. Growth for total number of tweets sent has been slow this month. What data would you look at to determine the cause of the problem?

  8. You’re a restaurant and are approached by Groupon to run a deal. What data would you ask from them in order to determine whether or not to do the deal?

• for similar restaurants (they should define similarity), average increase in revenue gain per coupon, average increase in customers per coupon

  9. You are tasked with improving the e ciency of a subway system. Where would you start?

• define efficiency

  10. Say you are working on Facebook News Feed. What would be some metrics that you think are important? How would you make the news each person gets more relevant?

• rate for each action, duration users stay, CTR for sponsor feed posts
• ref. News Feed Optimization
  • Affinity score: how close the content creator and the users are
  • Weight: weight for the edge type (comment, like, tag, etc.). Emphasis on features the company wants to promote
  • Time decay: the older the less important

    11. How would you measure the impact that sponsored stories on Facebook News Feed have on user engagement? How would you determine the optimum balance between sponsored stories and organic content on a user’s News Feed?

• AB test on different balance ratio and see

  12. You are on the data science team at Uber and you are asked to start thinking about surge pricing. What would be the objectives of such a product and how would you start looking into this?

• there is a gradual step-function type scaling mechanism until that imbalance of requests-to-drivers is alleviated and then vice versa as too many drivers come online enticed by the surge pricing structure.
• I would bet the algorithm is custom tailored and calibrated to each location as price elasticities almost certainly vary across different cities depending on a huge multitude of variables: income, distance/sprawl, traffic patterns, car ownership, etc. With the massive troves of user data that Uber probably has collected, they most likely have tweaked the algos for each city to adjust for these varying sensitivities to surge pricing. Throw in some machine learning and incredibly rich data and you’ve got yourself an incredible, constantly-evolving algorithm.

13. Say that you are Net ix. How would you determine what original series you should invest in and create?

• Netflix uses data to estimate the potential market size for an original series before giving it the go-ahead.

  14. What kind of services would nd churn (metric that tracks how many customers leave the service) helpful? How would you calculate churn?

• subscription based services

  15. Let’s say that you’re are scheduling content for a content provider on television. How would you determine the best times to schedule content?Â