I’m not a PM, but I have worked on the home screen for a major streaming service and wrestled with some of these trade-offs as algorithmic recommendations and ranking were introduced. I learned a few things along the way...🧵
Before the home screen is algorithmically ranked, it likely represents the product of many years of messy stakeholder interaction and negotiation, during which they all vied to have their thing at the top.
When new home screen elements are being tested, everyone wants to know how their new thing performs in the top slot. Performance at the top is in no way representative of the overall value of having the content somewhere on the page.
Some content performs comparably with other alternatives in the top slot, but almost no one will interact with it when it’s farther down the page. It may decrease overall page metrics when placed at the top, even though it looks competitive in element-wise metrics.
The various parts of the company that create and lobby for new page elements will tend to base their positioning arguments on element-wise metric comparisons:
“Maybe it doesn’t get quite as much interaction, but having it at the top has [some unquantifiable strategic benefit.]”
“Maybe it doesn’t get quite as much interaction, but having it at the top has [some unquantifiable strategic benefit.]”
But not everything can go at the top, and the team that owns the home screen is going to have to make some hard decisions. Over time, those decisions can have a profound effect on which parts of the company feel supported or screwed by the allocation of screen real estate.
That is particularly the case when performance is evaluated (at least in part) by how many users interact with their content. In that case, their ability to successfully lobby you to give their content wider exposure is a direct contributor to their performance evaluations.
When you try to introduce algorithmic ranking, people will tend to view it through the lens of past political battles about whose content gets the top slot. Whoever has it will want reassurances that their exposure won’t drop. Everyone else will want a bigger slice of the pie.
What most people won’t be prepared for is that ranking content in a personalized way changes the dynamics profoundly. There are no longer specific humans who determine which content gets the most exposure. The way you get more exposure is by refining your content.
What “good” content looks like is going to change significantly too though, which constitutes a double whammy for the rest of the company. In particular, anyone who has been grouping content based on a “something for everyone” philosophy is about to get a rude awakening.
Before you had the ability to select and order content in a personalized way, a shelf of blockbusters, bestsellers, or hits probably performed quite well. Everyone saw it and there was a decent chance they found something they liked.
When you start algorithmically personalizing, the metrics for the same shelf of best hits are likely to drop, because *most of the content is irrelevant to most people*. If I read mysteries, I want to see “Critically Acclaimed Mystery Novels” a lot more than “NYT Bestsellers.”
One fear people will have when they see the metrics for these editorially curated collections of popular stuff cratering is: “Oh no, there’s no longer a role for editorial expertise. Computers are taking away those jobs.” It’s a reasonable fear, but it’s not necessarily true!
Now that you have a way to show content only to people who are likely interested in it, there’s actually a lot *more* room for expert curation of niche and special interest content. You’re no longer limited to content with mass appeal.
There’s no longer one “top slot”; there are as many top slots as there are users. Many more than that even, because the page content probably isn’t static and will change from hour-to-hour and day-to-day.
All of the things that used to seem simple (but actually weren’t) are now going to reveal their full complexity to you.
The metrics for each individual page element are going to become quite difficult to parse. Engagement metrics (like CTR) reflect both how much exposure something received and how well it was targeted. You may be in the habit of viewing them as a proxy for content quality, but...
There’s no longer a clear way to separate the intrinsic appeal of the content from the dynamics of the recommender system in your metrics. (There never was, even when the system was “we choose a static order for everyone,” but now it’s going to become blindingly obvious.)
So you have to start shifting the way people think about content performance, away from “maximizing engagement with any one thing” and toward “using impressions efficiently.”
A shelf with a variety of bestsellers might get maximum engagement when shown to everyone, but it does so very inefficiently. A shelf with hits in a genre with narrow appeal can’t beat it in terms of maximizing engagement, but it may get incredibly high engagement per impression.
People won’t be used to thinking about content/page/performance/engagement metrics that way at first though, and it’s going to make every stakeholder conversation more complicated for quite a while.
Now that you’ve gone from a system where everyone sees the same thing and ordering is decided by politics to a system where everyone sees something different and ordering is decided by magic (er, ML/RL)...
Even defining what “promotion” means is challenging. 🤔
Even defining what “promotion” means is challenging. 🤔
