How the Whole Page Algorithm Works at Bing (Nathan Chalmers with Jason Barnard)

Nathan Chalmers with Jason Barnard at The Bing Series Nathan Chalmers talks to Jason Barnard about the whole page algorithm. Nathan had done his homework by listening to previous episodes…. And he loves the song :) https://www.youtube.com/watch?v=L5kJFlMIHVU A year ago I didn't know about the bidding system in Google and Bing ranking… and 6 months ago I didn't know there was a whole page team. Now I get to learn how the latter affects the former. First up, I had called the bidding system "Darwinism in Search". Turns out Darwin is the name of their algorithm that allocates the positions on the SERP to different rich elements. He tells me my initial analysis was good, but I made it all sound more mechanical than it truly is. Next reveal - there are 6 teams that work together to build the whole page, each with their own Darwinistic system, with the aim of creating the best user experience. The blue links are the base and they are NOT dying out, even if I thought they were. Then onto Brand SERPs - and although I had seen that they are nuanced, but I had underestimated just HOW nuanced. User behaviour on the SERP is key to the anatomy of that page of results. Whatever the SERP - Bing's aim is to get the user to success as fast as possible. Brilliant from Nathan - I would say that as he ends by saying "this whole Darwinism in Search thing is really cool". Catch the rest of the Bing Series: How Ranking Works at Bing – Frédéric Dubut, Senior Program Manager Lead, Bing Discovering, Crawling, Extracting and Indexing at Bing – Fabrice Canel Principal Program Manager, Bing How the Q&A / Featured Snippet Algorithm Works – Ali Alvi,  Principal Lead Program Manager AI Products, Bing How the Image and Video Algorithm Works – Meenaz Merchant, Principal Program Manager Lead, AI and Research, Bing How the Whole Page Algorithm Works – Nathan Chalmers, Program Manager, Search Relevance Team, Bing The full transcript from How the Whole Page Algorithm Works at Bing (Nathan Chalmers with Jason Barnard) Jason: A quick hello to start the show. Welcome to the show, Nathan Chalmers. Nathan: Awesome. I've been listening to your podcast for a couple of episodes and I love this song. Jason: You've been doing your homework. Lovely to meet you — we've just met literally not even a minute ago. Nathan: Not even. Jason: And you're the whole page guy. Nathan: I am the whole page guy. One of the whole page guys — there are literally dozens of us. Jason: Brilliant. Nathan: Just on PM Jason: what's PM? Nathan: Product management. Jason: A year ago I didn't know the concept of different teams working on different rich elements, pushing them into the SERP with the bidding system. Google explained it to me, Frederic (Dubut) re-explained it to me, and he mentioned the whole page team. I said "ooooh", because I was thinking Darwinism in Search... that it's not really Darwinism, is it? Nathan: Ironically enough, the name of the algorithm that does the placement of some of our rich elements is actually called Darwin. Jason: Oh yes — so I was right after all! Nathan: I read through your article on Darwin in Search and there's a lot there, but it's a little different. Jason: My article is fairly accurate, but not 100% — I threw a few things in that I think were a bit off. Nathan: I would say it's not as mechanical as the way you outlined it. At the end of the day, how does this whole page stuff work? Jason: Well, that was the big question. The idea of different rich elements bidding for a place just makes so much sense — each one has an adaptation of the algorithm run by machine learning, going back and forth. But the whole page team sits on top of all that and says, okay, you've made this great bid, videos, but you're not in. Nathan: Exactly. You have to remember, there are six teams that represent the whole page, not just one. So it's way more complicated. As Frederic was saying, the algo results are really the core of it — the blue links. Then you have the rich answer cards in the middle, those are a component too. But then there's the right rail with our entity pane, there's ads, rich captions — sometimes you see the little pictures on the algorithmic links, or the tables — there's a whole team behind each of those. These are all whole page teams working together to build the whole page. Jason: So I had the idea there was one whole page team, but in fact it's six smaller teams, each with their own Darwinistic system. Nathan: Exactly they each have their own Darwinistic system. And basically what we're all working towards is optimising for user satisfaction. Jason: Yeah, Nagu was telling me about that — and that's what strikes me. You're looking to serve your clients the best way you can, which is what we're all doing. Nathan: Exactly, exactly. Jason: People forget that. Sorry, go ahead. Nathan: Yeah, so a lot of teams will have an online metric around user satisfaction — that's where we look at how users are interacting with the page from the logs, where they're clicking, where they're not clicking, etc. And we have offline metrics too. I think Fredrik talked to you a little bit about this — the human judges. Jason: You call them human judges? Amazon calls them click workers or Mechanical Turk workers, and Google calls them quality raters. But you've got human judges. Nathan: We have human judges. They help us label our experiences. We have online metrics that help us label our experiences. We look at both, we optimise for both, and we try to figure out how the whole page should work. Jason: The question I had was this: I was obsessed a couple of months ago about the death of the blue link — I was convinced it was dying bit by bit. But I get the feeling we've hit the trough, as in whatever it is now, it's not going to change greatly. Nathan: It's not really the death of the blue link — the blue link is the core. But sometimes we can come up with an experience that's a little bit more authoritative or easier to digest. If you're searching for weather in Seattle, it's way easier if we just show you the weather card instead of having you click through. Jason: Yeah. And a lot of people are saying we don't get the click anymore, forgetting that as a user, we're very happy — as a marketer, maybe not, but as a user, yes. Nathan: Right. So basically, all these algorithms — especially when we're making placement decisions for the rich cards or the right rail — operate from a baseline assumption: the blue links are our base standard, the best Bing can offer. So if we put something else on top, is it going to be better for the user than just the blue links, or worse? We have a competitive system that measures how things compare. Jason: Yeah — Darwinistic. I'm so pleased you have things called Darwinistic. Nathan: It's interesting that it's Darwinistic too, because it's not just competition for an individual answer. We have hundreds of answers, and they all have different levels of quality depending on which market you're looking at or what the query is. Jason: Are you breaking it down by query intent type, by country, by language? It must be broken down into so many different ways. Nathan: So for answer placement — showing those rich cards at the top of the page — that situation is intractable when you look at all the intent types, markets, and combinations. There are literally hundreds. I work with some pretty smart guys and we can't go in and figure out each one individually. So what we've built is a system that learns from what's happening in production and teaches itself the best approach. Jason: It's learning from user interaction — what's working, what isn't. You're not actively separating things into categories or user intent. You're pushing all of this into machine learning. Nathan: Exactly. The machine learning looks at, okay, here's a weather answer — I know from previous experience with the weather answer and what people click on or don't click on, I can figure out what this answer type satisfies. So it learns how to optimise the placement. And we do this in a very generic way so it can work for everything. Jason: So what we're all forgetting as marketers — and I think it's really important to remember — is nobody's going in and setting the rules anymore. It isn't if/else, if/else. It's: here's the goal, here's what we're optimising for — please, machine, make it as good as you can. Nathan: Exactly. You set the rules for the machine, and then it's really just a satisfaction optimisation problem. Jason: Okay. I have a question about ads. My immediate reaction to the whole page team would be: what they do is look over the page and say, that's not making us money, and it could — stick more ads in. That isn't how it works, is it? Nathan: No. I'm not the best person to talk about how the ads work — there are six different teams. But basically the whole page team for ads looks at what dials can we turn to make a page more or less profitable, while still having goals on user satisfaction. Because we could make a ton of money by dropping all the algos and spamming all the ads, but we'd lose our clients. Those ads sit at a very prominent position on the page, so you're going to click on them just by random chance if there are more of them. The ads team has to trade off between money and user satisfaction, and they have models that do that. Jason: Perfect. Another question. I track brand SERPs — my thing is brand SERPs. I know boatloads about brand SERPs. And on the death of the blue link: I've got brand SERPs for a full year and they've consistently been at around seven blue links. Unchanged. So blue links aren't dying out. Jason: The question was — I get video boxes, I get image boxes. How do you decide what ranking they get? Sometimes they show up at position three,