How Ranking Works at Bing (Frédéric Dubut with Jason Barnard)

Frédéric Dubut with Jason Barnard at The Bing Series Frédéric Dubut talks to Jason Barnard about Bing's search algorithm. https://www.youtube.com/watch?v=JyLwHeEViNQ Episode #1 in a series about how ranking functions at Bing This conversation confirms that the overall system for ranking at Bing functions in the same way as Google (as explained by Gary Illyes) - Darwinism in Search But things rapidly become more interesting still… Frederic is the 10 blue links / core algorithm team lead but goes on to explain a little about how ranking works for featured snippets, images, videos… and intriguingly how the whole page algorithm works. After this interview, he generously organised a series of interviews with other team leads at Bing. The Bing Series Listen to (or watch) this episode with Frédéric to whet your appetite for the stunning revelations that the Bing team leads give me in the other four. Frédéric shared a lot of interesting information me in this conversation. But that is nothing to what I learned in the other four. How Ranking Works at Bing - Frédéric Dubut, Senior Program Manager Lead, BingDiscovering, Crawling, Extracting and Indexing at Bing - Fabrice Canel Principal Program Manager, BingHow the Q&A / Featured Snippet Algorithm Works - (this episode) Ali Alvi,  Principal Lead Program Manager AI Products, BingHow the Image and Video Algorithm Works - Meenaz Merchant, Principal Program Manager Lead, AI and Research, BingHow the Whole Page Algorithm Works - Nathan Chalmers, Program Manager, Search Relevance Team, Bing The Bing Series (Part 2) Assuming these 5 episodes are well received, we'll do another series of interview-cum-conversations later in 2020. Hopefully getting more detailed insights into other things Frédéric mentions - I'm hoping for Ads (that work on much the same principle as other rich elements), knowledge panels, local results… and more. Full Corrected Transcript for How Ranking Works at Bing (Frédéric Dubut with Jason Barnard) Jason Barnard: Welcome to the show, Frédéric Dubut. Frédéric Dubut: Thank you for having me. Jason Barnard: Absolute pleasure. We've met a couple of times. Frédéric Dubut: That's right. Jason Barnard: I've talked to people about the ranking algorithm and nobody will tell me any secrets. I suppose you won't either, but I wanted to talk to you about the candidate sets — the idea Gary Illyes described, where the blue links form the basis and the candidate sets are bidding for a place. They need to outbid the top blue link, and if they do, they win the position. Frédéric Dubut: Sounds about right. In the end, what we want is to really serve our users. The ten blue links are the basis of everything. And then, if the query is a question or something we can answer with an intelligent answer — what Google calls a featured snippet, and what we call Q&A internally — that comes on top. And then there are all the other answer types. So there's a different team for each. I'm the ten blue links. I have a colleague who handles Q&A. And then there's a team called Whole Page. Jason Barnard: [Laughs] Frédéric Dubut: The Whole Page team, as the name suggests, runs the entire page as an end-to-end product. They arrange ads when the ads team tells them there are ads to show. They look at the ten blue links. They look at potential answers — video answers, image answers, news answers — and if they think those answers are going to satisfy users more than some of the blue links, that's where they start inserting them. Jason Barnard: So my Darwinism framing is that these candidate sets bid for a place, and they live or die by whether they can convince the algorithm they have more value than a blue link. But you're saying it's actually teams deciding that their specific element is more interesting and inserting it manually? Frédéric Dubut: No, no, no. It's not manual. If, say, I'm working on videos, I generate the best video answer I can for a given query. But my team is not the one deciding whether it shows up on the page. That's the role of the Whole Page team. Jason Barnard: And it's all working on the same algorithm but with different weightings? Frédéric Dubut: Yes. For featured snippets, being accurate, being fresh, and being authoritative is going to matter much more than having links, for example. It's the same central algorithm working with different weightings, and each team is tweaking it for their specific rich element and their specific need. And what you show really depends on the query. Take a query like "Beyoncé." It's very important to show videos and news — that's what users want. In that case, the ten blue links matter less. The master algorithm makes the call. But for a simpler, more general query — "what is one plus one," say — the calculation is completely different. Jason Barnard: "What is one plus one" — I'd imagine that gets a direct answer, a featured snippet. But if you just type "one plus one" without the question framing, you might get a hosting company called One on One, or a song. It's not as simple as it looks. Frédéric Dubut: Probably, yes. And each team also signals how confident they are in their own answers. If the video team can pull strong results for "1+1" — whether it's a video explaining the maths or a Beyoncé song — that confidence level feeds into the Whole Page decision. Jason Barnard: Coming back to ads: is the same principle at work there? Is there an algorithm calculating whether an ad is valuable to the user, with a team behind it similar to the video team? Frédéric Dubut: Yes. The key principle for ads is that we still want to satisfy users. We want ads to be relevant, and we want users who click on an ad to find that what's on the other side actually satisfies their query. It's a sponsored auction, but the same principle of user satisfaction applies. Same system of candidate sets bidding for a place. The difference is transparency: we don't interleave ads within the organic results. Ads sit on top and at the bottom. But other answers, like video, can be inserted between positions three and four. Jason Barnard: And how do you decide whether a video appears at position one or between three and four? Is it saying: "This video satisfies the user enough to get on the page, but not enough to be the very first answer"? Frédéric Dubut: Yes, that's the right intuition. Whatever the algorithm places at position one is probably the best result for users. Showing the video between three and four says: the top three results are probably more satisfactory overall. Jason Barnard: But you said "intuition," not algorithm. The algorithm is built by humans, right? Frédéric Dubut: It's a machine learning model. The definition of the algorithm is built by a human — you put your intuition into it. What signals are important? What signals are not? And then, using machine learning, you train the model to balance all those signals. Jason Barnard: Machine learning is like cooking. You've got ingredients, utensils, and a chef. The data is the ingredients, the algebraic models are the utensils, and the intuition of the person building it is the chef. Is that right? Frédéric Dubut: Yes. For traditional machine learning, you still need to tell the machine what signals — or features, as we call them — you think are important. Jason Barnard: "Ranking features," not "ranking signals"? Frédéric Dubut: In machine learning, every kind of input is called a feature. You can have hundreds of them in any algorithm, not just in search. The classic example: if you're Zillow — a US real estate company — and you want to predict the price a house will sell for, your features might be square metres, number of bedrooms, location. Those are all different features that you, as a human, define as inputs. Jason Barnard: So when we in SEO say "ranking factors," we should really be saying "ranking features" in machine learning terms? Frédéric Dubut: That's right. You, the human, tell the machine: these are the things I think matter. Then you give it a lot of examples, and the machine assigns different weights. To carry your cooking analogy further: you tell the machine you probably need eggs, milk, and butter. Then the machine determines how many eggs, how much milk, and how much butter to make a good pancake. Jason Barnard: And the intuition of the person programming it is to get the machine off to a good start — pointing it in the right direction so it doesn't produce the world's worst omelette on the first attempt? Frédéric Dubut: Yes. And one of the nice things you can do is cook ten different pancakes with different proportions and have a human taster judge each one: this is very good, this is very bad, this is okay. The machine then adjusts the formula automatically. That's exactly what we do for web ranking. We have a set of queries and URLs, and our human labellers — sometimes called judges or raters — say: this is good, this is bad. Jason Barnard: The machine then looks at which features are the most predictive of something being good or bad? Frédéric Dubut: Exactly. Jason Barnard: That's the same principle as Google's quality raters? Frédéric Dubut: Yes. When we write the guidelines — which are essentially the product specification — we want them to produce universally good results, applicable across all the markets we operate in. General principles: what it means to be on-topic for a query, what determines a quality website. We want the guidelines to be objective enough that two different raters who understand them and judge the same query and URL will arrive at the same rating. Jason Barnard: But two people will never give exactly the same rating — there's always an element of judgement. Frédéric Dubut: True, which is why you need a large enough scale. And the guidelines are fairly objective.