How the Q&A / Featured Snippet Algorithm Works (Ali Alvi with Jason Barnard)

Ali Alvi with Jason Barnard at The Bing Series Ali Alvi talks to Jason Barnard about the search algorithm for featured snippets. First thing we learn is that this feels a lot like a soccer interview. https://www.youtube.com/watch?v=l0UhsQb5iAU Then Ali confirms what Gary Illyes said in 2019 - the different candidate sets use the core algo in a modular fashion. Ali is team lead for the Q&A candidate (Q&A is Bing's name for featured snippet) But also that all of the algos are end-to-end neural networks. We know what goes in, we see what comes out… but nobody knows what goes on in between :) And a nice clarification - Q&A are pulled from the blue links below it. Other rich elements such as video and images don't rely on the pages the 10 blue links provide - they have a separate selection process. Now that is interesting. Even more interesting - Ali answers the intriguing question "where do the descriptions for the blue link / core results come from?" (spoiler alert - it isn't from the core algo!) We talk a great deal about trust - Bing must trust the website providing the answer. So building trust over time has to be key. And then onto the main factors / features that affect ranking for Q&A are: accuracy, trust, authoritativeness, freshness… and not being offensive (aka safeguarding Microsoft's reputation). We also discuss Google's decision to remove the result from the main results when content is used as a featured snippet (Ali doesn't agree with Google here). And finally, dependence on annotations by the crawling and indexing team, as discussed with Fabrice Canel in the previous episode. It all fits together so nicely ! Catch the rest of the Bing Series: How Ranking Works at Bing - Frédéric Dubut, Senior Program Manager Lead, BingDiscovering, Crawling, Extracting and Indexing at Bing - Fabrice Canel Principal Program Manager, BingHow the Q&A / Featured Snippet Algorithm Works - (this episode) Ali Alvi,  Principal Lead Program Manager AI Products, BingHow the Image and Video Algorithm Works - Meenaz Merchant, Principal Program Manager Lead, AI and Research, BingHow the Whole Page Algorithm Works - Nathan Chalmers, Program Manager, Search Relevance Team, Bing Full Corrected Transcript for How the Q&A / Featured Snippet Algorithm Works (Ali Alvi with Jason Barnard) Jason Barnard: The camera is kind of far away. They usually have cameras right in people's faces. Anyway, welcome to the show, Ali Alvi. Ali Alvi: Thank you. Great to be here. Jason Barnard: That's the best name I've heard all day. I love your name. Ali Alvi: Thank you. Ali Alvi, rolls off the tongue, doesn't it? Jason Barnard: Yes, brilliant. Ali Alvi: Like the boxer. People debate whether it's "Ali" or "Ali." I say it like the boxer. Jason Barnard: All right. So here we are, looking out over Seattle from the Bing offices. You're the team lead for Q&A? Ali Alvi: Yes, I'm the lead PM for the team that handles Q&A in Bing, including the captions and the snippets you see under the URLs in the search results. Jason Barnard: The blue link descriptions. Better descriptions that get pulled dynamically. That's part of Q&A, so they can be generated as well? Ali Alvi: Yes. The algorithms we use to generate the snippets are essentially the same algorithms we use for Q&A. Google calls it "featured snippets" because a snippet is just a feature. We use a slightly different framing at Bing: we're saying this is an answer to a question, which is more explicit. And when you look at the architecture, we're not just taking a snippet and featuring it. We actually do a lot more than that in many cases. So broadly, it falls into this category: when a user comes and asks a question, or a query that looks like a question we can answer directly, that's the domain my team handles. In addition to that, I'm also the lead PM for a high-ambition AI initiative called Project Turing. Jason Barnard: Project Turing. That's a Microsoft initiative, particularly within Bing? Ali Alvi: There's a team of scientists and applied researchers working on high-ambition natural language processing algorithms. We're kind of the hub for those algorithms across all of Microsoft. That same team provides some of the models we use in Q&A. Think of it as a horizontal team that provides the brains for a lot of these scenarios, and Q&A is one of them. Jason Barnard: So you're the brains behind Bing? Ali Alvi: I wouldn't go that far. I represent the brilliant minds who are the minds behind Bing. Jason Barnard: So for Q&A, my journey to this conversation started when I asked Gary Illyes from Google whether there's a separate algorithm for featured snippets. He said, very dryly, "No," and then explained how it works. The idea is that you've got the basic algorithm for the blue links, and then there's a module alongside it that uses either different features, or the same features with different weightings? Ali Alvi: Maybe I should take a step back. Jason Barnard: Absolutely. I went too fast. Ali Alvi: You jumped straight to what makes results rank at the top. Let me back up. Search engines historically have been just ten blue links, and that's how it was for around fifteen years. Q&A, or featured snippets, started coming around about three or four years ago. The idea is: we have a query, we narrow it down to the ten, fifteen, or twenty most relevant documents, and then Q&A asks, "Can we have the machines read through those documents, do some comprehension on top, and extract the specific part that actually answers this question right on the spot?" Jason Barnard: So Q&A is actually based on the results underneath it. And video, images, and knowledge panels are based on completely separate processes? I'm beginning to understand. You're working vertically from the blue links and asking: what can we pull out that gives a definitive answer? Ali Alvi: Well, it depends on the question. If you ask "How tall is the Eiffel Tower?" that's very definitive. But sometimes you ask things like "What is the average salary for a computer scientist?" and there's no single definitive answer, so sometimes we give a range. You can also ask subjective questions like "What vegetables take the shortest amount of time to grow?" and there's no one right answer there either. Jason Barnard: I've been saying this for a while: if you want Bing to put you at number one, you're asking Bing to recommend your content as the solution or the answer. But the featured snippet is different. It's not recommending, it's saying, "This is the answer we've found to be the best." Ali Alvi: Yes. When you have a featured snippet or a Q&A, it becomes tricky from a user's perspective. They think Bing is telling them "This is definitively it." But the reality is, we're saying: given the context, this is the best answer we found. We're not declaring it the absolute truth. Jason Barnard: But isn't that a sign that people trust you and Google? We've got to the point where we just accept it as the answer, and when it's wrong, we get really upset. Ali Alvi: Absolutely. Part of my job is to channel that sentiment from users and drive that empathy through the whole product. Even when we picked the best answer we could find, sometimes it's not correct, or it's off-topic, or it's hurting people's sensibilities. When that happens, users perceive it as something Bing did. So we have to own the message. Jason Barnard: You have a feedback button on the SERP, so you get a lot of direct feedback from people? Ali Alvi: We get it right a lot of the time, but we do get it wrong. We make sure we're as close to the customer as possible. We call it "zero distance to our customers." That means doing user surveys, bringing people in-house, asking questions. And any feedback we get, we respond to internally, at least to direct it to the right people. Jason Barnard: Do you read everything? Ali Alvi: Yes. Everything. Jason Barnard: Back to the algorithm. It's based on the blue links. What are the most important features you feed into the machine learning algorithm? I'd immediately think expertise, authority, and trust. Ali Alvi: I was actually going to flip the question: as a user, when you come and ask Bing a question, what would you say makes a good answer? Jason Barnard: I'd imagine you're looking at expertise, authority, and trust. And it also needs to be accurate. Ali Alvi: Accuracy is a hard thing to judge. Jason Barnard: Good point. But I keep hearing that accuracy is based on accepted opinion. Ali Alvi: You need to figure out what accepted opinion is, and that's the biggest part of my job: defining the right metric. As a product manager for an AI team, we don't write code. We define what the algorithm needs to do and how to measure whether it's doing it correctly. That responsibility is 100% on the product manager. I'm the one who defines that metric and holds the entire team accountable for meeting it. Jason Barnard: So the metric is the secret sauce you'd never tell me. Ali Alvi: You already said what it was. It has to be relevant, authoritative, trustworthy, not offensive, and fresh. What I would add is that we're using almost entirely neural networks and deep learning-based solutions. And by definition, with deep learning, we don't know exactly what the features are. The machine takes text, gets the query, gets the passages we give it, and comes back and tells us which one to show. Jason Barnard: That reminds me of conversations where people say there's no point asking what the ranking factors are. So what I understand is: your team labels this as a question and this as a correct answer, you build a dataset, and you feed it into the machine with the metrics you're looking for? Ali Alvi: Exactly. And you can see that if the metric is wrong, the machine will latch on to whatever the metric says.