The Low-down on IndexNow From Mr. Bingbot (Fabrice Canel and Jason Barnard)

Fabrice Canel talks with Jason Barnard about the low-down on IndexNow. Fabrice Canel is a Principal Program Manager leading the crawling, processing, and indexing team at Bing. He is a veteran at Microsoft Bing that has dealt with crawling web pages since the beginning of Bingbot and now deals with the hundreds of billions of new or updated web pages every day! In this episode, Fabrice Canel and Jason Barnard discuss IndexNow - a stunningly great change that is happening to search engine indexing. One of the ultimate goals of search engines is delivering timely information to its users. However, crawling websites may take days or weeks so updated, added and deleted information often takes some time to reflect in search engines. This is a big problem for website owners and one they often struggle to deal with. So to relieve this pain point, Microsoft Bing has introduced IndexNow, a new protocol allowing websites to easily notify search engines whenever they make changes to website content. Because it's new, there are a lot of questions that need to be answered, and Jason asks Fabrice all the right questions, of course :) How does it work?Can it instantly crawl our websites?Is Google also adopting IndexNow?What CDN have adopted IndexNow?What CMS systems have adopted IndexNow?How does the API work? Fabrice (aka Mr. Bingbot) answers all these questions simply, clearly and extensively. That makes this episode a must-listen…. Especially as Fabrice makes it very clear he will not stop until he gets 80% adoption of IndexNow. He is absolutely convinced that this initiative will make everyone happy: website owners, users, and search engines. Tune in! What you'll learn from Fabrice Canel 00:00 Fabrice Canel and Jason Barnard00:35 The beginning of Bingbot01:59 Can Bingbot keep up with the infinite pages that need to be crawled?05:55 Fabrice Canel’s Brand SERP on Bing and Google07:18 IndexNow: What it is, how it works and why it's helpful08:55 Why an API layer is better than a web form11:37 The early adopters of IndexNow14:52 Getting the CMS and the CDNs to do the work15:57 How long will it take for IndexNow to catch on?19:30 Will IndexNow be integrated into the core of WordPress?23:13 Why using IndexNow can reduce crawl on a website24:27 Why IndexNow matters to small websites30:05 Does IndexNow guarantee instant indexing?32:20 How fast can websites be caught for abusing IndexNow?33:21 Will this make XML sitemaps redundant?35:07 The IndexNow API is super simple This episode was recorded live on video November 2nd 2021 Recorded live at Kalicube Tuesdays (Digital Marketing Livestream Event Series). Watch the video now >> Full Corrected Transcript for The Low-down on IndexNow From Mr. Bingbot (Fabrice Canel and Jason Barnard) Jason Barnard: Hi everybody and welcome to the second part of this double bill of Kalicube Tuesdays with Fabrice Canel. A quick hello and we're good to go. Welcome to the show, Fabrice Canel! Fabrice Canel: Thank you. I will just speak — no song from me. Jason Barnard: Right. Today we're talking about IndexNow. I've been calling you Mr. Bingbot, which is probably terribly rude. What's your official title? Fabrice Canel: I am a Principal Program Manager at Microsoft. I've been working at Microsoft for 24 years. Jason Barnard: A very long time. A very interesting time. I was looking you up on LinkedIn and you started at Microsoft in the late nineties. Did you actually start Bingbot? Fabrice Canel: Yes. We started Bingbot together — I even created the Wikipedia page for it. It was the evolution of MSN Bot: we started with MSN Search, then Live Search, then Windows Live Search, then Bing, and now Microsoft Bing. Jason Barnard: You've been on the team the whole time. Have you been programme lead from the beginning? Fabrice Canel: Not at first, but now I'm in charge of managing part of the team delivering the best index on the internet. Jason Barnard: You must be the only person in the industry who has been developing a crawling bot for 24 years or more — since the very beginning of the crawling era of the internet? Fabrice Canel: Yes, in the industry. I've seen all the evolution. When we started, there were maybe 20 pages and it was really easy. Jason Barnard: And now it's infinity. Fabrice Canel: Yes, infinity. The scale of the internet is growing year by year at an unknown rate. Some people estimate 60% year-over-year growth. The frontier keeps expanding because people now generate content using content management systems rather than Notepad, and they can automate the creation of websites entirely. And then you have spammers generating millions and billions of pages of useless content automatically, every day. Jason Barnard: And if it's growing at 60% a year, does that mean you can't keep up? Fabrice Canel: All search engines are managing a business. We are business-driven, obviously to satisfy customers, but we have to pick and choose. We cannot crawl everything on the internet — there are an infinite number of URLs out there. You have pages with calendars that go to the next day forever. It's really about detecting what is most useful to satisfy a Microsoft Bing customer. Jason Barnard: And a large chunk of this content is useless — the infinite calendar problem, spam, spun content. Would you say most content on the web is not useful, or is that exaggerating? Fabrice Canel: That's a little exaggerated. We are guided by key pages that are important on the internet, and we follow links to understand what comes next. If we focus on those key domains and key pages, that guides us to quality content. Our view of the internet is not to go deep forever and crawl useless content. It's about keeping the index fresh and comprehensive — containing all of the most relevant content on the web. Jason Barnard: And the key insight is that you prefer going wide over going deep. So if I have a site that's already near the top, you'll tend to focus on me rather than constantly searching for new things you don't already know about? Fabrice Canel: It depends. If you have a site that is specialised and covers an interesting topic that customers care about, we may absolutely go deep. But this is not me selecting where to go deep — nor my team. It's the machine. Machine learning is selecting to go deep or deeper based on what we feel is important for a Bing customer. Jason Barnard: That's something we tend to lose sight of. I'm talking to Fabrice Canel and I get the idea that you're programming this machine to come and look at my site — but you're not. The machine is deciding for itself. You're just giving it the goals you want it to achieve. Fabrice Canel: Absolutely. The main input we give the machine learning algorithms is: satisfy Bing customers. We look at various dimensions to do that. If you query for Facebook, you want the Facebook link at the top. You don't want some random blog talking about Facebook. Jason Barnard: Very good point. And when you say "customer" rather than "user," that changes our perspective on what we're asking you to do. The search results are a product, and you're thinking about what's good for your customer. Which leads us to the main topic today: IndexNow. How you solve the crawling problem and help manage what gets indexed. But first — a tradition on the show — let's look at your Brand SERP. [Brand SERP segment — Fabrice's personal Brand SERP on Bing is shown, including his photo and a large teddy bear in the image] Jason Barnard: There's a teddy bear in there. Is there a story behind it? Fabrice Canel: It's a family story. When I met my wife, I bought this bear in the US and brought it back. It's a big teddy bear we have at home. Family-related. Jason Barnard: Brilliant. And it's a real teddy bear — that makes it all the more fun. Now, on to IndexNow. Can we start with what it is, how it works, and why it's helpful? Fabrice Canel: Great question. We have a single fundamental problem: my platform and other search engines don't really know when Jason will post the next thing on the internet — the next post, the next video. Jason knows, but I don't. The current model of crawling is to try to figure out when things are changing. When will Jason post again? We can try to model it, but we really don't know. So we pull and pull and crawl and crawl to see if something has changed. We may learn from links, but at the end of the day, we go to the home page and try to figure it out. That model needs to change. We need input from the website owner. Jason can tell us via a simple API that his website content has changed — helping us discover the change, sending the crawler, and getting the latest content. It's an industry shift: from crawling and crawling to discover if something has changed, to the website simply notifying us that something has changed. Jason Barnard: That takes us right back 24 years to when all of this started — when we used to submit our web pages directly to search engines. Are we coming back to that? Fabrice Canel: The early days had the right model, but the wrong implementation: you had to go to a form and submit your URL to each search engine individually. What we've done now is create an API layer — an open interface — that lets you, your content management system, or your CDN tell us what has changed on your website. Ideally, it shouldn't even be the webmaster doing the notifying. It should be the underlying layer — the CMS — telling search engines automatically that something has changed. An API layer is better than a web form because people forget, and people don't want to deal with every search engine separately. This is an open protocol that all search engines can adopt, where websites notify us of every change once. Jason Barnard: If it's integrated into the CMS, it just gets pinged out automatically.