There has been some confusion about how Google deals with pages that may be more expensive to crawl, render, index and serve to Google Search – that is, JavaScript pages. Google does not have a monetary budget per site, meaning it will spend $X of the crawl budget on your site.
Yes, websites have a crawl budget, but not in terms of costs, but rather in terms of resources. You can check out Google’s official documentation on this topic.
Google’s Martin Splitt said on LinkedIn: “We don’t keep records of how expensive this page was for us or anything like that.” He added: “You don’t have to worry about the rendering being expensive, we take care of that.”
Martin goes on to explain that crawling is expensive, as are other parts of search. Google’s ultimate goal is to show the most relevant result, no matter how expensive that result may be. “The goal of Google Search is to provide users with relevant content for their searches. We’re not perfect at doing this for all searches, all the time, all the time, but let’s focus on JavaScript for a moment here,” he wrote.
“Google Search does a lot of things that are complicated and expensive (storage, bandwidth, teams around the world to keep everything running 24/7), etc. – JavaScript is just a tiny part of that,” he explains.
Martin then explained that JavaScript is part of the web and will likely continue to grow in importance. “We know that a significant part of the web uses JavaScript to add, remove, or modify content on web pages. We only need to render them to see everything. It doesn’t matter whether a page uses JavaScript or not, because we can only be reasonably sure of seeing all the content if it is rendered,” he wrote.
So just because more “expensive” pages need to be crawled doesn’t mean that Google won’t crawl them – the company will.
Forum discussion on LinkedIn.
