How does Googlebot work?

Googlebot is a crawler (a User-Agent). These programs, also called web crawlers or SEO spiders, analyze pages and fetch their resources. It is only a computer program, but knowing how it works is central to organic search.

In plain English, “crawl” is the way Googlebot walks the World Wide Web, page by page.

Google’s SEO spider: what it does

Starting from a directory of URLs it has already crawled, plus the sitemap data webmasters submit (the site’s structure map), its job is to:

• Walk and download each website’s data by following links, from one page to the next.

• Decide which site to visit, when, and how many pages are tied to it.

• Record new domains to index in search engines, site updates, links to URLs and images, and dead links.

• Use that data to update the Google index, and add the URLs to the list mentioned above.

Google’s search index is the directory of billions of websites made available to users. New content has to be indexed before it can appear in the SERP (Search Engine Results Pages). Indexing is what makes a piece of content (a domain, a site, a page, an article, and so on) reachable from search results. Googlebot only passes through. It does not vouch for the pages it records; that is the Google index’s job.

This program is the starting point of organic search for websites, the first step before content can appear in a search engine. The bot runs in parallel, in a matter of seconds, from machines close to the site’s server. That setup keeps performance and reach up, and above all it avoids clogging the bandwidth of the servers being crawled. About a million servers are used to crawl the web, so a bot’s IP is not fixed.

User experience, crawl frequency, and crawl budget

The User-Agent treats user experience as non-negotiable

The bot explores a site the way a user would, at a frequency it sets itself (based on your site’s popularity, your technical choices, and so on). It always tries not to harm the user experience. It then tests the site and sets a cap on “crawl rate” (how many machines crawl at once, and the wait between two passes) based on:

• The server connection. If the site responds quickly for a long stretch, the cap goes up. If the site is slow to respond, or hits any server error, the cap goes down.

• The crawl-rate limit webmasters set in Search Console. Raising it does not automatically intensify the crawl, because Googlebot still sets its own actions.

A crawl budget you can take lightly

Crawl budget is the mix of crawl frequency and the amount of crawl resources allocated. The resources available for a crawl are the number of simultaneous connections to a URL and the pause between two visits.

In many cases, a website is seen on the day it goes live. Google says crawl budget is something you can take lightly.

More resources are allocated to a site that is:

• Popular, so the Google index can be kept up to date.

• Migrating, given the large number of URLs that need indexing.

• At risk of going stale. Google cares about user experience, so it wants to de-index abandoned sites.

According to Google, if your domain has fewer than several thousand URLs, it will be crawled efficiently. On the other side, for sites or pages that have never been involved in an indexing request, Googlebot will restrict its crawl even if it has not hit its cap.

If you run a large site, you need to measure how crawlable it is: how often it is crawled, and which resources are allocated to that crawl. Depending on crawl frequency, Googlebot will take a larger or smaller share of your web server’s bandwidth, so you need to know that frequency relative to your traffic.

Set Googlebot’s actions yourself

Google uses many User-Agents, including Googlebot, to act on the web. You can tell them what to do. Here is the list of the different User-Agents:

Source: https://support.google.com/webmasters/answer/1061943?hl=fr&ref_topic=4610900

There are three ways to give them directives:

Get to know the robots.txt file

You can tell a User-Agent to stay off your site, but blocking Googlebot in robots.txt is the same as blocking all of them.

The first two lines give it full access; the next two do the opposite.

Finally, you point it to the location of the sitemap data.

As mentioned earlier, Googlebot records every link, then follows them and catalogs whatever is on the page. That said, you can tell it whether it should follow a given link.

A note on robots.txt is in order. It is the first file Google reads when it crawls a site. It sits at the very top of the server folder, and that location is decisive: if it is not exactly at the top, it does not exist as far as Google is concerned.

If the robots.txt file with the User-Agent rules is on your server, it improves your credibility in search. Google also flags you if the file is missing, with the note: “robots.txt – file not found”.

Important note: links marked “follow” or “nofollow” will be recorded and visited without distinction if there is no robots.txt file on the server.

HTTP X-Robots-Tag instructions

Tell robots not to index this page while still giving them access to the content.

You can combine several instructions.

Note: if the index attribute is set, the full content and the images will be indexed.

Meta tags

As with the X-Robots-Tag, these meta tags tell robots whether to crawl and index your page.

Directives for every User-Agent, with no distinction.

Do not index the images on the page.

In Search Console, you can check the crawl-error page. That is where every crawl error Googlebot hits is logged. You can then fix them and move on.

Know that any error hurts your organic search (SEO), and therefore your visibility. You will never reach zero errors, but Google recommends reading these reports regularly. Its support site is also full of information, advice, and tools on crawling.

Googlebot is the starting point of your organic search, so it is worth studying if you want a sound SEO strategy. Our SEO agency is made of people who watch how Googlebot works every day and can help you on SEO projects.