Googlebot

The Googlebot scans the content of countless web pages every second so that they can then be displayed in the search results. Google has two different bots in use to check websites in their desktop and mobile versions.

The many synonyms of the Googlebot

Especially in the world of search engine optimization, the Googlebot has many different names, which can be confusing at first, especially for beginners. Among other things, the bot is known as a crawler. This name comes from the English word “to crawl”. It means something like “crawl”, as the Googlebot “crawls” an web page from top to bottom. This term is one of the better known and is used relatively often.

In addition to Crawler, the Googlebot also goes by the term Spider. This is what the bot was called, especially in the early days of SEO. Similar to the crawler, the term spider was created in an attempt to represent the bot visually. However, it does not crawl from top to bottom, but shimmies down like a spider on its thread, jumps from page to page and thus spins a coherent web of links.

Googlebot sounds like a small program but is a giant

Google’s crawler is often portrayed as a harmless, cute robot. However, the bot is very strong and can even bring websites to their knees that are not prepared for it. This is because it follows every link on a website and examines the URL that is hidden behind the link. The bot is very fast at this and calls up several URLs per second. This can cause some servers to collapse under the load.

But he also can not do everything

The crawler can do a lot, but also has problems with some things. For example, the Googlebot can still only understand images with difficulty and is therefore dependent on so-called alt and title tags. These must be entered by the respective content creator. This not only helps the Googlebot, but also users with impaired vision, who can have all image content read aloud by their output device.

Also JavaScript, which very many pages use, cannot be executed by the bot. Therefore, important content should not be loaded with JavaScript or hidden behind JavaScript. Because the crawler can not find them. Although Google has now developed a bot that can render JavaScript, it always visits the website after the “dumber” bot has examined the page. So it is risky to rely on the “smart” bot. If the first crawler does not find any content because it is all hidden by JavaScript, no signal will be sent to the “smarter” crawler because the page will be considered “empty”.

The Googlebot can be controlled

There are many different ways to control the Google bot to improve the performance of your site. Among other things, the ranking can stand or fall with the Googlebot. After all, it is responsible for indexing and tells the algorithm what content is on the page. If the crawler cannot access pages or content, they will not be indexed and will not be found on the Internet.

Control through internal linking

Since the bot follows every link it finds on a page, this is an effective way to control which subpages are found first and therefore tend to be more important. For this reason, links within the text are also very important to direct the crawler to additional relevant content.

However, too many internal links are harmful. Especially many links in the navigation and footer can bloat any page with a large number of links. This can confuse the crawler, as Google often assigns higher relevance to linked pages.

Robots.txt, nofollow and noindex tags

For the more direct methods to control the Googlebot, some programming knowledge is necessary. In robots.txt you can specify pages that the bot is not allowed to visit. Google adheres to this and will not crawl these pages.

Much more common and easier to use are the two tags nofollow and noindex. With their help, the crawler receives instructions on how to proceed with links and the indexing of a page. The noindex tag is located in the head of the HTML and tells the bot that the entire page should not flow into the index. This way only the relevant pages will be indexed. With nofollow the crawler gets the directive not to follow a link. This helps to control the crawler by allowing it to follow only certain links. Furthermore you don’t give away a page rank. Because with every follow link to another domain a small part of the own page rank is given away.

No Google without Googlebot

Without the industrious Googlebot, the world-famous search engine could not function. Only through this crawler can pages be included in the search results at all. It is therefore all the more important to have a basic knowledge of how the Googlebot works and how it can be controlled. With a few tricks, the ranking can be optimized and the bot can be relieved of some work.

Questions about Googlebot: