Loading speed, web and servers
crawler
In short and in plain language
A crawler is a program that fetches websites on its own.
It follows links from page to page. Search engines do this, for example.
Each request gives the server work, just like a visit. But crawlers often come in large numbers. This can slow the server down.
Definition
A crawler is a program that fetches websites automatically and follows links, for example for search engines, AI services or audit tools. Every request puts load on the server like a visit, but often in large numbers and quick succession. Deep listing and filter pages can therefore cost an unexpected amount of computing time.