What MarkupCheckBot does
MarkupCheckBot is the crawler of MarkupCheck, a technical SEO audit and schema markup checker. It visits a website only when a MarkupCheck user starts a scan of that website, or on a daily, weekly or monthly schedule that a user who verified the website as their own turned on (Premium). It reads each page the way a search engine would and checks titles, headings, canonical tags, structured data, images, links and other technical SEO signals. The results are shown only to the user who started or scheduled the scan.
MarkupCheckBot does not:
- build a search index or a public archive of your pages;
- collect content to train AI models;
- sell or share the pages it reads;
- fill in forms, sign in, add items to carts or send anything other than GET and HEAD requests;
- run JavaScript or load ads and tracking scripts.
How to recognise it
Every request from MarkupCheckBot sends this User-Agent header:
MarkupCheckBot/1.0 (+https://markupcheck.com/bot)
What it fetches
- robots.txt, before any page, to read the rules and the
Sitemap:lines. - XML sitemaps, to find the pages to scan.
- HTML pages of the scanned website, plus the links it finds on them. It stays on the scanned host and never follows links to other websites.
- Links to other websites, once per scan: a HEAD request (or a GET when the HEAD answer is unclear) to each external link found on the scanned pages, up to 500 per scan, only to check that the linked page still exists. It reads nothing from those pages and never crawls them.
- A few related files for each page: the favicon, the share image (
og:image), the web app manifest, and a HEAD request (or a one-byte GET) to linked PDF files to check that they load, and a HEAD request to up to 30 images per page to read their file size. - The http, https, www and non-www versions of your home page, once per scan, to check where they redirect.
Each request times out after 10 seconds.
How it obeys robots.txt
MarkupCheckBot follows the robots.txt standard (RFC 9309). Its product token is MarkupCheckBot. It uses the group that names it when there is one, and the User-agent: * group otherwise. It does not scan pages that the rules
disallow, and it honours Crawl-delay. If robots.txt answers with a server
error (5xx or 429) or does not respond, MarkupCheckBot crawls nothing; a missing
robots.txt (4xx) allows everything.
To keep MarkupCheckBot out of your whole website, add this to your robots.txt:
User-agent: MarkupCheckBot
Disallow: /
To keep it out of one folder only:
User-agent: MarkupCheckBot
Disallow: /private/
Blocking MarkupCheckBot does not affect Google, Bing or any other crawler. If you run MarkupCheck on your own website, a block also stops your own scans.
Every scan also tests the homepage with Google PageSpeed Insights. Those requests come from Google's own servers, not from MarkupCheckBot, so this robots.txt rule does not apply to them.
The one-time check of a link from a scanned website to yours is a single HEAD or GET
request for that page, like a link checker, and does not read your robots.txt. A Disallow rule for MarkupCheckBot on your website does not stop those checks, but
it does stop MarkupCheckBot from scanning your pages.
How fast it crawls
MarkupCheckBot is built to stay light on your server:
- It waits at least 100 ms between page requests to the same host, so a scan never sends more than about 10 pages per second.
- A scan uses at most 5 connections at once (2 on the Free plan), and each connection pauses 250 to
750 ms before every page. If your robots.txt sets a longer
Crawl-delay, each connection waits that long instead. - A scan reads at most 2,000 pages. Websites the user has not verified as their own get at most 100 pages.
- Scans run only when a user starts them, or on a schedule for a website the user verified as their own, and each user has a daily scan limit.
Contact
If MarkupCheckBot causes a problem on your website, or you have a question about it, write to us at support [at] markupcheck [dot] com. Please include your domain and the time of the requests, and we will look into it.
How the crawler finds pages and stays in scope is explained in more detail in the crawler documentation.