Page discovery & queues
MarkupCheck runs a custom concurrent crawling engine written in Go. Scans start by parsing
the target property domain and seed URL. The crawler recursively extracts anchors (<a href>) from HTML pages and feeds discovered links back into the queue database schema.
Preferred URL & redirects
Before a scan starts, MarkupCheck checks the http / https and www / non-www versions of the
root URL and records where they redirect (the Redirect variants card). For
a domain item, the crawl then starts from that preferred URL. For example, if https://example.com/ redirects to https://www.example.com/, the
scan, robots.txt and sitemap lookups all start on www.example.com. A URL prefix
item always starts on its own prefix.
When a crawled URL redirects (301, 302, 307 or 308), it is recorded as a redirect, not as a copy of the page it lands on. The Pages List shows it as 301 → target URL, and the target is queued and scanned as its own page.
Redirect rows:
- are not scored and are left out of averages, summaries, reports, exports and checklists;
- are not sent to Google Search Console URL Inspection (their target is);
- do not count toward your plan's pages-per-scan limit (a scan keeps at most twice the limit in rows, so a site full of redirects still ends).
Use the 301 Redirect status filter in the Pages List to see them. Internal links that point to a redirecting URL are worth updating to the final URL.
Robots.txt & compliance
Before fetching the first seed link, the crawler pulls the target's /robots.txt file.
It validates paths against standard User-Agent directives:
- Obedience: It respects
Disallowblocks targeted at*or specific crawl spiders. When robots.txt answers 5xx or 429, or does not respond, nothing is crawled (RFC 9309); a 4xx answer allows everything. - Sitemaps: Reads
Sitemap:directives in robots.txt as secondary page seed discovery lists. - User-Agent: Identifies itself as
MarkupCheckBot/1.0 (+https://markupcheck.com/bot). The robots.txt token isMarkupCheckBot. See MarkupCheckBot for what it fetches and how to block it.
Scoping & bounds
To prevent infinite loops, the crawler strictly enforces constraints:
- Domain scope: Only crawls pages belonging to the exact property domain (subdomains are excluded unless defined as separate property items).
- Document type: Only indexes content returned with
text/htmlheaders (images and documents like PDFs are inventoried but their internal links are not recursively crawled). - Depth limits: Maximum crawl depth paths are bounded to avoid trailing calendar loops or query parameter traps.
Politeness & rate limiting
To avoid overload on target servers, the crawl workers enforce delays:
- Politeness Delay: Waits at least 100ms between page requests to the same
host name, and each worker pauses 250ms (750ms on Free) before every page. A longer robots.txt
Crawl-delayreplaces that pause. - Concurrency Cap: A scan uses at most 5 workers (2 on Free) to prevent high burst load.
Signed-in users find more about how scans are queued and stored in the app: Crawler engine in the app.