Skip to main content

Page discovery & queues

MarkupCheck runs a custom concurrent crawling engine written in Go. Scans start by parsing the target property domain and seed URL. The crawler recursively extracts anchors (<a href>) from HTML pages and feeds discovered links back into the queue database schema.

Preferred URL & redirects

Before a scan starts, MarkupCheck checks the http / https and www / non-www versions of the root URL and records where they redirect (the Redirect variants card). For a domain item, the crawl then starts from that preferred URL. For example, if https://example.com/ redirects to https://www.example.com/, the scan, robots.txt and sitemap lookups all start on www.example.com. A URL prefix item always starts on its own prefix.

When a crawled URL redirects (301, 302, 307 or 308), it is recorded as a redirect, not as a copy of the page it lands on. The Pages List shows it as 301 → target URL, and the target is queued and scanned as its own page. Redirect rows:

  • are not scored and are left out of averages, summaries, reports, exports and checklists;
  • are not sent to Google Search Console URL Inspection (their target is);
  • do not count toward your plan's pages-per-scan limit (a scan keeps at most twice the limit in rows, so a site full of redirects still ends).

Use the 301 Redirect status filter in the Pages List to see them. Internal links that point to a redirecting URL are worth updating to the final URL.

Robots.txt & compliance

Before fetching the first seed link, the crawler pulls the target's /robots.txt file. It validates paths against standard User-Agent directives:

  • Obedience: It respects Disallow blocks targeted at * or specific crawl spiders. When robots.txt answers 5xx or 429, or does not respond, nothing is crawled (RFC 9309); a 4xx answer allows everything.
  • Sitemaps: Reads Sitemap: directives in robots.txt as secondary page seed discovery lists.
  • User-Agent: Identifies itself as MarkupCheckBot/1.0 (+https://markupcheck.com/bot). The robots.txt token is MarkupCheckBot. See MarkupCheckBot for what it fetches and how to block it.

Scoping & bounds

To prevent infinite loops, the crawler strictly enforces constraints:

  • Domain scope: Only crawls pages belonging to the exact property domain (subdomains are excluded unless defined as separate property items).
  • Document type: Only indexes content returned with text/html headers (images and documents like PDFs are inventoried but their internal links are not recursively crawled).
  • Depth limits: Maximum crawl depth paths are bounded to avoid trailing calendar loops or query parameter traps.

Politeness & rate limiting

To avoid overload on target servers, the crawl workers enforce delays:

  • Politeness Delay: Waits at least 100ms between page requests to the same host name, and each worker pauses 250ms (750ms on Free) before every page. A longer robots.txt Crawl-delay replaces that pause.
  • Concurrency Cap: A scan uses at most 5 workers (2 on Free) to prevent high burst load.

Signed-in users find more about how scans are queued and stored in the app: Crawler engine in the app.

← Back to MarkupCheck Dashboard