Crawling & Indexing (robots.txt / sitemap)

[← Back to Home](/docs/rules)

Checks robots.txt syntax and reachability, and sitemap.xml/sitemap-index validity, size limits, and content — the two files that control whether crawlers (search or AI) can discover and index a site at all. Source: src/lib/crawlAudit.ts (analyzeCrawlFiles).

34 rules in this category.

---

robots-disallow-all

robots.txt blocks all crawlers

What triggers it & why it matters: User-agent: * includes Disallow: /, which tells search engines not to crawl any pages on the site.

How to fix it & effect on score: Remove or narrow the blanket Disallow: / rule unless the entire site should stay out of search results. Fixing this removes the 15 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-empty

Sitemap contains no URLs

What triggers it & why it matters: N parsed as a urlset but has zero <url> entries, so it cannot help crawlers discover pages.

How to fix it & effect on score: Populate the sitemap with <url>``<loc>…</loc>``</url> entries for indexable pages. Fixing this removes the 10 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-fetch

Could not fetch XML sitemap

What triggers it & why it matters: Failed to load N. Crawlers may also struggle to retrieve it.

How to fix it & effect on score: Ensure the sitemap URL is publicly accessible and returns HTTP 200. Fixing this removes the 10 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-http

Sitemap returned HTTP (actual HTTP status code)

What triggers it & why it matters: N did not return a successful response, so crawlers cannot use it.

How to fix it & effect on score: Fix the sitemap endpoint to return HTTP 200 with valid XML. Fixing this removes the 10 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-not-xml

Sitemap response is not XML

What triggers it & why it matters: The sitemap URL returned content that does not look like XML (no leading < tag).

How to fix it & effect on score: Serve a valid XML sitemap at this URL. Fixing this removes the 10 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-parse

Sitemap XML could not be parsed

What triggers it & why it matters: The sitemap file appears malformed and could not be parsed as XML.

How to fix it & effect on score: Fix XML syntax errors and ensure the file is well-formed. Fixing this removes the 10 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-root

Invalid sitemap root element

What triggers it & why it matters: Expected <urlset> or <sitemapindex> as the root element, but found <(actual value)>.

How to fix it & effect on score: Use a standards-compliant sitemap structure per sitemaps.org. Fixing this removes the 10 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-index-empty

Sitemap index contains no child sitemaps

What triggers it & why it matters: N is a sitemap index but lists no <sitemap> entries.

How to fix it & effect on score: Add <sitemap>``<loc>…</loc>``</sitemap> entries pointing to child sitemap files. Fixing this removes the 9 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-missing-loc

N sitemap entry is/ies are missing <loc>

What triggers it & why it matters: Every <url> entry must include an absolute <loc> URL. Entries without one are ignored by crawlers.

How to fix it & effect on score: Add a valid absolute URL inside <loc> for every <url> entry. Fixing this removes the 9 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

robots-error

robots.txt returned HTTP (actual HTTP status code)

What triggers it & why it matters: Crawlers expect a plain-text robots.txt at the site root. A non-200 response can cause unpredictable crawl behavior.

How to fix it & effect on score: Fix the server response for /robots.txt so it returns HTTP 200 with valid plain text. Fixing this removes the 8 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-index-missing-loc

N sitemap index entry is/ies are missing <loc>

What triggers it & why it matters: Each <sitemap> entry in an index must include an absolute <loc> URL.

How to fix it & effect on score: Add valid <loc> URLs for every child sitemap. Fixing this removes the 8 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-index-size

Sitemap index lists N child sitemaps (limit is 50,000)

What triggers it & why it matters: A sitemap index may reference at most 50,000 sitemaps.

How to fix it & effect on score: Reduce the number of child sitemaps or split across multiple index files. Fixing this removes the 8 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-invalid-loc

N sitemap URL(s) is/are not valid absolute URLs

What triggers it & why it matters: Sitemap <loc> values must be full http:// or https:// URLs, not relative paths.

How to fix it & effect on score: Use absolute URLs in every <loc> element. Fixing this removes the 8 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-missing

No XML sitemap found

What triggers it & why it matters: No sitemap was declared in robots.txt and common paths like /sitemap.xml did not return a valid file.

How to fix it & effect on score: Create an XML sitemap, validate it, and add a Sitemap: line to robots.txt. Fixing this removes the 8 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-size

Sitemap lists N URLs (limit is 50,000)

What triggers it & why it matters: The sitemaps.org protocol allows at most 50,000 URLs per file. Larger sets must be split.

How to fix it & effect on score: Split the sitemap into multiple files and reference them from a sitemap index. Fixing this removes the 8 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

robots-fetch

Could not fetch robots.txt

What triggers it & why it matters: Failed to load N. Crawlers may not discover your sitemap or crawl rules.

How to fix it & effect on score: Ensure /robots.txt is publicly accessible. Fixing this removes the 7 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

robots-no-groups

robots.txt has no User-agent blocks

What triggers it & why it matters: The file contains text but no recognizable User-agent groups, so crawlers may ignore the rules entirely.

How to fix it & effect on score: Structure robots.txt with User-agent and Disallow/Allow directives. Fixing this removes the 7 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-file-size

Sitemap file is N MB (limit is 50 MB uncompressed)

What triggers it & why it matters: Sitemaps larger than 50 MB uncompressed should be split into smaller files.

How to fix it & effect on score: Split the sitemap and reference the parts from a sitemap index. Fixing this removes the 7 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-index-invalid-loc

N child sitemap URL(s) is/are invalid

What triggers it & why it matters: Child sitemap URLs must be absolute http:// or https:// addresses.

How to fix it & effect on score: Fix the <loc> URLs in the sitemap index. Fixing this removes the 7 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-index-unreachable

N referenced child sitemap(s) could not be reached

What triggers it & why it matters: At least one sitemap listed in the index returned a non-200 response or failed to load.

How to fix it & effect on score: Ensure every child sitemap URL returns HTTP 200 and valid XML. Fixing this removes the 7 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-off-domain

Sitemap URLs are all on a different domain

What triggers it & why it matters: All N URLs point outside (actual value). Cross-domain sitemaps are usually a misconfiguration for the scanned site.

How to fix it & effect on score: Point the sitemap at URLs on the same host as the site being analyzed. Fixing this removes the 7 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

robots-missing

No robots.txt found

What triggers it & why it matters: No robots.txt file was returned at (actual value)/robots.txt. Search engines will crawl freely, but you lose a place to declare sitemap locations and crawl rules.

How to fix it & effect on score: Add a robots.txt at the site root with crawl rules and Sitemap directives. Fixing this removes the 6 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

robots-no-sitemap

robots.txt does not declare a sitemap

What triggers it & why it matters: No Sitemap: directive was found. Declaring sitemap URLs in robots.txt helps crawlers discover them reliably.

How to fix it & effect on score: Add Sitemap: https://yoursite.com/sitemap.xml to robots.txt. Fixing this removes the 6 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

robots-empty

robots.txt is empty

What triggers it & why it matters: The file exists but contains no directives, so crawlers get no guidance on sitemap location or restricted paths.

How to fix it & effect on score: Add User-agent rules and at least one Sitemap: line pointing to your XML sitemap. Fixing this removes the 5 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-duplicates

N duplicate URL(s) in sitemap

What triggers it & why it matters: The same URL appears more than once, which wastes crawl budget and can confuse indexers.

How to fix it & effect on score: Remove duplicate <loc> entries so each URL appears once. Fixing this removes the 5 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

robots-malformed

robots.txt contains unrecognized directives

What triggers it & why it matters: Found N line(s) with unknown directives (e.g. "(actual value)"). Crawlers may skip invalid rules.

How to fix it & effect on score: Use only standard directives: User-agent, Disallow, Allow, Sitemap, and Crawl-delay. Fixing this removes the 4 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-lastmod

N sitemap entry has/ies have invalid lastmod dates

What triggers it & why it matters: lastmod values should use ISO 8601 format (e.g. 2024-06-15 or 2024-06-15T08:00:00Z).

How to fix it & effect on score: Fix or remove invalid <lastmod> values. Fixing this removes the 4 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

sitemap-mixed-domain

N sitemap URL(s) on a different domain

What triggers it & why it matters: Some entries reference external domains. That can be intentional, but often indicates stale or copied sitemap content.

How to fix it & effect on score: Keep sitemap URLs on the same domain unless you intentionally index external pages. Fixing this removes the 4 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

robots-content-type

robots.txt has an unusual Content-Type

What triggers it & why it matters: The file was served as "(actual value)". robots.txt should ideally be text/plain.

How to fix it & effect on score: Serve robots.txt with Content-Type: text/plain; charset=utf-8. Fixing this removes the 3 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

---

robots-present

robots.txt is present and readable

What triggers it & why it matters: This rule only ever surfaces as a passing signal in the current source — it fires (as a pass) when: robots.txt is present and readable. There is no separate failing issue() call under this exact id; the failure condition is the logical negation of the pass condition, and in most cases is covered by a related rule id documented elsewhere in this category (see the table above for a sibling rule on the same feature).

Effect on score: Meeting this condition costs nothing and confirms a positive signal. Missing it doesn't directly deduct points under this id — it typically shows up as a deduction under the related failing rule instead.

---

robots-sitemap-declared

robots.txt declares N sitemap(s)

What triggers it & why it matters: This rule only ever surfaces as a passing signal in the current source — it fires (as a pass) when: robots.txt declares N sitemap(s). There is no separate failing issue() call under this exact id; the failure condition is the logical negation of the pass condition, and in most cases is covered by a related rule id documented elsewhere in this category (see the table above for a sibling rule on the same feature).

Effect on score: Meeting this condition costs nothing and confirms a positive signal. Missing it doesn't directly deduct points under this id — it typically shows up as a deduction under the related failing rule instead.

---

sitemap-index-valid

Sitemap index is valid (N child sitemap(s))

What triggers it & why it matters: This rule only ever surfaces as a passing signal in the current source — it fires (as a pass) when: Sitemap index is valid (N child sitemap(s)). There is no separate failing issue() call under this exact id; the failure condition is the logical negation of the pass condition, and in most cases is covered by a related rule id documented elsewhere in this category (see the table above for a sibling rule on the same feature).

Effect on score: Meeting this condition costs nothing and confirms a positive signal. Missing it doesn't directly deduct points under this id — it typically shows up as a deduction under the related failing rule instead.

---

sitemap-valid

Sitemap is valid (N URL(s))

What triggers it & why it matters: This rule only ever surfaces as a passing signal in the current source — it fires (as a pass) when: Sitemap is valid (N URL(s)). There is no separate failing issue() call under this exact id; the failure condition is the logical negation of the pass condition, and in most cases is covered by a related rule id documented elsewhere in this category (see the table above for a sibling rule on the same feature).

Effect on score: Meeting this condition costs nothing and confirms a positive signal. Missing it doesn't directly deduct points under this id — it typically shows up as a deduction under the related failing rule instead.

---

sitemap-lastmod-missing

Only N of M sitemap URLs have a lastmod date

What triggers it & why it matters: Fewer than half the sitemap URLs carry . Search engines and AI crawlers use it to decide what to refetch, so updates show up late. (geo-score counts sitemap freshness as an AI-readiness signal.)

How to fix it & effect on score: Have the CMS or sitemap generator add an accurate to every URL. Fixing this removes the 2 pts deduction from the Crawling & Indexing (robots.txt / sitemap) category score on the next scan.

Evidence: official requirement (documented by the engine or standard).

---