Crawlability

Crawlability is whether search engine and AI bots can reach, fetch, and render your URLs at all. It depends on a correct robots.txt, healthy server responses, clean status codes, and a URL space that doesn't trap crawlers in infinite loops. Get these right and bots spend their limited attention on the pages that matter — get them wrong and even great content never gets seen.

Getting robots.txt right

Your robots.txt lives at the root of each host — https://example.com/robots.txt — and there's exactly one per origin (scheme + host + port). Because https://www.example.com and https://example.com count as separate origins, you usually consolidate them with a 301 to your canonical host. Google honors up to 500 KiB; anything past that is ignored.

Write User-agent blocks specific to the bots you care about, then close with a generic User-agent: * block. Matching behavior differs by engine: Google applies the most specific path, regardless of order in the file, while Bing follows first-match-wins. If you serve international audiences, audit against both interpretations. An Allow: only overrides a Disallow: when its path is more specific (a longer match). Always end the file with absolute Sitemap: directives.

A block that combines site policy with per-bot rules might look like this:

User-agent: Googlebot
Disallow: /search
Disallow: /*?sort=
Allow: /search/static-landing/

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /premium/

User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /checkout

Sitemap: https://example.com/sitemaps/index.xml

See XML & HTML Sitemaps for how the sitemap side of this fits together, and URL Structure for keeping the paths you're matching predictable.

robots.txt mistakes that quietly hurt you

• Blocking rendering resources. Disallowing /wp-content/ or /static/ can cut off the CSS and JS Google needs to render. Never block resources required for rendering.
• Assuming Disallow deindexes. It doesn't. It prevents crawling, but URLs Google already knows can still appear as URL-only listings. Removal requires noindex, which itself requires the page to be crawlable. This is the boundary between crawlability and Indexability.
• Setting Crawl-delay. Google ignores it; Bing and Yandex honor it. A Crawl-delay: 10 on a large site can silently destroy Bingbot's coverage.
• Path-prefix surprises. Disallow: /api matches both /api/v1/users and /apiary. Use Disallow: /api/ for path-segment safety.
• A BOM character at the start of the file can break some parsers.
• Returning 5xx for /robots.txt. Google treats sustained 5xx as "disallow everything," so a broken robots file can wall off your whole site. Fix non-200 responses fast.

Crawl budget vs. indexing authority

Crawl budget is crawl rate limit × crawl demand. It matters for sites larger than about 10,000 URLs or with frequent updates. For sites under 1,000 pages, crawl budget is essentially never the bottleneck — focus on quality signals instead.

Here's the distinction that trips people up. Statuses like Crawled – currently not indexed and Discovered – currently not indexed are almost never crawl-budget problems. Google has already reached the URL (or deliberately deprioritized fetching it) and is declining to index it because the URL lacks enough authority (PageRank) to clear the indexing bar — not because it ran out of crawl quota.

The consequences:

• More crawling won't fix it. Re-submitting, pinging sitemaps, or bumping <lastmod> will not force indexation of a low-authority URL.
• Don't reach for the Google Indexing API for ordinary pages — Google restricts it to Job Posting and livestream markup and treats other use as spam.
• The real fix is authority plus a re-crawl trigger: route Internal Linking to the URL from already-indexed, trafficked pages, earn external links through Off-Page Authority, then nudge a re-crawl with a minor content or slug change plus a fresh internal link from an indexed page.

This holds at any site size, and it's separate from crawl-*rate* throttling, which is a server-load control rather than an indexing lever.

Keeping crawl efficient

When crawl budget does matter, efficiency comes down to fast, honest server responses:

• Keep HTML TTFB under 200ms. Google documents sub-second as "fast," and TTFB over 1s measurably reduces crawl rate. Core Web Vitals covers the wider performance picture.
• Keep the 5xx error rate under 1% over a rolling 24 hours. Sustained 5xx above 5% causes Googlebot to back off within hours.
• Return accurate status codes: 200 for success, 410 Gone for permanent removal (Google removes it roughly twice as fast as a 404), 301 for moves — see Redirects for SEO — and 503 with Retry-After for planned maintenance.
• Use If-Modified-Since / ETag correctly. Returning 304 Not Modified for unchanged content is the single most underused crawl-budget optimization.
• Keep XML sitemaps tight: only canonical, indexable, 200-status URLs. A sitemap that's 30% noise burns budget.

Common efficiency leaks include soft 404s (a 200 status serving "not found" content), internal links pointing at non-canonical or redirected URLs, accidentally blocking /sitemap.xml with a rule like Disallow: /*.xml$, and slow APIs powering server-side rendering — if a /products/[id] page depends on a 2s API call, crawl rate collapses.

To check your own state, analyze server logs for requests per day per bot and the 2xx/3xx/4xx/5xx distribution, then compare crawled URLs in the last 30 days against total canonical URLs in your sitemap. A ratio under 50% signals a coverage problem. Watch for bloated HTML over 500KB and confirm Last-Modified/ETag headers are present.

Crawl traps: infinite URL space

A crawl trap is an infinite or near-infinite URL space that bots can wander into. A single category page can spawn millions of variants, exhausting crawl capacity on junk.

To mitigate:

• Add Disallow: /*?sort= and similar patterns for non-indexable parameters.
• Use rel="canonical" to consolidate variants (it helps consolidation but doesn't save crawl).
• For pagination, return 404 beyond the real page count instead of a 200 with empty content.
• For calendars, only generate the next-month link when content exists, and noindex empty future-month pages.
• Consider nofollow on internal facet links — but use it cautiously, since it affects PageRank flow.
• For filters that shouldn't create URLs at all, drive them with data-* attributes and JS.

Signs of a trap in your logs: URL patterns where unique URL count grows superlinearly with crawl count, path depth beyond 7, parameters with more than 100 unique values, future-dated URLs, and repeating path segments like /a/b/a/b.

What to do

1. Confirm robots.txt returns a 200 with text/plain, sits under 500 KiB, and lists absolute Sitemap: URLs that all resolve to valid XML.
2. Render-test key templates to prove no CSS, JS, or font directory is blocked.
3. Build a per-bot policy for the crawlers you care about (Googlebot, Bingbot, and AI agents like GPTBot, ClaudeBot, and PerplexityBot) and verify rules against both Google and Bing matching logic.
4. Replace any Disallow-based "removal" with noindex where you actually want pages gone.
5. Audit server logs for TTFB percentiles, 5xx rate, and the crawled-vs-sitemap coverage ratio; fix slow SSR paths and soft 404s.
6. Add ETag/Last-Modified support and serve 304 for unchanged content.
7. Hunt crawl traps with pattern, depth, and parameter-cardinality analysis, then block or collapse them.
8. For "discovered/crawled – not indexed" URLs, build authority through internal and external links plus a re-crawl trigger — don't just re-submit.