Technical SEO

Technical SEO is the layer that controls whether search engines and AI systems can reach, render, and trust your pages before any content or authority signal matters. It covers crawlability, indexability, sitemaps, URL structure, HTTPS, redirects, and how your pages are rendered. Get this layer wrong and your best content stays invisible; get it right and everything else you do has a foundation to build on.

How the technical layer fits together

Technical SEO sits underneath on-page SEO, structured data, and off-page authority. If a page can't be crawled, rendered, and correctly interpreted, no amount of content quality or link building will surface it.

A useful mental model is a stack of gates. A page must pass each gate in order: it must be reachable (crawlability), allowed into the index (indexability), served over a secure and stable connection, reached through clean redirects, and rendered so its content actually appears in the HTML a crawler receives.

Two shifts define the field in 2026. First, AI search surfaces now depend on the same technical signals as classic search, plus a few AI-specific ones. Second, rendering strategy has become decisive because most third-party AI crawlers don't run JavaScript. For the broader picture, see SEO in 2026, AI Search Engines, and The Agentic Web.

Crawlability: getting bots to your pages

Crawlability starts with robots.txt. Serve one file per origin at the root, keep it under 500 KiB (Google ignores bytes beyond that), and never block CSS or JS that pages need to render. Remember that Disallow prevents crawling but does not deindex — URLs already known can still appear as URL-only listings. To remove a page you need noindex, which requires the page to be crawlable.

AI crawlers now belong in your robots audit. User-agents like GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, and Applebot-Extended deserve explicit policies rather than being ignored. Note that Google-Extended exists only as a robots.txt user-agent — there is no google-extended meta directive, and treating it as one is a common mistake.

A few sharp edges to watch:

• Returning 5xx for /robots.txt makes Google treat the whole site as "disallow everything."
• Google ignores Crawl-delay, but Bing and Yandex honor it — a Crawl-delay: 10 can silently gut Bingbot coverage on a large site.
• Path matching differs: Google applies most-specific-path-wins, Bing uses first-match-wins. Use Disallow: /api/ (with the trailing slash) so you don't accidentally match /apiary.

Crawl budget and the authority bottleneck

Crawl budget only matters for large or frequently-updated sites (roughly 10k+ URLs). Below about 1k pages, focus on quality signals instead. Keep TTFB fast, hold 5xx errors under 1% over any rolling 24-hour window, and use 304 Not Modified responses via If-Modified-Since / ETag — one of the most underused crawl-efficiency wins.

Crucially, Crawled – currently not indexed and Discovered – currently not indexed are almost never crawl-budget problems. Google has reached the URL and is declining to index it because it lacks enough authority to clear the indexing bar. More crawling won't fix that: re-submitting, pinging sitemaps, or bumping <lastmod> won't force indexation, and the Indexing API is restricted to Job Posting and livestream markup — other use is treated as spam. The real fix is authority plus a re-crawl trigger: route internal links from already-indexed, trafficked pages, earn external links, then nudge a re-crawl with a minor change.

Crawl traps — infinite or near-infinite URL spaces — are the other big drain. Faceted navigation combinatorics, session IDs in URLs, calendar pages stretching into the future, pagination that returns 200 for empty pages, and search-result indexing all spawn endless variants. Mitigate with targeted Disallow patterns for non-indexable parameters, 404s beyond real page counts, noindex on empty future pages, and filters that don't generate URLs at all.

Indexability: controlling what gets in

Indexability is governed by canonical tags, robots meta, the X-Robots-Tag header, and hreflang.

Use a single self-referential rel="canonical" in the <head>, pointing to an absolute, 200-status, indexable URL, with tracking parameters stripped. Do not canonicalize paginated pages back to page one — each paginated page should self-canonicalize. And remember that canonical is a hint, not enforcement; when you truly want duplicates gone, use a 301.

For robots meta, the old noindex, follow advice for passing link equity is largely obsolete: Google recrawls noindexed pages less and eventually treats their links as nofollow. AI-specific levers have arrived too — nosnippet and max-snippet:0 now reduce eligibility for AI Overviews, so they're your tool if you want a SERP listing without AI summarization.

Use X-Robots-Tag for non-HTML resources like PDFs and images that have no <head>, and for bulk directory rules. Watch for CDNs that strip custom headers or serve them differently by bot user-agent.

For multilingual sites, hreflang requires reciprocal return links, a self-reference, an x-default fallback, and correct ISO codes (en-GB, not en-UK). Every hreflang URL must be absolute, return 200, and equal the canonical URL. Missing return links are the single most common hreflang error worldwide. See URL structure for how these interact with your address patterns.

Sitemaps: discovery versus authority

XML and HTML sitemaps do different jobs and aren't interchangeable.

An XML sitemap surfaces pages only when your site already has enough authority for Google to want them; it doesn't manufacture indexing. Keep it tight: only canonical, indexable, 200-status URLs, with accurate <lastmod>. Google now uses <lastmod> as a genuine crawl-scheduling signal — but only if it's honest. Setting it to the regeneration timestamp on every URL trains Google to ignore it. Google ignores changefreq and priority entirely.

An HTML sitemap, because it's a real page, is often the cheapest way to push orphaned or "not indexed" pages over the bar on low- and mid-authority sites. Link it from persistent chrome like the footer, and split it into themed hubs on larger sites. When indexing is the problem, prioritize the HTML sitemap plus internal linking over sitemap pinging or changefreq/priority tuning.

Note that llms.txt (a proposed standard) is not a sitemap substitute or a ranking/citation signal — Google's guidance classifies it as unnecessary. The right analogue for helping machines find your content is a sitemap: XML for discovery, HTML for authority.

For large sites, use a sitemap index (one level of indirection only, no nesting) capped at 50,000 entries. Segment child sitemaps by content type so you can isolate which section has indexing problems in Search Console. Image sitemaps now use only <image:loc> — the caption, title, geo, and license tags have been deprecated since 2022, with licensing signaled through structured data instead. News sitemaps must contain only articles from the last 48 hours.

URLs, HTTPS, and redirects

Keep URLs lowercase, hyphen-delimited (underscores read as a single token), descriptive, and shallow. Enforce a single trailing-slash convention and 301 the other variant, since /page and /page/ are distinct URLs that otherwise split signals. Classify parameters into content-changing, presentational, and tracking, keep parameter order consistent (alphabetical, server-side), and never put tracking parameters in canonical tags or internal links. The Search Console URL Parameters tool was removed in 2022, so parameter handling now relies purely on robots.txt, canonicalization, and linking discipline.

On the transport layer, HTTPS is mandatory and HTTP/2 or HTTP/3 aids crawl throughput. Serve the full certificate chain, redirect HTTP to HTTPS in a single hop, and set HSTS. Security headers don't rank directly, but an overly strict Content-Security-Policy can block Googlebot from loading render-critical JS or CSS — verify that Googlebot-fetched resources aren't CSP-blocked.

For redirects, use 301 for permanent moves (or 308 when HTTP method preservation matters) and 302/307 only for genuinely temporary cases. A persistent 302 is eventually treated as a 301, but don't rely on that. Aim for a maximum of one hop — collapse A→B→C→D into direct redirects to the final target — and fix internal links that point at redirecting URLs. Watch for redirect loops caused by force-HTTPS, force-slash, and canonical-host rules interacting.

Rendering and Core Web Vitals

Rendering strategy is where recent advice diverges most. SSR, SSG, and ISR put primary content, headings, links, canonical, meta robots, and structured data into the initial HTML. Client-side rendering (CSR) hides that content behind a render pass. Googlebot uses evergreen Chromium and renders within minutes to hours, so the old "weeks of delay" warning is outdated — but rendering still costs budget.

The decisive point in 2026 is that most third-party AI crawlers do not execute JavaScript; they fetch raw HTML only.

That means CSR content is largely invisible to LLM training and citation pipelines. If AEO and GEO visibility matters, your content must be in server-rendered HTML, and structured data must be server-rendered too — client-injected JSON-LD is sometimes missed and never seen by non-JS crawlers.

Dynamic rendering — serving prerendered HTML to bots and a JS app to users via user-agent sniffing — is officially deprecated by Google and carries cloaking risk. Third-party prerendering services inherit the same classification when they serve bots a different payload; treat them as a temporary stopgap for legacy CSR apps, log a migration path, and watch for stale snapshots, timeouts that serve empty shells, and caches that ignore noindex/canonical changes.

Rendering also drives Core Web Vitals. Current field-data thresholds at the 75th percentile:

INP replaced FID in March 2024, so any report still surfacing FID is measuring a dead metric. Hydration is the dominant INP killer — large bundles that hydrate the whole page block the main thread, while partial or selective hydration materially improves it. Keep the LCP element in the initial HTML, avoid lazy-loading it, and use fetchpriority="high" plus preloads.

Auditing across agents and over time

Because directives, content, and headers can legitimately or illegitimately differ per agent, fetch each page across the full crawler matrix — Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Applebot, CCBot, and a real browser — and diff the results. Any divergence in primary content, headings, links, canonical, meta robots, or JSON-LD between bot and human payloads is a cloaking risk. Confirming that non-JS crawlers receive complete primary content in raw HTML is the single most important GEO render check.

Resolve conflicting signals by precedence: robots.txt disallow first (a disallowed-but-noindexed page stays indexed as URL-only — the top silent indexing bug), then HTTP status, then the most restrictive of X-Robots-Tag and meta robots, then canonical, then hreflang. Weight fixes by signal loss, not raw count — a site-wide noindex or render-blocked primary content is critical, while changefreq cruft is cosmetic.

Finally, single-snapshot audits miss the most damaging failures, which are deployment-induced and transient. Track per-URL state over time to catch sudden noindex spikes, robots.txt changes, canonical shifts, fabricated lastmod, render regressions, and crawl-rate drops before rankings suffer.

What to do

1. Confirm robots.txt returns 200, stays under 500 KiB, blocks no render resources, and sets explicit policies for AI crawlers.
2. Verify each indexable page has one self-referential canonical to a 200-status URL, with tracking parameters stripped and paginated pages self-canonicalizing.
3. Diagnose "not indexed" pages as an authority problem: add internal links from indexed pages and build an HTML sitemap, rather than pinging or resubmitting.
4. Keep the XML sitemap free of noise and make <lastmod> honest; drop changefreq and priority.
5. Enforce one trailing-slash convention, consistent parameter ordering, and single-hop 301s with no chains or loops.
6. Serve HTTPS with a full cert chain and HSTS, and confirm your CSP doesn't block render-critical resources for Googlebot.
7. Move indexed and cited content to SSR, SSG, or ISR, with server-rendered structured data, and retire any dynamic rendering or prerendering setup.
8. Check Core Web Vitals against the current thresholds, treating hydration cost as your primary INP lever.
9. Fetch every page across the full crawler matrix, diff bot versus human payloads, and confirm non-JS crawlers get complete raw HTML.
10. Re-audit over time and prioritize fixes by signal loss so transient, deployment-induced failures surface before they cost rankings.