Indexability

Indexability is whether a search engine is allowed to store a page in its index and show it in results. It's controlled by signals like canonical tags,…

7 min read · updated 2026-08-08

Indexability is whether a search engine is allowed to store a page in its index and show it in results. It's controlled by signals like canonical tags, noindex directives, X-Robots-Tag headers, and hreflang annotations — get any of these wrong and a page you want ranking can quietly disappear, or the wrong version can rank in its place. Indexability is separate from crawlability: a page can be crawlable but still blocked from the index, and directives can conflict in ways that produce results you didn't intend.

Canonical tags

A canonical tag tells search engines which URL is the preferred version of a page. Use <link rel="canonical" href="https://example.com/page"> in the <head>, with an absolute URL and exactly one instance per page. Add a self-referential canonical to every indexable page — yes, even when the canonical equals the current URL.

The canonical target must return a 200 status, be indexable (not noindex), contain substantively similar content, and ideally be linked to internally. Cross-domain canonicals are valid for syndication scenarios, but they must be honored consistently. For non-HTML resources like PDFs, use the HTTP header form: Link: <https://example.com/page>; rel="canonical".

Remember that a canonical is a hint, not an enforcement mechanism. If you need to consolidate duplicates permanently, a 301 redirect is the enforcing tool; canonical only suggests.

Common canonical mistakes

Pointing the canonical at a redirected URL — Google may then pick a different canonical on its own.
Pointing it at a noindex'd URL — the signals conflict, so Google ignores it.
Having multiple canonicals (for example, one value in the HTML and a different one in the HTTP header).
Using relative canonicals — technically valid but error-prone behind reverse proxies.
Placing the canonical in the <body> — invalid and ignored.
Canonicalizing paginated pages (?page=2) back to page 1. Advice in 2024 already discouraged this; by 2026 it's a clear error. Each paginated page should self-canonicalize.
Leaving tracking parameters like ?utm_* in the canonical — strip them.
Mixed-case host or trailing-slash inconsistency between the canonical and the actual URL.

When you review canonicals, confirm there's exactly one in the <head>, the URL is absolute and HTTPS, it returns 200 and is indexable, the host matches your preferred host (www vs apex), and the target appears in your XML sitemap. Watch for canonical chains (A→B→C) — Google usually follows them, but they're fragile, so flag them.

noindex and nofollow

<meta name="robots" content="noindex"> blocks a page from being indexed. nofollow in a robots meta tag tells engines not to follow any links from that page. You can target specific bots with directives like <meta name="googlebot" content="noindex">.

The old pattern of noindex, follow to preserve link equity is largely obsolete. Google has stated that noindex pages are recrawled less often, so over the long term their links get treated the same as noindex, nofollow. Don't count on follow to keep passing equity.

AI and snippet controls in 2026

<meta name="robots" content="noai, noimageai"> is respected by some AI crawlers (Adobe among them) but not by Google.
<meta name="google-extended" content="noindex"> does not exist as a meta directive. Google-Extended is a robots.txt user-agent only — a common mistake to watch for.
nosnippet / max-snippet:0 now reduces eligibility for AI Overviews. If you want a standard SERP listing but not AI summarization, that's the lever to pull. See AEO & GEO for how this interacts with AI answer surfaces.

Common noindex mistakes

Applying noindex to a URL that is also disallowed in robots.txt. Google can't crawl it, so it never sees the noindex, and the URL can remain indexed as a URL-only entry.
Shipping a staging noindex to production. Periodically crawling clones like staging.example.com is good practice.
Adding noindex to pages that have inbound external links, which orphans that equity — see Off-Page Authority.
Conflicting directives, such as <meta name="robots" content="index"> plus an X-Robots-Tag: noindex header. The most restrictive directive wins.
Using noindex instead of 401/403 or real authentication for genuinely private content.
Slapping noindex on thin category or tag pages instead of fixing or consolidating the content.

Track noindex coverage over time; a sudden spike usually means a deployment problem. Also confirm noindex'd URLs aren't sitting in your sitemap or being internally linked from high-authority pages.

X-Robots-Tag

X-Robots-Tag is the HTTP header equivalent of the robots meta tag, and it's essential for resources that have no <head>.

X-Robots-Tag: noindex, nofollow
X-Robots-Tag: googlebot: noindex
X-Robots-Tag: unavailable_after: 25 Dec 2026 15:00:00 PST

Use it for:

PDFs, DOCX files, and images — the only way to apply noindex to them.
Bulk directory-level control via nginx or Apache config.
Time-bound indexing with unavailable_after for time-sensitive promos or events.

Common X-Robots-Tag mistakes

Setting X-Robots-Tag: noindex globally and then trying to whitelist exceptions — this all-or-nothing thinking usually backfires.
CDNs that strip custom headers. Verify the header actually reaches Googlebot, since some CDNs filter differently based on the bot's user-agent.
Setting the header on a 3xx response — only the final URL's header matters.
Value casing: the header name is case-insensitive, and while NoIndex, Noindex, and NOINDEX all work in practice, the spec value is noindex.

The header your Googlebot user-agent receives should be identical to what a regular browser gets. A difference between the two is a cloaking risk — a topic tied closely to HTTPS & Security Headers.

hreflang

hreflang signals language and regional targeting to search engines. Pick one implementation method and don't mix them:

HTML <link rel="alternate" hreflang="..." href="..."> in the <head>.
HTTP Link: header.
XML sitemap <xhtml:link> annotations.

A few rules are non-negotiable. Links must be bidirectional: if page A references B with hreflang="es", B must reference A with the corresponding hreflang. Missing return links — the most common hreflang error worldwide — cause the whole annotation set to be ignored. Always include hreflang="x-default" as a fallback, and every page must self-reference. Use language-REGION codes built from ISO 639-1 for language and ISO 3166-1 Alpha-2 for region.

<link rel="alternate" hreflang="en" href="https://example.com/" />
<link rel="alternate" hreflang="en-GB" href="https://example.com/uk/" />
<link rel="alternate" hreflang="de" href="https://example.com/de/" />
<link rel="alternate" hreflang="es-419" href="https://example.com/latam/" />
<link rel="alternate" hreflang="x-default" href="https://example.com/" />

Common hreflang mistakes

Invalid codes like en-UK (ISO is GB) or es-LA (use 419 for Latin America or a specific country).
Missing return links.
Pointing to non-canonical URLs — hreflang URLs must equal the canonical URLs.
Pointing to a redirected or noindex'd page, which breaks or conflicts with the annotation and gets it ignored.
Mixing implementation methods across pages, creating partial maps.
Forgetting x-default, leaving unmatched locales with no fallback.
Using relative URLs — hreflang URLs must be absolute, including the protocol.
Applying hreflang to content that isn't genuinely translated or regionalized, creating duplicate content across locales.
Canonical/hreflang conflict: a page that canonicalizes to a different-language version while also listing hreflang alternates. Each language version must self-canonicalize.

Check	What to validate
Bidirectional	Every hreflang edge has a reciprocal return link
ISO codes	Language against 639-1, region against 3166-1 Alpha-2 (plus `419`, `x-default`)
Self-reference	Each cluster member references itself
x-default	Present per cluster
Target status	All hreflang URLs return 200, no 3xx/4xx/5xx
Canonical match	Each hreflang URL equals its target's canonical
One method	No mixed HTML + sitemap + header
Absolute URLs	Reject relative
No noindex	No hreflang target is noindex'd
No duplicates	Two `en-US` entries to different URLs is an error

What to do

Add a single, absolute, self-referencing canonical to the <head> of every indexable page, and confirm each target returns 200, is indexable, and appears in your sitemap.
Stop paginated pages canonicalizing to page 1 — make each ?page=N self-canonicalize.
Replace any noindex, follow equity workaround; use redirects for permanent consolidation instead.
Audit for the killer combo of robots.txt disallow plus noindex, and split those so Google can actually crawl and see the directive.
Use X-Robots-Tag for PDFs, images, and other non-HTML files, and verify the header survives your CDN and reaches Googlebot.
Decide your AI-snippet posture: use nosnippet or max-snippet:0 if you want SERP listing without AI summarization, and remember Google-Extended lives in robots.txt, not a meta tag.
Build a full hreflang graph and fix any missing return links, invalid ISO codes, missing x-default, or targets that redirect, 404, or noindex.
Enforce noindex on staging and preview environments so it never ships to production by accident.

For the broader picture, see Technical SEO and the rest of LLMRanks Learn.

save this card

Download card

1080×1350 · post it anywhere

put it to work

See how ChatGPT, Gemini and Google AI actually talk about your brand.

Check your AI visibility — free