A page that Google has never stored cannot rank, and an AI answer cannot cite a URL its crawler was never allowed to fetch. Indexing and crawl issues are the SEO issues that sit underneath every other fix: titles, content, and GEO formatting do not matter until the canonical URL is eligible to be retrieved.
This guide is the follow-up to common SEO and GEO issues that hurt rankings. Use that page for the wider audit. Use duplicate content and keyword cannibalization when the wrong URL is the one Google keeps. Use this one when Search Console says the page was crawled or discovered but is not indexed, the sitemap lists URLs that should not be indexed, or robots.txt is blocking Google or the crawlers behind AI answers.
Not ranking is not the same as not indexed
A ranking problem means the URL is in the index and still loses the query. An indexing problem means there is no URL for that query to rank. Teams waste months rewriting copy while the page is excluded.
Check the exact URL, not the folder and not a redirected variant. In Google Search Console, the URL Inspection report is the source of truth for one address: whether Google last crawled it, which canonical Google chose, and whether a noindex or robots rule is in the way. Coverage reports then show the pattern across the site.
These are the states that explain most “website not indexed” cases:
| State | What it usually means |
|---|---|
| Crawled – currently not indexed | Google fetched the URL and chose not to keep it. Often thin, duplicate, or low-value compared with other pages. |
| Discovered – currently not indexed | Google knows the URL, usually from a sitemap or a link, and has not crawled it yet. Common for orphans and low-priority URLs. |
Excluded by noindex | A meta robots tag or HTTP header is still asking Google to drop the page. |
| Blocked by robots.txt | Googlebot is not allowed to fetch it. The URL can still be listed if something links to it, but Google cannot read the content. |
| Soft 404 | The server returns 200 for an empty, error, or “no results” page. Google treats it as not a real page. |
| Page with redirect | The sitemap or internal links point at an old URL. Only the final canonical can rank. |
| Alternate page with proper canonical | A duplicate correctly points elsewhere. Expected for parameters and language alternates — a problem only if the preferred URL is the one being dropped. |
“Crawled – currently not indexed” is a quality decision, not a queue you can flush with another sitemap submission. “Discovered – currently not indexed” is a discovery and priority problem. Fixing the wrong one wastes the sprint.
Sitemap errors that send the wrong URLs
An XML sitemap is a hint, not a command. Sitemap errors still matter because they spend crawl attention on URLs you do not want, and they leave out the ones you do.
Remove any URL that:
- Returns a redirect, 404, or 410
- Carries
noindex - Is a parameter, staging, or print variant
- Is not the canonical (wrong host, wrong protocol, or the non-preferred slash)
- Is a thin tag, author, or attachment page you do not intend to rank
List the English and Traditional Chinese canonicals you actually want indexed. A bilingual Hong Kong site should not put both languages only in one language’s sitemap and hope hreflang does the rest. Each indexable URL belongs in the sitemap once, with the same host the canonical tag uses.
Then confirm the sitemap file itself returns 200, is not blocked by robots.txt, and is the file submitted in Search Console. A sitemap that only exists on a staging host, or that references last year’s domain, is a technical SEO issue even when every URL inside it looks fine.
robots.txt blocks, including AI crawlers
Disallow: / left on after launch is still the fastest way to take a site out of Google. More often the rule is narrower and just as damaging: a production robots.txt copied from staging that disallows /insights/, /zh/, or every URL with a query string that your canonicals happen to use.
robots.txt stops a crawler from fetching. It does not reliably keep a URL out of the index if other sites link to it. For pages you want removed, use noindex on a URL that Googlebot is allowed to crawl, or return 404/410. Using robots.txt to “noindex” a page is a long-standing technical SEO mistake: Google cannot see the noindex tag if it cannot fetch the page.
AI access is a separate decision, and it is easy to get wrong:
- Googlebot crawls for Google Search. AI Overviews draw on indexed Search results, so a block on Googlebot is both an SEO issue and a GEO issue.
- Google-Extended is not Googlebot. Disallowing it opts a site out of some Gemini training and grounding uses. It does not, by itself, remove pages from Google Search or AI Overviews.
- OAI-SearchBot, PerplexityBot, and Bingbot are examples of crawlers that answer engines use when they retrieve live pages. Blocking them on purpose is a policy choice. Blocking them by accident — a wildcard disallow, or a rule copied for every bot — is a GEO issue if the goal is to be cited in ChatGPT, Perplexity, or Copilot.
- GPTBot is OpenAI’s training crawler. Blocking it is not the same as blocking ChatGPT search.
Read the live robots.txt on the production host. Then test the exact article URL against Googlebot and against any AI crawler you intend to allow. A rule that looks harmless on the homepage can still match every guide under a folder.
Orphan pages and crawl paths
“Discovered – currently not indexed” often means the only discovery path was a sitemap. Google can crawl that URL and still treat it as unimportant because nothing on an indexed page links to it.
Orphan pages are an SEO issue and a GEO issue for the same reason. Search crawlers and many AI retrievers follow links from hubs. A guide that is only in the sitemap, with no link from the articles index, a related guide, or the relevant service section, is easy to skip.
On a bilingual site, check both languages. An English article that links only to other English URLs, while the Chinese version has no in-content links at all, leaves the Chinese URL as an orphan even when hreflang is correct. Give every indexable page at least one normal link from an indexed page in the same language, using an anchor that states the topic.
Crawl depth matters less on a small site than internal links do. A five-click path from the homepage is a warning; a page with zero internal links is the defect. Broken hrefs, redirect chains, and anchors that never name the topic are the rest of that link graph. The internal links and redirects guide covers them.
When the HTML is empty until JavaScript runs
If the answer, the headings, and the links appear only after client-side rendering, a crawler that does not execute JavaScript — or that times out — stores a shell. Googlebot renders many pages, but it does not guarantee a full render on every crawl, and some AI crawlers fetch the raw HTML only.
Keep the primary explanation in the initial HTML: the title, the H1, the opening answer, and the links to related guides. Use JavaScript for interaction, not for the only copy of the text you want cited. If URL Inspection’s rendered HTML is missing the paragraph you expect to rank, the indexing issue is the template, not the keyword.
How crawl issues become GEO issues
Generative engine optimization assumes a retrievable page. How to rank in AI search describes citation-shaped content. That work cannot succeed when:
- The URL is excluded, so there is no passage to quote
- robots.txt blocks the crawler that product uses
- The sitemap advertises a redirected or parameter URL, and the model cites that address
- The fetched HTML has no answer, because the text was injected by JavaScript
- English and Chinese canonicals disagree, so the model quotes a thin translation or an outdated host
A brand can look “absent from AI” when the real defect is that no crawler can read the page. Confirm fetch and index status before you rewrite for citations.
What to fix first
- Inspect the exact canonical URL. Note the coverage state, the declared canonical, and the canonical Google selected.
- If robots.txt blocks Googlebot, remove that rule and request indexing only after the URL returns 200 and is indexable.
- If the page is
noindexby mistake, remove the tag, confirm the HTTP header is clean, and check the template so the next deploy does not put it back. - Strip the sitemap down to indexable canonicals, including both languages, and resubmit the file Search Console already knows.
- Add an internal link from an indexed hub in the same language. Do not rely on the sitemap alone.
- Compare raw HTML with the rendered page. Put the answer in the HTML if it is missing.
- If the state is “crawled – currently not indexed” and the URL is genuinely thin or duplicative, improve that page or consolidate it. Another ping will not change the decision. The duplicate-content guide covers consolidation.
- Decide, explicitly, which AI crawlers may fetch the site. Allow the ones that match the visibility you want. Do not treat a training-crawler opt-out as a Search problem, and do not block Googlebot in the process.
After Google recrawls, check the same URL again before you judge rankings or AI citations. A page that is still “discovered” has not failed the content test yet. A page that is indexed and still absent from answers is a GEO content problem, not an indexing one.
Questions teams ask
Why is my page crawled but not indexed?
Google fetched it and declined to store it. The usual reasons are thin or duplicate content, a soft 404, a page that adds nothing beyond a stronger URL on the same site, or a canonical that points elsewhere. Start with URL Inspection, then compare the page with the URL you actually want to rank.
Do sitemap errors cause SEO ranking problems?
They cause indexing problems, which then look like ranking problems. A sitemap full of redirects, noindex URLs, and non-canonical hosts tells Google to spend time on addresses that should not rank. Clean the file so it lists only the canonicals you maintain.
If I block AI crawlers, do I hurt Google rankings?
Blocking Googlebot does. Blocking Google-Extended, GPTBot, or other non-Googlebot tokens does not, on its own, remove a page from Google Search. It can still be a GEO issue: some AI products will stop retrieving you. Read the token you are disallowing before you treat it as an SEO fix or as a privacy opt-out.
What should a Hong Kong site check that a single-language site does not?
Both language URLs. A Chinese page blocked by a /zh/ robots rule, missing from the sitemap, or never linked from a Chinese hub will not be indexed even when the English version ranks. Hreflang does not override noindex, robots.txt, or a sitemap that omits the URL. Local listing issues are separate; those sit in the Hong Kong SEO and GEO guide.
What to review this month
Export the coverage rows for “crawled – currently not indexed”, “discovered – currently not indexed”, “blocked by robots.txt”, and “excluded by noindex”. For each URL that should rank, record one cause and one change. Then test the production robots.txt against Googlebot and the AI crawlers you want to allow.
Search visibility starts with a URL that can be fetched and stored. The SEO issues above are why that step fails, and why AI rankings never get a page to cite.