How Google finds your pages: crawling and indexing, explained
Your site can be brilliant and still invisible. Here's how Google actually discovers, reads and stores pages — and where things go wrong.
Before your site can rank anywhere, Google has to do three things: find it, read it, and store it in its index. Plenty of sites fail at step one or two without anyone noticing — the pages exist, the design is lovely, and Google simply never gets a usable copy. Understanding the pipeline makes almost every SEO problem easier to diagnose.
Step 1: Discovery — how Google learns your pages exist
Google finds URLs in two main ways: by following links from pages it already knows, and by reading sitemaps submitted through Search Console. Internal links matter more than people expect — if a page isn't linked from anywhere on your own site, Google has to stumble across it from outside.
What breaks discovery
- Orphan pages — live, but no internal link points at them.
- No sitemap, or a stale one that lists deleted URLs.
- Broken navigation — menus that only work with JavaScript Google struggles to render.
- New sites with zero links from anywhere else. Discovery takes weeks instead of days.
Step 2: Crawling — Google reads the page
Discovery gets a URL into Google's queue; crawling is when Googlebot actually fetches it. Google doesn't crawl every URL every day — it rations attention based on how healthy and how important your site appears. A fast, reliable server earns more frequent visits; timeouts and 5xx errors earn fewer.
The things that quietly block crawling
- robots.txt mistakes. One
Disallow: /during a staging migration that never got reverted will hide an entire site, and Google will happily keep it hidden for months. - noindex tags. A
noindexmeta tag left on real pages is the classic "my page vanished" cause. It's a directive, not a suggestion. - Canonical confusion. If every page canonically points at the homepage, Google concludes you have one page.
- Slow servers. Google explicitly throttles crawling of sites that respond slowly or error out.
Step 3: Indexing — the page enters the library
A crawled page isn't automatically indexed. Google renders it, evaluates quality and duplication, and decides whether storing it is worthwhile. Thin, duplicated or near-empty pages often get crawled forever and indexed never. This is why "the page is live, I can open it in my browser" has never been the test that matters — you check with site: searches or the URL Inspection tool in Search Console.
The five-minute health check
For any page you care about, check four things:
- Is it indexed? Search
site:codexio.co.uk/your-page/. Missing means a problem upstream. - Is anything blocking it? View source: look for
noindexand check the canonical URL is the page itself. - Does robots.txt allow it? It's at
yourdomain.co.uk/robots.txt. - Is the sitemap current? Every URL in it should be one you want indexed.
Our free SEO audit tool runs the page-level checks — indexability directives, canonical, meta data, link counts — in one pass, so you can spot the blockers in seconds.
Why this is step zero of SEO
Keyword research, content and links all assume Google can see your pages. Fixing a stray noindex or a stale sitemap routinely does more for a small site's traffic than months of content work, because it releases work that was already done. It's the first thing we check in any technical SEO engagement — and the cheapest win on the board.
Check your own site in seconds with our free SEO audit tool — and if you’d rather hand the whole job over, talk to us.
Found this useful? Share on LinkedIn · Email it to someone