How to Spot and Solve Crawl & Indexing Issues in Google Search Console

There's a frustrating reality many website owners face — you've built a solid site, written great content, and yet Google seems to be ignoring you completely. Pages aren't showing up. Traffic isn't coming. And you have no idea why.

Nine times out of ten, the culprit is a crawl or indexing issue. And the tool that exposes exactly what's going wrong is sitting right there, free, waiting for you to use it — Google Search Console.

This guide walks you through how to spot these issues, understand what they mean, and fix them for good.

What "Crawling" and "Indexing" Actually Mean

Before jumping into fixes, it helps to understand what Google is actually doing when it visits your site.

Crawling is the discovery phase. Google sends a bot — Googlebot — to follow links across the web and visit your pages. Indexing is what comes next — Google reads what it found, decides whether it's worth storing, and adds it to its database of searchable content.

Your page only appears in search results if it has been both crawled and indexed. Miss either step and you're invisible — no matter how good your content is.

Step 1: Open the Pages Report

Log into Google Search Console and navigate to Indexing → Pages in the left sidebar.

This is your diagnostic dashboard. Every URL Google knows about on your site is sorted into one of four categories:

Valid — Indexed and live in search results. These pages are doing their job.
Valid with Warnings — Indexed but flagged for something worth investigating.
Error — Not indexed. Something went wrong. These need your attention first.
Excluded — Skipped by Google, either deliberately or by mistake.

Click into the Error tab immediately. This is where your rankings are quietly leaking away.

Step 2: Understand What Each Error Is Telling You

Google doesn't just say "something's wrong" — it tells you specifically what it found. Here's how to read each error type:

"Submitted URL not found (404)"

A page listed in your sitemap returns a 404 — it doesn't exist. Google showed up and found nothing.

Fix: Set up a 301 redirect from the broken URL to a relevant live page. If the page was deleted by mistake, restore it. Either way, remove the dead URL from your sitemap.

"Server error (5xx)"

Your server failed to respond when Googlebot knocked. This is almost always a hosting problem — downtime, overload, or misconfiguration.

Fix: Check your hosting dashboard and server error logs. Persistent 5xx errors are a sign your current hosting plan isn't keeping up with demand.

"Redirect error"

Googlebot hit a redirect loop — Page A sends to Page B which sends back to Page A — or a redirect chain so long it gives up. Either way, it moves on without indexing anything.

Fix: Audit your redirects and make sure every old URL points directly to its final destination. One redirect, one hop.

"Blocked by robots.txt"

Your robots.txt file is actively telling Googlebot to stay away from pages you probably want indexed. This happens a lot after website migrations or CMS updates.

Fix: Use the robots.txt Tester in GSC under Settings. Identify which rules are blocking important pages and remove them.

"Crawled — currently not indexed"

This one is subtle. Google visited the page, read it, and decided it wasn't worth adding to its index. Usually means the content is thin, duplicated, or not distinctive enough.

Fix: Rewrite the page with more depth, original insight, and stronger internal links pointing to it from other pages on your site.

Step 3: Dig Into the Excluded Pages

Not everything in the Excluded tab is a disaster, but it all deserves a careful look.

Noindex tag detected — Someone (a plugin, a developer, a CMS setting) added a noindex instruction to a page you actually want Google to find. Track down the source and remove it.

Canonical pointing elsewhere — A canonical tag is supposed to tell Google which version of a page is the authoritative one. If it's pointing to the wrong URL, Google will ignore your actual content in favour of wherever that canonical leads.

Discovered — currently not indexed — Google found the page through a link but hasn't crawled it yet. This is a crawl budget issue. The page isn't getting enough internal link authority to be prioritised. Fix this by linking to it more prominently from within your site.

Step 4: Use the URL Inspection Tool

For any individual page you're concerned about, the URL Inspection Tool gives you a full breakdown. Paste a URL into the search bar at the top of GSC and you'll see:

Whether the page is currently indexed
When Google last crawled it
A rendered view — what Googlebot actually sees when it visits

That rendered view is especially important if your site uses JavaScript to load content. What looks complete in your browser may appear as a blank shell to Googlebot. If the rendered HTML is missing chunks of your content, you have a JavaScript rendering problem that needs addressing.

Step 5: Request Indexing After Every Fix

Once you've resolved an issue, don't wait for Google to rediscover the page on its own schedule.

In the URL Inspection Tool, click "Request Indexing" after fixing any page. This puts the page in a priority queue for Googlebot to revisit — usually within a few days rather than weeks.

Make this a habit every time you publish new content, update an existing page, or fix a technical issue. It's a small step that consistently speeds up results. This is particularly important after a site migration or after working with a website design company on a major structural change.

Step 6: Clean Up Your XML Sitemap

Your sitemap is the official list of pages you're asking Google to crawl. A disorganised sitemap wastes crawl budget and sends mixed signals about what actually matters on your site.

Go to Indexing → Sitemaps in GSC, submit your sitemap URL, and check for any errors flagged.

The rule is simple: your sitemap should only include pages you want indexed. No 404s. No redirected URLs. No pages with noindex tags. Strip it back to your best, most important content and let that guide Googlebot's attention.

Step 7: Check Core Web Vitals and Mobile Usability

Page experience signals influence how frequently Google crawls your site and how it prioritises your pages. Inside GSC, check:

Experience → Core Web Vitals — Pages with poor loading speed, layout shifts, or slow interactivity get crawled less often and ranked lower.

Experience → Mobile Usability — Google uses mobile-first indexing, meaning it primarily evaluates the mobile version of your site. Any rendering issues on mobile directly impact your search visibility.

If your site was built without performance in mind, or hasn't been updated in years, this is often where the hidden damage is found. A properly structured site from a professional designing company will handle these fundamentals from the ground up — saving you months of troubleshooting later.

Step 8: Monitor Crawl Stats Regularly

Under Settings → Crawl Stats in GSC, you get a window into Googlebot's ongoing activity on your site:

How many pages it crawls per day
Your server's average response time
Which file types are being visited most

Watch for sudden changes. A drop in crawl frequency often means something changed — a robots.txt update, a drop in site authority, or a server performance issue. A spike in errors alongside falling impressions in the Performance report is almost always a sign something broke recently.

Check this monthly at minimum. Catching a problem early is far easier than recovering from months of lost rankings.

A Maintenance Routine That Keeps Problems Away

Fixing crawl and indexing issues once isn't enough — they come back. Here's a simple rhythm to stay on top of things:

Weekly: Check the Pages report for new errors. Fix anything that appears.

After every publish: Use URL Inspection to request indexing on new pages.

After every site change: Review robots.txt and resubmit your sitemap.

Monthly: Review Core Web Vitals, Crawl Stats, and mobile usability for any regressions.

Final Word

Crawl and indexing issues are invisible to your visitors but devastating to your rankings. They don't announce themselves. They just quietly remove your pages from search results while everything on the surface looks fine.

Google Search Console is the diagnostic tool that makes the invisible visible. The data is free, the fixes are mostly straightforward, and the impact on your organic traffic can be significant.

If you're starting a new site or dealing with persistent technical problems on an existing one, the smartest investment you can make is building on a solid technical foundation. A reliable website design partner who understands both development and SEO will ensure these issues never become a problem in the first place.

Get the structure right. Let Google in. Watch your pages rank.

Have a crawl or indexing error you can't figure out? Share it in the comments — let's work through it together.