How to Fix Pages Stuck in Crawled – Not Indexed

Many site owners have recently reached out to me, frustrated by pages that Google has crawled but refuses to index. When I dig into these cases, a clear pattern emerges: most of these pages suffer from quality issues. They often fall into the category of commodity content , material that simply rehashes what countless others have already published, offering nothing truly new or more helpful to searchers.

This article walks through how I analyze the “crawled – currently not indexed” report in Google Search Console, introduces a tool to pinpoint which pages deserve closer scrutiny, and offers actionable tips for improvement. However, a fair warning: if you have a large volume of pages stuck in this state, recovery is an uphill battle.

Insights from Google’s Search Central Event

At the Google Search Central event in Toronto in April 2026, a Googler shared how Search works. When Google crawls a page, it downloads it. If the system deems it useful, it may be placed into the index. The presenter noted that AI has lowered the barrier for content creation. As a result, Google now prioritizes content that demonstrates two key qualities: personal experience and unique knowledge that no one else possesses.

Why might Google crawl a page and then decide not to index it? Two primary reasons emerged:

1. Technical Issues (Rare but Possible)

I’ve seen this only occasionally. In one recent case, a site underwent a migration, and all its pages ended up in the crawled-not-indexed bucket. Using the Page Inspection tool in GSC, I ran a live test. The result was shocking: the page showed only a heading, a few boilerplate words, and zero meaningful content. The culprit? A `robots.txt` rule: `Disallow: /?`. This was intended to block URLs with parameters like `?replytocom`, but the site’s new theme relied on those exact parameters for CSS and JavaScript files. Google was effectively blocked from seeing most of the content. After removing that rule, pages slowly began reappearing in the index.

If your live test confirms Google can see your content, a technical issue is unlikely. Also, note that some pages , like `/feed/` or paginated ones , are naturally expected to appear in this report.

2. Quality (The More Common Culprit)

The Googler explained that another reason pages aren’t indexed is simply: “we looked at it and found it not to be good.” If thousands of pages already cover the same topic, Google may decide your version offers no added value. Sometimes, Google experiments by temporarily indexing a page to see if users respond positively. The goal is to find which pages produce “happier users.”

Commodity Content: The Likely Root Cause

Commodity content is the most frequent reason pages get stuck. This is content that almost anyone could write , it merely repeats what’s already online. In contrast, non-commodity content brings a unique perspective, first-hand knowledge, or experience that others lack.

Take this very article. Anyone could use AI to define “crawled – currently not indexed.” But I’ve shared my real-world experience as a professional, a specific technical example, and insights from a Google event. That makes it non-commodity.

My Observations on Stuck Pages

These pages aren’t junk. They’re often decent , as good as what’s already ranking. And that’s precisely the problem: they’re not special. Here’s how I analyze them:

1. In GSC, go to Pages > crawled – currently not indexed.

Google’s Podcast on the Indexing Report

In a recent Search Off the Record podcast, John Mueller and Martin Splitt discussed this topic. They emphasized that if Google has strong concerns about a site’s overall quality, it will crawl and index far fewer pages. The key takeaway: when you see a pattern of many pages not being indexed without a technical reason, you must step back and assess overall site quality. This is hard because it’s your “baby.” But try to see it through a stranger’s eyes. If most of your site is AI-generated with no unique value, users will notice. Also, quality isn’t just text , it’s the full user experience, including ads, interstitials, and page load speed.

How to Fix the Issue

This is the tough part. If excessive ads or filler content are the problem, those are fixable. If there’s a technical glitch, resolve it and request reindexing. But if it’s a quality issue, you’ll need to invest significant effort.

Many sites once thrived by covering topics thoroughly. Now, the trend is to anticipate every related query and cover them all. But creating content at scale this way risks a scaled content penalty. I suspect the June 2026 spam update hit many sites producing commodity content at scale , even without a manual action, traffic can drop inexplicably.

AI has made content creation easy, but if your SEO agency uses AI to churn out content on any topic, it’s likely not original or insightful. The exceptions are agencies that use clever AI pipelines to interview a business and extract real experience into original content.

I don’t recommend using AI to write content without human input. But you can brainstorm with AI to improve it. Remember: the word “effort” appears 120 times in Google’s Quality Rater Guidelines. Draw from your own experience to add genuine value.

Try this simple prompt: “Is this content likely to be considered commodity content?” Then ask: “Give me 20 ideas that help me draw from my first-hand experience to make this article even more helpful and substantially better than anything else on this topic.”

Tools to Assess Your Pages

I’ve created two tools at tools.mariehaynes.com using Google’s Antigravity:

1. Filter your crawled-not indexed URLs: Export the list from GSC as a CSV. Upload it to strip out `/feed/` and other non-essential pages, so you can focus on the URLs that matter.Google is becoming increasingly strict about what it indexes. The path forward requires genuine effort and unique value.