Selective Indexing: SEO in the Age of AI Search

Search engines and AI platforms can crawl more content than ever, but that doesn’t mean they want to keep, refresh, or cite every page they find.

In this episode, Bill Hunt and Bill Scully discuss the three-gate model of selective indexing: crawl capacity, crawl demand, and information gain. They cover why technical SEO alone no longer guarantees inclusion, how cheap AI-generated content is reshaping crawl economics, and why some problems live at the site level while others live at the page level.

The episode closes with a practical framework for auditing crawl waste, strengthening information gain, and deciding whether your next page actually deserves to be published.

If a page disappeared tomorrow, would anyone, human or machine, lose anything important? That’s the question every content team needs to start asking.

---

---

• Old question vs. new question: It’s no longer can this be crawled? — it’s why should a search engine or AI system keep spending resources on this page?
• The three-gate model: Crawl capacity (can the site be served efficiently?), crawl demand (does anyone want this content?), and information gain (does the page add something new?)
• Evidence behind the shift: Google’s updated crawl documentation, a large-scale index-removal event from May 2025, and cross-industry moves toward don’t make the crawler guess (IndexNow, Cloudflare Crawler Hints)
• Nine crawl-worthiness factors: grouped into Value, Trust, Efficiency, and Permission
• Site-level vs. page-level problems: why fixing infrastructure and inventory issues has to come before page-level optimization can work
• International SEO implications: why translation alone isn’t information gain, and how localization actually earns it
• A six-step action plan: remove crawl waste, make change explicit, build around demonstrable demand, engineer information gain, improve machine efficiency, clarify licensing
• Five myths debunked, including more content = more visibility and AI-generated content is automatically bad
• A hands-on exercise: score a 25–100 URL content cluster against nine factors and sort problems into the three gates

---

Youtube: https://youtu.be/WSFrJA6Yfd4

Spotify: https://open.spotify.com/episode/33Xq2cQyJaSW2pMs14TubQ?si=euDeXkakTWuluq0pokgGWQ

Apple Podcast: https://podcasts.apple.com/us/podcast/selective-indexing-seo-in-the-age-of-ai-search/id1755789311?i=1000781973824

---

The New Economics of Web Crawling: A Framework for Selective Content Ingestion — One Year Later

Cloudflare — Sample terms

---

Bill Scully (00:00.000)
Welcome to episode 44 of the International Web Effectiveness Podcast. I’m Bill Scully, along with Bill Hunt. Today we’re going to be talking about selective indexing — SEO in the age of AI search. We’re going to explore the fundamental change in how we should think about crawling and indexing, including why being crawled does not mean being indexed.

We’ll cover the three gates that Bill has put together as a framework, and why every page must pass them. And we’ll end with what international and enterprise websites should be doing now. So Bill, you wrote a really good article that got me thinking — I actually wrote a LinkedIn post based on it. Explain to me why you wrote the article “The New Economics of Web Crawling: A Framework for Selective Content Ingestion” one year later.

Bill Hunt (01:11.776)
It’s a mouthful, huh? Well, I’ve really been trying to put on my economist hat lately, to look at things through a different lens — from both sides of the equation, the crawler/ingester side and the business side. This was actually a follow-up to an article I wrote around the same time last year, which itself followed one from the year before that. That earlier one was part of a keynote where I was basically ranting that people don’t care about being crawled or cleaning up their house, and that at some point it’s going to start costing them.

So that’s why I called it “the economics have changed” — that’s literally the trigger. Previously I didn’t have a lot of ammunition from an economics perspective. That’s what’s changed. I tried to cite different things that have happened, but the big change is all of this AI stuff.

Back in the day — and we’re talking like last year — crawling and ingesting stuff into Google was fairly straightforward. Common Crawl crawling it so it can be used by all the AI platforms was fairly straightforward too. Where economics comes in now is the processing cost, the token cost. There’s obviously a storage issue, but it’s really the processing cost to sift through all that stuff and figure out what’s worthy of being an answer when somebody asks AI Overviews, ChatGPT, Claude, or wherever.

That’s really what prompted it, because I was listening to Scott — God, I can never remember his name — Scott Galloway.

Bill Hunt (03:24.127)
On his podcast about growth, they were talking about all these companies, and I saw it in my own portfolio too — somebody actually said the quiet part out loud: AI companies may not be making money. They’re selling excess capacity that people aren’t using. And companies were saying, “We’ve exhausted our token budget for the entire year — it’s costing us too much to process information.”

Simultaneously, Google announced for the first time ever that they had negative free cash flow, losing money because of the costs of processing things for AI. And everybody’s saying the cost to process this stuff with ChatGPT is too much. I’ve had five of my advisory clients say they’ve had to cut back on tools because the cost to process this stuff is getting too expensive.

So if everybody’s saying it’s too expensive to process, they’ll probably process less. That was the catalyst on the economic side. The other piece was the signals we were getting. We had Adam Ghent on earlier, talking about his crawling system, and he’d noticed in a report that almost everybody reporting data on index rates was seeing them drop — and the “crawled but not indexed” signal had gone through the roof.

We intuitively knew this was non-beneficial content, light content, duplicative content, but we didn’t really have anything concrete to apply to it. Then a few people, like Marie Haynes and others, started digging in and noticed a correlation, timed in many cases with Google events — especially in Toronto, where a good third of the presentation was about

Bill Hunt (05:31.611)
additive content, or information gain. Of course the search community poo-pooed this — “information gain, whatever, just take my stuff, it’s good stuff, I wrote it with AI and puked it out.” And look, I’m not demeaning anybody’s hand-written, purposeful content. I put blood, sweat, and tears into mine too. But as someone who travels a lot, I’ve completely stopped looking at travel blogs, because all people did was switch out an image.

Bill Hunt (06:01.413)
It’s almost the same stuff every time. There’s nothing about their actual experience — not all of them, but many are just repurposed or AI-written trip reports, even if they did take the trip. That’s what I found when I was asked to evaluate a website for a client who was complaining it wasn’t doing well.

They said, “Help us, we can’t figure out the problem.” When I looked at the GSC Index Status report — which we’ve talked about many times, it’s an invaluable report — I looked at the pages that had been crawled but not indexed, meaning Google had seen them.

I noticed a pattern: they all belonged to one unique folder. I took all of them and ran them through a Claude skill I’d written to evaluate uniqueness and information gain — what I call “additive content” — compared to the rest of the website. It came back and said these were 100% duplicates of the content on the main page for that location.

So there was nothing additive. A hundred pages, all the same type of information, across a hundred locations — services or amenities that were already covered on a different page. Google doesn’t need this.

Bill Scully (07:40.003)
Yeah, and this isn’t anything new — Google’s been telling us forever to write unique content, write from your own experience. That’s what stops people from just rewriting the first ten pages into one longer article. Doing that might create completeness, but it has no information gain.

Bill Hunt (08:05.970)
Right. I know Corey’s been talking about this with his crawling work — the actual term for it is “cost of retrieval.”

Bill Scully (08:14.582)
Yeah, cost of retrieval. Think about it in terms of Google being an engineering business — they need to make their processes efficient, and they’re going to do what they can to reduce the cost of retrieval. If something’s already out there, there’s really no reason for them to even look at it.

Bill Hunt (08:47.154)
Exactly. So why now? That’s the whole point. We’ve been talking about this for twenty-plus years — Corey’s been ranting about it, plenty of people have, and Google’s talked about it too. So what’s changed, and why are they cracking down now? It comes back to token cost.

Bill Hunt (09:12.711)
If 86% — I think that’s the latest number — of queries on Google return an AI Overview, that has to be generated or synthesized, which means resources are going toward it. It’s the old Tootsie Pop question: how many licks to get to the center? Is it five pages, ten pages, twenty pages, a thousand pages before there’s enough consensus for an answer? Whatever that threshold is, that’s when they say, “I have enough.”

So if you’re source number 11 — in my article I wrote “11 to N” — and Google already has 10 good, vetted sources, are you even going to be considered? What’s the rubric for adding something new — your experience, your personal view, going deeper? That’s the dilemma, and it’s what led me to build the three-gate model.

If you go to Google’s crawl budget page, it looks deceptively simple. I just tried to turn it into an easier-to-read model — one that doesn’t just say “if you have fewer than a million links, this doesn’t apply to you.”

Bill Scully (10:58.947)
Right, but you have to think like a lawyer — being crawled is not the same as ranking.

Bill Hunt (11:04.579)
Exactly. Google’s told us this.

Bill Scully (11:05.656)
So yeah — “I’ll crawl everything, and you know what? I won’t index it.”

Bill Hunt (11:11.091)
That’s the point. They used to crawl everything because they didn’t know any better. There’s a model I’ve been nervous to put out there — I’ve been working on it for two years — where I tried to create a grid, again thinking like an engineer.

A URL comes into view. How do I decide what to do with it? We get URLs three ways: during a crawl, from an XML sitemap, or via direct submission. So if I’m limiting resources, which of these matters most? This is actually the training course we’ve been trying to build, to teach developers to think like Google and translate that into how they build their own websites. If you’re natively limiting stuff, how do you do it?

So imagine you have a huge list of URLs.

Bill Scully (12:28.590)
Say that again — if we’re natively limiting stuff?

Bill Hunt (12:33.251)
If you’re natively limiting stuff, how do you, as a website owner, make sure you’re not tripping a switch that gets you excluded?

Bill Scully (12:42.039)
Right.

Bill Hunt (12:44.995)
So that’s the logic I was trying to build. If a URL passes those tests — all the criteria — I put it in the next box, which is where Google decides whether to go fetch it. Remember when Danny Sullivan, at a conference in New York, mentioned Lily Ray’s interview question about the steps Google takes for indexing, retrieving, and rendering?

Everybody assumed it was one simultaneous process, and the community exploded when they learned that wasn’t true.

Google doesn’t find a URL and magically crawl it, magically render it, and magically dump it into the index — that’s not how it happens. It’s the same thing here. Crawl capacity is too big a word for it, really — it’s more like: do you make it easy for me to get it? I think in your notes you put it more simply as, “Can I get it easily?”

I tried to frame it like a supply chain: if I’m a company selling products and I have five vendors I can buy from, and one can deliver today, that’s why Amazon is winning. I can get something from an Amazon warehouse ten miles from my house today — something that’s probably not in stock at Target or Walmart. So where am I going to order from?

Bill Scully (14:24.568)
Yeah, yeah, yeah.

Bill Hunt (14:28.485)
Amazon — despite wanting to shop local or in a physical store, I know they probably won’t have it, but I can get it today, or by 4 a.m. tomorrow, from Amazon.

Bill Scully (14:41.132)
Yeah, they’ve reduced the friction.

Bill Hunt (14:43.908)
They’ve reduced the friction. They make it easy to crawl.

Bill Scully (14:47.660)
For people who haven’t read the article, I think it’d help to walk through your framework. Tell me if I’m wrong, but it’s a three-gate model: first is crawl capacity — can the website serve crawlers effectively? Then crawl demand — does the search system have a reason to spend resources on this site or URL, which ties into the feedback loop from information gain. And then information gain — does the individual page contribute something not already available in the corpus?

So if you don’t have anything of value, you don’t pass the crawl-demand gate. If your site is slow or difficult to navigate, you don’t pass the first gate. And it works forwards and backwards with crawl capacity.

Bill Hunt (15:40.111)
There are two parts to crawl demand.

Bill Scully (15:58.359)
Crawl demand and information gain.

Bill Hunt (16:00.880)
Right, and there’s a nuance to crawl demand: is anybody actually asking for it? We all use keywords or prompts — well, I think we’re banned from saying “keywords” nowadays, but we use query data to decide what content to write. So imagine a search engine asking the same question: where would you spend your precious resources?

Bill Scully (16:08.492)
Yeah, search demand — that’s another one.

Bill Hunt (16:37.923)
It’s like that old argument on Spotify — “The Star-Spangled Banner on a banjo” only gets retrieved once a year. So are we going to go get more banjo music, or get something people actually want? That’s literally it. Does the data tell us anybody wants what you have?

If you’re putting out low-demand content, they may not need it. And then the third gate is: can we actually get into your store to buy it? Do you have what we want? Is there anything different from what’s available elsewhere? Do I drive across town to a specialty store, or go to Walmart and get something with more limited options? Those are the three gates.

If you fail any of them — you’re hard to crawl and slow, so I’m probably not going to fetch content from you; your content isn’t in high demand or doesn’t change frequently, so I’m probably not going to get it from you; or you’re not adding anything to what I already have — in any of those cases, I probably don’t need it from you.

Bill Hunt (18:24.985)
That’s as simply as I can put it. So stepping back to the first one — one of my biggest rants is whenever I open Search Console with a client—

Bill Hunt (18:46.796)
[pause] — sorry, I paused while you were away. Feel free to cut that part.

Bill Scully (18:52.738)
Yeah, you can cut that blank part out. Sorry, my dog’s been going crazy — I don’t know if you heard it.

Bill Hunt (19:04.207)
I could hear it, but I think it’s fine — we’re a friends-and-family podcast. Anyway, the rant is this: if it’s hard for me to get your content — if you open Search Console and see that for every good URL, I’ve had to wade through ten bad ones — that’s not an efficient use of my precious server resources.

Bill Scully (19:08.361)
That’s right. Okay.

Bill Hunt (19:32.939)
If your content isn’t changing, isn’t popular, and isn’t additive, then we don’t need it. Full stop. This is more important now because processing costs have gotten so prohibitive that crawlers across the board are going to get more selective.

Bill Scully (20:02.901)
And I think this affects hreflang too — do sites really need every regional page if they’re all exactly the same?

Bill Hunt (20:12.749)
Exactly, and there’s a whole other layer to this from an hreflang perspective. First off, no AI system uses hreflang — none of them. It’s a Google-serving function only. Second is a concept I call “canonical tokenization.” Let me use my favorite punching bag, Nike.

Nike has 100-plus websites, 37 of them in English, and every one of those English pages is identical. 22 of them show a dollar sign for currency, and the only indication that a given page is meant for a particular market is a country-locator link at the bottom. That’s the only difference.

So if I’m an AI system and I come across a page that’s 99.999% identical to what I already have, I’m going to do canonical tokenization — collapse that information, say the UK or Liechtenstein version of a shoe page, into the existing dataset. No new fields get added, because there’s nothing materially different on the page.

You could argue the price next to the dollar sign is different, so maybe that gets pulled in as a price variation — that’d be the only thing. So when it comes to serving a result, the most dominant version of that page in the world is the one that’s going to show up. We actually saw an example of this—

Bill Hunt (21:56.031)
—that we haven’t talked about since my Search Summit presentation. Companies were quicker to update product information in, say, Norway and the UAE for a particular product, and Google ingested that into its AI system. So when people asked about the product,

it was answered in English, with two citations to the right. Clicking the first one took you to a localized Norwegian page; clicking the second took you to a UAE page in Arabic. They were shown because they had newer information that matched the person’s prompt — the US page still had the

Bill Scully (22:36.197)
Wow.

Bill Hunt (22:44.805)
—emotional and marketing content, whereas the other pages had the exact technical specification. Google presented it in English, my preferred language, with two localized citations, but clicking into them took you to the local market.

Bill Scully (23:02.520)
And this was in the AI Overview? The main body had the newer info, but you didn’t know where it was coming from — you just saw the two references. Wow.

Bill Hunt (23:04.769)
Yes, an AI Overview.

Bill Hunt (23:11.346)
There was no link to it — it just described the answer, but there were two citations with images to the right, one to Norway in Norwegian, one to the UAE in Arabic. When I compared the last-updated dates on all three pages to unravel how this happened,

Bill Scully (23:19.351)
So—

Bill Hunt (23:34.908)
—it didn’t even have the same information. It wasn’t old information; it was actually new information.

The newer information was in those other markets, so because my query was a bit more current than what the US page reflected, it pulled from those markets instead. But it consolidated all of that into one answer and cited two pages from other markets — that’s the canonical tokenization collapsing information even across languages,

into a single corpus.

Bill Scully (24:08.680)
Yeah. And it’s not just AI Overviews — search results are becoming agentic as well.

Bill Hunt (24:19.145)
Right, and that’s the problem — that’s the reason for the change, and why it’s more important now than ever: the economics of processing this data mean they’re going to be exponentially more selective about what they’re taking in. That’s why we’re seeing “crawled, not indexed” rates go through the roof.

That’s also why we’re seeing “aware, not crawled” — another status Google’s been talking about recently. People are saying, “I see this a lot — you’re aware of the URL, but you haven’t crawled it yet.” According to their latest crawl budget update, new content now starts with a very conservative view: are there other signals across the web or ecosystem that tell us whether this is worth our time?

Like we’ve seen when a site relaunches — it might get 10 or 20 URLs crawled, then it slows down while Google runs it through the rubric to decide whether to fetch more.

Bill Scully (25:10.270)
Yeah.

Bill Scully (25:20.499)
Let’s talk about some of the evidence behind this. Adam Ghent — our friend who was previously on the podcast, from Indexing Insight — reported that more than 25% of roughly two million monitored URLs were removed from Google’s index, not related to AI at all, during a period in late 2025.

When they dug into it, the removed pages had little or no recorded search performance. That’s an indication of why keep something if it has no value? If removing a substantial number of indexed pages produces no measurable traffic loss, what did indexing those pages ever accomplish — for the website or for the search engine?

Bill Hunt (26:07.555)
Exactly — what did it get for either side? We SEOs are hoarders. We have this argument two or three times a week: keep this old page because someone might reference something from a press release twenty-seven years ago. There’s an argument for keeping it.

And you can keep it — just don’t expect it to be indexed. You can have fifty gazillion pages with forty-two parameter combinations; just don’t expect every iteration to be indexed. They don’t need it. That data point was almost a canary in the coal mine — it’s why we had Adam on to talk about it.

Bill Scully (26:45.780)
Yep.

Bill Hunt (26:55.826)
The second signal was Google updating its crawl documentation, which they seem to be editing almost weekly now.

Bill Scully (26:55.893)
Right, right.

Bill Hunt (27:04.599)
They tell you explicitly: not every crawled page will be indexed. If it doesn’t meet some threshold of value, they’re not going to do it. They also tell us crawling is now more conservative — before, they’d come in and take everything at the buffet back to the table. They’re not doing that anymore.

Bill Scully (27:05.093)
Yeah, absolutely.

Bill Scully (27:24.079)
Yeah, news sites used to get gobbled up immediately.

Bill Hunt (27:27.605)
Not anymore. If you’re a hundred-page website, this probably doesn’t apply to you as much, but everyone else, it will. It comes down to questions like: how unique is this? Do we need the 47th version of it? I think this is also related to why Google’s been vocal at their search summits about the operating model of nearly every GEO tool.

What do these tools do? You put in a set of prompts, run them, scrape the answer and citations, see who’s cited, then go fetch those pages. People used to scrape the top 10 results and build a page from that. Now we’re doing the same thing at scale with AI — not just the top 10, but everyone who’s cited and referenced.

We pull all of that in, shake it up like a cocktail, and pour out a “me too” page that matches everybody else in the citation block.

Did we add anything? We’re just at parity with what’s already cited. Is being at parity — maybe with a little bonus material scraped or AI-generated — enough to change Google’s mind about us?

Bill Hunt (29:00.684)
Going back to the eligibility gates: did you meet the criteria to even be considered? If you’ve only produced “me too” parity content — no Reddit mentions, no research citations, nothing authoritative pointing to you, no breakdown of features or supplementary signals that show you deserve to be the answer — especially for conditional prompts like “best,” “fastest,” “most efficient” — failing any part of that scoring rubric means parity content won’t help you.

I think this is Google trying to preempt everyone racing to reach parity with whoever’s already cited.

Bill Scully (29:55.200)
Right. And something you mentioned earlier that I’d missed completely: Google updated its crawl documentation to suggest adding a 304 Not Modified response. That’s practically unheard of — nobody runs Screaming Frog to check for 304s.

Bill Hunt (30:12.450)
So now they’re telling us to.

Bill Hunt (30:18.572)
Because they’re not really maintaining that state — you could serve a 304 to Screaming Frog, but that would imply you’d stored the prior state. Here’s the interesting part: this is already happening at the CDN level.

If you’re on a CDN, they’re doing it for you. Last year I wrote about Cloudflare introducing crawler hints —

essentially serving a 304 to tell Google, when a repeat request comes in, that nothing’s changed. At the time it was 52%; now they’re saying it’s 55% of every request coming into Cloudflare for Google — meaning the page

hasn’t changed since the last time Google requested it. So if Google got it Monday and you haven’t changed it — and this has to be a material change — that matters. For anyone thinking, “I’ll just change the date, swap a paragraph, use that plugin that reshuffles paragraphs every third day of the month” — none of that works, because they’re looking for material change. Changing a price every day isn’t material.

I don’t have a precise, quantifiable definition of “material change,” but Cloudflare has documented what counts — meaningful changes to significant portions of the page — and that’s essentially what determines whether they cache it or not. And if they’ve got twenty-five, twenty percent of the web…

Bill Hunt (31:54.461)
Other CDNs are doing something similar, and Bing’s doing it too with IndexNow. Everybody’s already putting native throttling in place.

Google’s essentially saying, “Help us out — we’d rather fetch what’s changed than what hasn’t.”

Bill Scully (32:07.661)
Yeah.

Bill Scully (32:16.284)
Right.

Bill Hunt (32:17.715)
That’s the benefit — and honestly, to me that’s probably the biggest signal. I have a whole article ready on this: how SEOs are sort of Google’s PR and unpaid workforce, because whenever Google wants material change, they throw a carrot to the SEO community and we run with it. Like mobile — Google believed more people were searching on mobile, and who made things mobile-friendly? The SEO community. Who drove AMP adoption? The SEO community. Who drove Core Web Vitals? The SEO community — because developers still don’t care about Core Web Vitals, since most people in the US are on gigabit connections and nobody worries about bandwidth.

But all of these things are driven by the SEO community, which is why it makes sense Google keeps pushing this out. There’s no rubric or scorecard yet, which is why it’s not getting adopted widely — but to me, it’s still a massive signal that they don’t want to ingest unchanged content. And the fact that CDNs are doing it too is another signal worth watching.

Bill Hunt (33:28.375)
Another one I’ve written about is the Munich court ruling in Germany that’s making Google potentially liable for synthetic answers. With the ten blue links, they were just surfacing what people had on their webpages. When they synthesize an answer instead, they’re being held more directly liable for it — and that’s not just a Google issue, it applies to any AI platform.

So if I were Google, I’d become exponentially more critical about what I use and how much of it I use to synthesize an answer. That means limiting what gets ingested — another form of native governance that pushes toward only using more reputable sources.

Bill Scully (34:02.261)
Yeah, sure.

Bill Scully (34:15.198)
Right — somehow they’re determining authority and trustworthiness. If an AI platform can be held responsible for a generated claim, then—

Bill Hunt (34:20.565)
—and for accuracy and relevance—

—they’re going to reduce what they bring in and use. So if you’re a “me too” page, not adding anything, and not validating your content with schema, structured data, or other cross-validation, I think what happens is they’ll go to the primary source rather than a review site or affiliate. But then why don’t platforms always go straight to the source for information?

Because the source often buries or downplays that information. Like I wrote in a Search Engine Journal article recently — I wanted the miles-per-gallon figure for the Ford F-150. Ford wasn’t in the AI Overview, nor in any answer, because they bury that number. It was Edmunds and other car review sites that surfaced it, because they’re not afraid to show it.

So what shows up for an answer like that in the future? Maybe nothing — maybe it falls back to the ten blue links. And the loser in that scenario is the brand that doesn’t show up. From there I went into what criteria they might use — I came up with about nine, if they’re going to be more selective about what they crawl and ingest.

Bill Scully (35:48.427)
Those were information gain and use demand, which you covered already. I think part of it is internal and external links to a page, which demonstrate interest, and frequency of change — I’ve seen case studies where updating a page moves it from position six or seven up to four. There’s also the token-efficiency angle we didn’t cover yet: structured signals — headings, canonicals, schema, content organization — reducing interpretation costs. As a search engine, and there are patents on this, they’ll ingest the headings and first line of a paragraph to determine almost immediately whether it supports the heading and is worth continued ingestion. It’s getting really complicated.

Bill Hunt (36:53.403)
We’re also seeing

research into what’s actually extracted from the DOM. If you look at Google’s new agentic scorecard — which is causing chaos around the llms.txt file — they talk about how parsable the DOM is.

Bill Scully (36:58.862)
Yeah.

Bill Hunt (37:12.804)
I was looking at a page recently with a bunch of if-then fail rules built into it. One example: a real estate site had seven properties listed for sale, but above that, in the DOM, was an error message saying “no properties available.” As Google extracted the page,

it hit the “no properties available” text and ignored everything after it — including the seven actual listings — then flagged it with a soft 404. They were confused about why it was a soft 404, and when we dug in with a site: search, we found Google had surfaced the “no properties available” message.

That message was supposed to be a technically hidden warning, but a lazy implementation had left it visible in the DOM, ahead of the actual listing. This gets back to crawl depth — what’s actually being fetched, and where content sits in relation to the overall page structure. We’re also seeing issues with structured signals —

on that same page, five H tags all referenced external tools, like mortgage calculators or distance tools — meaning the very headings Google might use to understand what follows were essentially red herrings, adding no value. The team argued those tools needed to be there because they’re useful,

Bill Scully (38:58.707)
Right.

Bill Hunt (38:58.820)
—and sure, keep them, just don’t wrap them in an H2 if there’s nothing substantive underneath. Style them with CSS instead — same font, same size — just don’t use a heading tag. Once we explained that, they agreed. This is the kind of thing we need to think about creatively, because it’s exactly what’s being extracted.

And then there’s commercial impact — a two-way street. Is there commercial benefit to the ingester? For Google: can they put ads against it, will it drive traffic? For an AI platform: is it something they could sell or package, or does it increase user loyalty because of answer quality? The last criterion I included was

legal and licensing clarity — can we extract this without getting sued? Look at what Cloudflare is doing with robots.txt: they’re essentially trying to make it a pseudo-contract, implying that visiting the file means you accept the terms for crawling and indexing.

That’s why we’re seeing a reduction in robots.txt requests from AI bots — there’s no real legal precedent that visiting a robots.txt file, the way Agentic or OpenAI’s crawlers do via Common Crawl, means you’ve agreed to the site’s terms.

I don’t think that’s going to hold up in court.

Bill Scully (40:25.553)
I haven’t heard about people creating unique terms and conditions within a robots.txt file.

Bill Hunt (40:29.998)
If you look at one generated by Cloudflare, it literally spells it out.

Bill Scully (40:37.825)
Interesting.

Bill Hunt (40:39.160)
It’s written like a contract — that by fetching it, you agree to terms. I’ll put a link in the show notes to where they explain it. Here’s the other thing: if you set up on Cloudflare today on a lower tier, all bots are automatically blocked, and you can enable their robots.txt file with one click. But here’s the crazy part, Bill —

Bill Scully (40:42.448)
I haven’t seen that — I’ll have to check it out.

Bill Hunt (41:04.529)
The robots.txt file is actually evaluated after the gate at the CDN level.

Bill Scully (41:09.554)
Yeah, I’ve seen that — it gets blocked, and it’s not even reflected in the robots.txt file.

Bill Hunt (41:13.331)
Right, because it’s handled in the WAF rules, which break things out by which AI bot it is and the purpose — training, RAG, or an agent bot, like something built with a Claude Skill. That’s why some of these AI agents, especially second-tier ones, aren’t even fetching robots.txt anymore —

because they don’t want a record of having fetched it, which could be used to argue implied acceptance of terms, even though there’s technically no explicit agreement. It’s a one-way thing, but functionally a pseudo-contract. That’s the clarity piece. And remember when they introduced the “toll booth” concept, regulating who could access content through payment —

that’s when I built out a whole framework for how AI would decide: do I actually need this content badly enough to pay sixteen cents to fetch it? And if I’ve already fetched it once, do I need to fetch it again?

Bill Scully (42:10.719)
Mm-hmm.

Bill Hunt (42:15.513)
Right — why spend a dollar fifty fetching the same content ten times? That’s why we need to think more broadly about this. Everyone gets excited about selling their blog content for five cents a visit, but once an AI system has it, it doesn’t need to visit again. What did you materially change that would make it worth paying again to re-fetch?

Bill Scully (42:21.987)
Right, right.

Bill Hunt (42:39.682)
This is why publishers are saying, “We’re just blocking Google and all the bots.” They’ve retooled their business models around loyalty, apps, and tools — building something people actually want to use instead of just being a traffic middleman. They’re saying, “We don’t actually need Google’s crappy nineteen percent of our total traffic.”

Bill Scully (42:52.382)
Mm.

Bill Scully (43:00.065)
Yeah.

Bill Hunt (43:00.370)
That’s the reality of it. I think this opens up some of what you were getting at — it’s all about value to everyone involved. Is there a purpose? Is there value in coming to get it, better than what they already have? Things like that. It just makes sense.

Bill Scully (43:23.666)
Yeah, I think we’ve covered a lot.

I’m sure most people listening are struggling with the reduction in pages being indexed, and I think we’ve covered the reasons pretty well. But let’s

Bill Scully (43:49.731)
—go through some things people can actually do today.

Bill Hunt (43:53.835)
First thing everyone should do: if you have Screaming Frog — I don’t know if other crawlers do this too — it’s painless. Get Screaming Frog, connect it to your Search Console account,

and crawl your site. If you have a larger site, break it into chunks of around 1,800 URLs — you can pull a thousand, or up to 2,000, through the API. Run each chunk over a few days and look for patterns in how many pages Google is aware of but hasn’t indexed, or has crawled but not indexed. Going back to your point, Bill —

if you find those pages, be very critical about what you actually consider “critical.” Just because a page is on your website doesn’t mean it has value. I’m sorry to tell you your baby’s ugly, but really apply a tough rubric.

Bill Scully (44:50.613)
If your site went down, would it matter to anyone but you?

Bill Hunt (44:55.565)
Exactly. So look at that. But going back to gate number one — log into Search Console today. It’s free, costs you nothing but time.

Look at your “crawled but not indexed” ratio. If it’s worse than roughly one-to-one — ideally you’d have zero errors, but that’s nearly impossible — say you have a hundred errors out of ten thousand pages, that’s not bad. But look at the actual errors: if you’re intentionally blocking things, fine, but if Google flags something you didn’t mean to block, that’s worth investigating.

Bill Scully (45:37.591)
Totally agree.

Bill Hunt (45:38.051)
Are those 404s intentional? You can rule some of those out. But when you’re looking at redirects, canonical issues, or “aware but not indexed” / “crawled but not indexed” pages — especially if you’re using the API or Adam Ghent’s tool — it’ll tell you how many days since Google became aware of the page and when they last visited.

Look at those ratios, then remove anything broken. Fix your 404s. Fix your redirects.

Bill Scully (46:10.665)
Especially in the sitemaps — get them out.

Bill Hunt (46:12.717)
From the sitemap, right. I just wrote a guide for a company that was pushing back on this. Here’s the thing: if someone mistyped your URL on their blog, the only way to fix that broken link is to ask them to fix it. But if you have a 404 on your own site, from your own link, you can fix that yourself.

If you have a redirect sitting in your XML sitemap, your HTML sitemap, or — God forbid — your menu… I had one case recently where a company had 4,000 broken 404s reported. It turned out to be a single broken URL in the site header, showing up 4,000 times.

So it was technically one broken header link, reported 4,000 times. Fixing the header URL — however long that takes, sometimes months depending on the company — makes all 4,000 of those 404s disappear at once, because they can no longer be 404s. So fix the things you can fix.

Bill Scully (47:10.935)
Mm-hmm.

Bill Hunt (47:25.618)
And then there’s things like redirect chains. Did you see Google’s warning about site search spam? If people are spamming your site search with parameter variables, once they know your site search function exists, they append spam strings to your search URL, run a bot against it,

build a fake sitemap, inject it, and get Google to crawl thousands of fake “real” pages.

Bill Scully (48:02.313)
Ha, I hadn’t heard that.

Bill Hunt (48:03.067)
Right — this is site search spam, and Google’s response is basically: your site search result pages don’t need to be indexed at all. It’s an exploit that can be solved by simply blocking your site search from indexing. SEOs push back — “no, that’s a gold mine of query data” — but Google’s saying they don’t want that indexed.

Bill Scully (48:25.824)
Right. And there’s a second step too — if people are searching for something on your site, chances are they can’t navigate to it normally, so fix that, or create content for it.

Bill Hunt (48:27.954)
Fix all of it.

Bill Scully (48:41.750)
That’s another friction story.

Bill Hunt (48:42.130)
Right. And the other thing we’ll put in the show notes: for the information-gain side of this, if you run the crawl we talked about, or just export the list from Search Console for free — all the URLs Google’s crawled but not indexed — look for a pattern, like a shared folder or page type. There’s a prompt I wrote for this,

building on a similar prompt from Marie Haynes — I’ll link both. Take all those URLs, load them into a CSV or Excel file, feed them into AI with the prompt, and run it. It looks for information gain first, because Google’s

telling you it didn’t want the page for some reason. It’s looking for light content, duplicate content, junk content — using the rubric she and I put together. If there’s no information gain, the prompt tells you which page it duplicates, or explains why there’s a lack of gain. Then you can task someone to fix it. If it’s duplicative of another page,

and that page is indexed, you can consider merging them. If it’s an exact duplicate — or in my tool’s case, gives you an actual duplication percentage — and it’s over roughly 85%, you make the call: merge it in, or add to it. If you merge, 301-redirect the old page into the surviving one. Go through those steps to add information gain, and

don’t just use a tool to tell you what everyone else has. Sit down and ask: what’s my claim to fame here? How does what’s on this page relate to my business? How does my business apply it, or think about it differently? Add that kind of content — it’s what will make the page genuinely different if the issue is a lack of external information gain.

Bill Scully (50:42.241)
Yeah — before publishing a page, ask: what will the reader know or be able to do after reading it that they couldn’t get from the existing results?

Bill Hunt (50:52.463)
Exactly.

Bill Scully (50:54.608)
Add value — proprietary data, interviews, your own experience, original examples, evidence.

And you could provide tools as well, to help people do what your page is talking about — that seems to be a newer approach.

Bill Hunt (51:17.305)
The example I always come back to is Gun Dog Supply. They sell hunting-dog gear — collars, remotes, trackers, training dummies, that kind of thing — similar to some of the suppliers and component sellers you work with. What most people do, and what you have to break clients of, is—

Bill Scully (51:21.517)
Is what?

Bill Hunt (51:41.072)
Most people just take what’s on the box label or the spec sheet and put it on the website — which is fine, but what I like about Gun Dog Supply is that they add their own stamp on top of it. There are two brothers, Rob and Steve. Steve goes out, hunts, trains his dog, and adds his own commentary — not just an FAQ bolted on, but real additions to what’s on the box.

Then they add videos on top of that. So it’s not identical to the Amazon listing or the manufacturer’s page — they’ve added real-world experience. Same idea applies to, say, a pool supply company in New England: what chemical composition works best when you’ve got rain one day and a hundred degrees the next?

You’ve got green stuff, blue stuff, purple stuff happening in your pool — whatever it is. Same with grass types or sprinkler systems. Instead of just saying “we service sprinkler systems,” explain: when do I do my seasonal switchover? What does that involve? Why does it matter? Again, not just an FAQ — there’s no FAQ rich snippet anymore anyway —

add to what everyone else already says about why you need a certain kind of grass. If you’re in New England — eight states and a lot of ground to cover — and you’re a Connecticut landscaper, give me something specific to Connecticut landscaping. If this is your business, there’s no better

source of information about it than you — that proprietary knowledge just needs to get out of your head and onto the page. Easier said than done for most businesses, but get an AI recorder and talk into it while you’re out mowing someone’s lawn.

Bill Scully (53:24.353)
Yeah. Yeah.

Bill Scully (53:32.420)
That’s a great idea — you can get one of those, right?

Bill Hunt (53:37.526)
I use Audio Pen. When I’m on the treadmill, I talk into it on my phone, and when I’m done, it can format the output — I can choose whether I want it as an article, a rant, a newspaper-style post, or just notes. I do that whenever something pops into my head. Imagine you’re driving to a job site,

or you’ve got a new set of chemicals — I’m not talking about producing a full video, just talking it out. Earbuds in, hands-free, you say what’s on your mind, dump it into AI, and ask it to chunk it into, say, five pieces on pool chlorine for Connecticut pools.

Use your downtime, or the time you can talk to yourself like a crazy person, to record this stuff — let AI process it, then turn it into page content that’s actually additive.

Bill Scully (54:38.019)
Yeah, yeah, yeah. All right, I think we nailed it.

Bill Hunt (54:46.322)
That was a lot.

Bill Scully (54:47.167)
That was a lot. Anything you want to finish up with?

Bill Hunt (54:52.549)
No, I think that covers it. Reduction in crawling is here to stay, and the more we can make content easy and valuable to crawl, the better it is for everyone.

Bill Scully (55:04.751)
I’d encourage everybody to check out Bill’s article, “The New Economics of Web Crawling: A Framework for Selective Content Ingestion,” on billhunt.com. That’s right?

And that’s our call for ratings — we hope you’re finding value in these conversations, and if you’ve made it this far, I’m sure you are.

Bill Hunt (55:14.773)
Yep. We’ll put a link in the show notes.

Bill Scully (55:28.439)
We’d really appreciate it if you could take a moment to rate and review the podcast on your favorite platform. We also want to hear from you — your feedback helps motivate us to keep making these episodes and plan future ones. Let us know what you’re thinking, so we can answer your questions. And with that, I think we’ll call it a wrap. Thanks, Bill.

Bill Hunt (55:51.000)
Cool. Thanks, everyone.