In this episode, Bill Hunt and Bill Scully talk with Adam Gent from Indexing Insight to explore the often-overlooked world of Google indexing performance. They discuss why many SEOs under-prioritize indexing, the limitations of Search Console, and the real meaning behind “crawled previously indexed” pages.
Adam explains how Google assesses page quality using engagement, backlinks, and benchmark scores, and why pages frequently move in and out of the index—especially during core updates. The episode highlights the importance of monitoring index rates, understanding quality signals, and using page value management to identify content that may be holding back visibility.
A must-listen for SEOs looking to improve how Google sees—and indexes—their content.
Listen now on your favorite platform:
Youtube: https://youtu.be/pm1UtKeQYnU?si=OQg0trxPCnOXTJU6
Spotify: https://open.spotify.com/episode/6Uat7ylqk09FkPjgAYA8V0?si=FvQjYEEpRQOJTsXPOt4dFg
Apple Podcast: https://podcasts.apple.com/us/podcast/google-indexing-with-adam-gent/id1755789311?i=1000737416709
Key Takeaways
- Indexing performance management is often not prioritized by SEOs.
- Google Search Console provides limited insights into indexing issues.
- Crawled previously indexed pages indicate active removals from the index.
- Page quality is assessed based on user engagement and backlinks.
- Benchmark scores influence which pages are indexed or de-indexed.
- Pages can fluctuate between indexed and not indexed frequently.
- Monitoring index rates is essential for understanding search eligibility.
- Quality signals are crucial for determining what gets indexed.
- Active removal of pages is common during Google core updates.
- Page value management can help identify underperforming content.
Show Note Links
Indexing Insight Tool: https://indexinginsight.com/
Indexing Insight Newsletter: https://indexinginsight.substack.com/
Transcript
Bill Scully (00:00)
Welcome to episode 39 of the International Web Effectiveness Podcast, where Bill Hunt and I, Bill Scully, focus on the strategies and techniques necessary to improve your web presence and manage your global websites.
Before we get into it, make sure to follow the International Web Effectiveness Podcast wherever you listen, so you don’t miss an episode. And if you enjoy these conversations, please leave us a review. It helps motivate us to keep making more shows in the future.
In the last episode, we reviewed the major topics challenging our industry, such as Google’s removal of the num=100 parameter.
Today we’ve got another fantastic guest: Adam Gent from Indexing Insights. We’ll be chatting with him about the state of indexing—Google’s shrinking index, large-scale sites quietly losing thousands of URLs, and the big question behind it all: why? And hopefully, what SEOs can actually do about it.
We’ve prepared a few questions for Adam, but this will be an open conversation. I’ll start with the first one.
So Adam, why is indexing performance management not viewed as mission critical?
Adam Gent (01:28)
In my experience, it’s because it’s very difficult to monitor and report on. You get a free tool in Google Search Console, but for many websites—those with millions or even billions of URLs—you’re dealing with an overwhelming number of data points. You can download a thousand rows at a time, and then you have to add properties one by one to your Search Console account. That’s hard to do at scale.
You can work through the Google Search Console API, but that’s also very difficult. So in my experience, that’s why people don’t treat indexing performance as mission critical. It becomes “just another data point,” and to be honest, it’s not well understood.
When I started building a tool that monitored indexing behavior, I realized I didn’t fully understand what I was looking at until I started seeing patterns shift dramatically. That changed how I understood indexing. And now, especially with the impact of AI, it’s surprisingly important to understand what’s happening inside the index.
Bill Scully (02:31)
Yeah. I think Google Search Console doesn’t really give you that information clearly. It’s not intuitive at all. When you talk to most SEOs, they’re using it to check rankings. They’re not looking at the status of pages.
And like any database, if you only look at it from one view, you miss the connections between the other views.
I reached out to you because I saw your presentation deck from Brighton SEO. You had this concept you call “crawled previously indexed.” To me, that should be the standard view people look at. Everything else feels like looking in the rear-view mirror.
Adam Gent (03:25)
Yeah — let me tell you a story about that.
“Crawled previously indexed” means a page that was indexed but was then actively changed to not indexed by Google. In Google Search Console, this ends up in the “Crawled – currently not indexed” report.
When I first built the tool behind Indexing Insights, this active removal was happening so frequently that we actually had to create a new indexing state for it. It was that common in the data. So now our customers can see exactly which pages are being actively removed from Google’s index in real time.
And you won’t really see this in Google Search Console. Or rather, it’s there, but Google doesn’t surface it clearly.
Here’s a fun fact: the “Crawled – currently not indexed” report is sorted by last crawl time. If you have 2,000–4,000 pages in that state, and Google hasn’t recrawled a page for 130 days, it gets actively de-indexed. Those pages end up at the bottom of the report—so you may never see them.
What you do see at the top are the recently crawled pages: pages Google looked at but decided not to index again. But the actively removed pages are buried.
Bill Scully (05:09)
Right.
Adam Gent (05:21)
So on large sites, you often won’t see pages that are being actively removed from the index. That’s why customers are using our tool—because they can see it happening in real time and take action. That’s not something you can easily do in Google Search Console.
And honestly, you wouldn’t know any of this unless you played around with the data the way I have. Google doesn’t make it clear in their documentation, which is incredibly frustrating.
Bill Scully (05:45)
Yeah, yeah. I think you described it really clearly. I always question the data because it’s only a subset in Google Search Console. But I never gave conscious thought to the fact that the report basically cleans itself every day. If you had a huge drop in the past, it would be gone—you would never know it unless you had looked at it within, what, 90 days?
Adam Gent (06:22)
Yeah, something like that. The trend lines in the reports are 90 days. But from our data, there’s a 130-day indexing rule. If a page is not crawled—and by the way, people often think that if Google recrawls a page, it will automatically get re-indexed—it doesn’t mean that. It literally means the crawler hasn’t come back; it hasn’t recrawled the page.
Bill Scully (06:24)
Right.
Adam Gent (06:48)
And that means the state in the index changes independently. Crawling and indexing are separate processes as far as I can tell. That’s what you’re seeing in the page indexing report.
Bill Scully (07:05)
I haven’t had a chance to look at your tool, but does your tool map the last crawl date with the… wow, really?
Adam Gent (07:05)
Yes, we can. In our support documentation, we have a page about this. We have something called the crawl coverage report, which breaks down all of your data into time buckets. We take the last crawl time, calculate the days since the last crawl, and group pages into 30-day, 60-day, 90-day, and 130-day buckets.
We can tell customers which pages are likely to be actively removed from Google search results and put into the “not indexed” status because of the last crawl time. The trend was so prevalent that it became part of our reporting. The 130-day indexing rule isn’t my rule—it was discovered by some professional SEOs in Europe—but we have the data to back it up.
It was such a strong trend that now, in our reports, any page past 130 days has a 99% chance of being de-indexed. Customers can see which pages are likely to be removed before it happens.
Bill Scully (08:30)
Wow. So I’m going down a rabbit hole here, but obviously that’s a leading indicator of problems. Right? If you see, say, a section, page type, or template trending toward the 90-day “not crawled” range, there’s a high risk of those pages being dropped at 130 days.
Adam Gent (09:01)
Yeah. From what I can tell—reading the technical documentation, DOJ trial documents, and so on—the index essentially tells the crawler what to do. The crawler recrawls pages based on signals in the index and the information Google has previously processed.
If your pages aren’t—air quotes—“high quality” (and we can define quality, which I talked about in my Brighton SEO talk), Google will crawl them less. Eventually, your pages can be actively forgotten. Over time, at about 190 days, if a page hasn’t been recrawled because of signals in the index and the crawler doesn’t come back, the index status changes to discovered – currently not indexed.
That means Google is forgetting your pages because the quality signals just aren’t there. Eventually, if the page is never recalled, it becomes a URL unknown to Google. The funny thing is, you’ll never see this backward movement in Google Search Console. It doesn’t show URLs as “known to Google” unless it’s never seen them before. Most of these pages end up in the discovered – currently not indexed bucket.
I wrote a newsletter and a blog post on Indexing Insights about this. I found that roughly 90% of pages in Discovered – Currently Not Indexed are actually URLs unknown to Google. Essentially, Google forgets these pages over time due to the lack of quality signals in the index.
Bill Scully (11:08)
Well, yeah. Let me stop you there. I can see my brain going into engineer mode—so many questions must have come out of this conversation.
Bill Hunt (11:21)
Well, I mean, I’m sort of letting you drive it because I know this is a very passionate topic for me. I take a much less gentle approach because I think it’s ridiculous that people don’t pay attention to this. It’s like wanting to win the lottery but not buying a ticket.
Adam, you highlighted reasons, and I’ll wrap them in excuses. We can monitor a lot of this. Having spent 30 years at the enterprise level in search, I’ve dealt with this. Everything doesn’t have to be 100% inclusive—let’s start with our mission-critical pages and make sure they are covered.
In all the big companies I’ve worked with, we created multiple accounts in Search Console at the product-category level, at the folder level. That way, we could see and understand data for specific sections of the site. Yes, it’s hard. Everything in SEO is hard. It requires effort.
One thing I want to point out: at Search Central Live in Tokyo, Google had a really good slide—it was in Japanese, but I’ve taken notes. They broke this out well. Bill, you and I have discussed this before, but I don’t think enough people realize it: Google is as dysfunctional internally as everybody else. They have a crawling team, an indexing team, and a serving team—and these teams rarely talk to each other.
Adam, exactly as you described, there’s some signal, some trigger, some sensitivity telling Google “go get this” and “don’t get that.” If you think about the crawling and indexing process: Google encounters a set of URLs—either from external sources or because they discovered them on your site via your XML sitemap or other methods.
I’ve been trying to write about this for probably six months. I gave a keynote at PubCon about the importance of understanding indexing. Bill, you probably saw my reaction—it’s all about compute. One speaker, now on spam defense, shared numbers about the sheer effort required to deal with spam: 7 trillion URLs, more than half of which are garbage. Legitimate spam, garbage—they have to sift through all of it.
So you start thinking: where do we put our effort to go fetch something? Adam, as you’re seeing with your tool, or if you pull this data into Airtable or another platform, you can start to see what hasn’t been encountered. This is why tracking indexing is so important.
Bill Scully (14:03)
Hmm.
Bill Hunt (14:26)
Let me give you a perfect example. I know a company that won a project to implement schema on 15,000 pages. They installed the schema, but six months later, their client said, “I’m not seeing the performance you promised with featured snippets and rich results.” The agency responded, “Well, it takes time, Google’s doing this…”
When I was called in, the first thing I did was use Screaming Frog and the API. Every day for a week, I pulled 2,000 points across these URLs. I discovered that 98% of the URLs with schema hadn’t been indexed.
If a page hasn’t been indexed—especially if the schema is injected through JavaScript and hasn’t been rendered—Google hasn’t seen any of it. So any changes you make, any material SEO effort, or geo-specific content doesn’t matter until Google revisits and re-indexes the pages. Only then can the performance clock start ticking.
My point is this: if we have a set of critical URLs, we should use tools like yours—or any tool that tells us when a page was last seen—as an indicator. If it hasn’t been seen, it obviously can’t start affecting the performance we expect.
This is fundamental to an SEO’s job. There’s no excuse for not doing it. In fact, this company built it into their workflow: testing and monitoring indexing is now part of their onboarding and ongoing monitoring protocol.
I love seeing tools like yours. One question I had: you mentioned in your intro that you learned a lot about crawling and indexing while building your tool. That’s one thing I’d love you to expand on: what are two or three things you learned? You mentioned “seen but not indexed” and what that means, but what are some other nuggets you discovered by being deep in the data, figuring out how it works, and presenting it in an actionable way? What have you seen that most people haven’t yet discovered?
Adam Gent (17:04)
Good question. So in terms of lessons from me—lessons from building the tool with my tech co-founder—number one: your page can be indexed one day and 24 hours later, it can be de-indexed.
Checking the indexing status is not always indicative of what’s happening. It’s like in search analytics: impressions go up and down, average positions shift—you think something’s wrong because something is happening. A page can literally be there one moment and gone the next.
Here’s the interesting part: it can fluctuate. A page can disappear one day and be back the next. So it goes index, not index, index, not index. I talk about this in a newsletter and in the Brighton SEO deck, but to quickly summarize: I think there is a benchmark.
Google manages its index using some kind of benchmark score for all its pages. It has to, because there’s a limit to how much it wants to index. Sure, it could index the entire internet, but it doesn’t want to because it’s a business—it’s expensive. So this benchmark score comes into play.
Bill Scully (18:30)
You…
Adam Gent (18:35)
That benchmark score works with what’s called a soft limit, just below the hard limit. The soft limit is a buffer so they don’t hit the hard limit. If you have too many pages indexed that are eligible to appear in search results, and you go above the soft limit, your benchmark score determines what stays indexed. Anything above the benchmark score is indexed; anything below it is not.
Bill Scully (18:59)
You…
Adam Gent (19:04)
If the benchmark score shifts upwards, more pages get de-indexed because they don’t meet the score. If it decreases, more pages get indexed. So pages can jump in and out of this quality “benchmark score.” It’s dynamic.
Bill Scully (19:14)
Yep.
Adam Gent (19:33)
Exactly. I’ve seen people check a page and it’s indexed, then check again and it’s not, then a week later it’s indexed again. That’s because the benchmark is dynamic—it changes daily based on global search demand, search interest, what’s in the index, what’s eligible to appear, and what’s not.
The third key point, which I didn’t think about at first, is that pages are actively being removed from Google search results at scale. When I looked at the “Crawled – currently not indexed” report initially, I thought pages were just waiting in a queue. No—they’re assessed, crawled, processed, and actively removed fairly quickly.
Lots of low-quality content gets indexed temporarily—like product pages with just an image, a title, and a buy button—then they’re removed. Google has to index, process, and remove because there’s so much spam. There are thousands of use cases they consider for what should be indexed. Some content we’d consider low quality still gets indexed because it doesn’t violate one of those use cases.
Bill Hunt (21:44)
And I think that’s where that benchmark score comes in.
Adam Gent (21:44)
Exactly. And there’s even more to it, but those are the three main points.
Bill Scully (21:53)
Let’s dig into that a little because I have some questions. Were you going to ask something?
Bill Hunt (21:57)
No, I was just going to say that benchmark score. So, I’m working on a piece and we’re also building a course for developers—sort of SEO for developers—and we’re taking a true developer mindset. One thing we’re asking developers to do is brainstorm benchmark criteria: if we had limited resources to ingest content into our search engine, what limits would we set?
It’s interesting to think about needs and trends. The Google Trends guy came from Israel and mentioned that about 15% of everything they see is net new. So there are various variables they apply. Another factor is whether content is referenced elsewhere. I’ve been building a grid for this, which I’ll share once it’s finished. I hesitated at first because I didn’t want to deal with community commentary, but now I just don’t care.
The idea is: where would I limit indexing? What’s in the news, what’s in demand, and is it additive? That’s why we might see more local content getting pulled in.
Then there’s the question of quality between crawling and indexing. Bill had a question for you about canonicals, and this connects to a previous Search Central Live episode. Someone asked Google about duplicates, and Google explained they have a duplicate team within crawling, which sits between crawling and indexing. They define duplicates, handle canonicals, and decide how tags are applied.
This gets even more complex with rendering. If you’re injecting hreflang, canonical tags, or schema via JavaScript, it may be applied instantly—or there may be a lag because Google has to render it.
Another factor I’ve heard referenced multiple times recently is compute. Google is shifting resources from crawl and indexing to their AI operations.
Bill Scully (24:38)
That’s what I was thinking was happening. Go ahead.
Bill Hunt (24:41)
Right. They’re being very selective. Adam, I think your benchmark score is really in flux as they try to prioritize content they actually need to fetch.
A few other points:
First, this explains why some pages are dropped or not revisited for long periods—they’re not additive. Google already has certain knowledge “locked” in their model. For example, the capital of France is Paris. They don’t need 9,000 pages to confirm that. They need one true source. If we want to know what’s happening in Paris today, then we need RAG—retrieval-augmented generation—to fetch fresh content.
Second, Cloudflare gating may impact crawling. If you’re using Cloudflare and it’s managing your robots.txt (the default now), it adds long legal text. I wrote an article about this: just because Google sees a statement in your robots.txt doesn’t mean they’ve accepted it as a contract. Some sites reported crawl rates dropping almost to zero after adding such statements. Google had no comment.
Finally, all this crawling can create huge hosting costs. Small sites have reported $5,000–$10,000 bills because of crawl load from AI and related indexing.
So Adam, in your monitoring—since you’re mostly mining Google data—do you see AI-driven RAG and indexing needs making it more critical to monitor index rates and inclusion in Google? Especially since Google has said Gemini and their other AI products rely on indexed content as a primary source.
Adam Gent (28:13)
That was a lot, but I’ll try to answer. I think all these points are tied together, so bear with me. I’ll start with page quality and then connect to AI, since they relate.
These are my thoughts after looking at data, reading about it, and watching pages get de-indexed. For example, the DOJ documents show…
Bill Scully (28:37)
Hmm.
Adam Gent (28:48)
The engineers—the people who are asked questions—talk about signals. And these signals… there are lots of different ones, but essentially, signals are pieces of information stored about every page that Google crawls and indexes. They call them Doc IDs.
Google doesn’t actually store things as page URLs like humans think of them. Instead, it stores the Doc ID of a piece of content and assigns it the canonical URL based on all the signals it finds. I think it’s something like 40 signals. Around each Doc ID, there’s PageRank, vector embeddings, knowledge graph connections—it understands where the page sits in the site architecture, the domain, the host, and more.
Google has lots of signals for every page. There are raw signals—things it literally extracts from your page like links, content, title tags, meta descriptions. And there are computed signals, like PageRank or vector embeddings.
What’s interesting—Martin Splitt mentioned this, and I found it very compelling—is that Google’s index is so large not because of all the page content it stores, but because of all the signals it generates around the content. The largest part of the index is these signals.
Bill Scully (30:22)
Hmm.
Adam Gent (30:28)
I found this interesting when doing research for my Brighton SEO talk and reading DOJ documents. They talk about three key computed signals: topicality, quality, and popularity. I won’t go into all three—they could easily fill a podcast—but quality is especially interesting.
Page quality has been discussed a lot by Googlers over the years. John Mueller has emphasized page quality in his hangouts. Google talks about the quality of the page, the website, and the user experience. Interestingly, DOJ documents show Google measures things like user engagement, backlink quality, and content coverage within a topic. They use vector embeddings to understand how a page fits into a topic and whether similar content already exists online.
For Google, the difference between websites isn’t just the content itself—Google has said it doesn’t really understand content quality in that sense. The difference comes from engagement signals: how many people visit the page, whether the domain gets traffic, how pages are internally linked, and the site’s link architecture.
All of these signals are aggregated into a page quality score. One of the biggest signals is user engagement—what they call “good clicks.” This data isn’t available in Google Search Console, but it’s mentioned in DOJ documents. Essentially, it’s about how people interact with a page in search session logs: do they stay on the page, revisit it, or bounce around?
Why does this matter for AI? ChatGPT doesn’t have decades of experience building quality algorithms. It’s essentially a super-advanced autocorrect, pulling averages from the internet. Google’s index, however, is built to evaluate quality in a way that aligns with user behavior: lots of clicks, sustained engagement, and backlinks indicate high-quality content.
The tricky part is that what Google considers quality isn’t necessarily what we think quality is. For Google, quality is defined by user interaction: people click on a result, get their answer, stay on the page, and engage. It doesn’t necessarily mean the content itself is objectively high-quality.
When it comes to AI, Google can leverage this index to understand quality in a way that AI models like ChatGPT cannot. So if you ask me whether the index is important, I’d say it’s massively important for the future.
Bill Hunt (34:24)
Well, those are proxies for quality.
Adam Gent (34:43)
In a zero-click world, knowing if a page is eligible to appear in search results—if it’s indexed—helps you understand if it’s actually answering questions. If a page hasn’t been crawled in 90 days or 60 days, that could indicate it’s not being prioritized. As technology improves and AI gets faster, crawl cycles may speed up, but if your page hasn’t been crawled in a while, it might simply not be answering a question.
You can then pull in log file data or referral traffic from AI (if available in detail through analytics) to get a better understanding. That’s why I think indexing matters. It’s a proxy for quality. If a page is eligible to appear and the indexed green bar indicates it’s above a certain benchmark threshold, you should have the data to know which pages are green and which are gray.
If pages are gray and actively being removed, as I see daily from customers, you need to understand why. Sometimes it’s internal linking. Most often, though, it’s because nobody visits these pages—nobody engages with them.
So why publish 10,000 pages if nobody’s going to bother with them? Google will de-index them. That’s why monitoring indexing is so important.
Bill Scully (36:16)
Yeah.
Bill Hunt (36:18)
Right. Along those lines, indexing is a proxy for quality and need. For example, if you create a mega glossary with 10,000 entries, hoping to rank for a featured snippet or provide an AI overview for something like “Supply Chain Management,” it probably won’t work. That information may already be locked in elsewhere. The real question is: what value do you add? If you’re SAP, for instance, how do you add unique insight to supply chain management?
I was doing a workshop for developers on AI and testing Google AI mode. I asked, “Why didn’t this brand get chosen?” In the results, Google explained that the content was deep in the site—five levels down—and full of “fluffy marketing” language, rather than being directly linked from the homepage with clear, actionable content. This shows how surfaceable and quality content matters.
So if content isn’t being seen, revisited, or re-indexed, any restructuring, schema, or linear content formatting won’t have an effect. That’s why monitoring indexing is critical.
Another point is overall quality. I had someone reach out after a replatforming and domain migration where they completely changed their URLs. In Search Console, we looked at the report showing good pages versus errors. 96% of Google’s pings returned something other than a 200 indexable status—meaning Google encountered barriers. Their developers and SEO team initially thought this wasn’t a problem, assuming it was because Google hadn’t done a core update.
But the reality is, if Google tries to crawl your site repeatedly and encounters errors—imagine knocking on a door ten times and it’s always locked—Google is less likely to return. For every good page, there were ten error pages. Nobody in that group of fifteen people recognized that as a problem.
That’s why I emphasize cleaning up your site, making content valuable, and teeing yourself up for success. The more organizations elevate this, the better. What you’re doing—monitoring indexing and quality—is directly tied to overall web effectiveness.
I’ll be quiet for a moment to give you a chance to talk about where you see things going with your solution, and Bill may have one or two follow-up questions, but we want to respect your time.
Bill Scully (41:13)
I’ll just ask a quick question because we want to get to the good stuff that Adam has. I think there are at least two things forcing pages to be de-indexed. One is their capacity for indexation. Bill, you mentioned they’re giving compute to AI, so now there’s less space to keep stuff indexed. That number changes.
The other factor is quality. If your quality is low and you drop out, at least in my experience, the only way to get Google to come back at scale and relook at all those pages is to make a shift on your site so that during the next core update, Google decides to recrawl the whole site. Is that your experience? You must have some experience in recoveries with your tool, right?
Adam Gent (42:13)
Yes, but it isn’t exactly the way you think. Some of our clients—this leads into what I’m calling the future concept of page value management. One thing I’ve always done as a consultant is look at which pages are contributing effectively to SEO performance and which aren’t.
If you pull in different metrics—popularity, quality in terms of links, internal linking, impressions, clicks over time—you can do a kind of content audit. What I noticed, and now teach clients, is that there’s a flaw in Google’s system. We talked about page quality and using signals for crawl and indexing. If a page is above the benchmark score, it gets indexed.
But what if your site is moderately authoritative or has good authority in your niche? If you start publishing tens of thousands of pages, Google will index them. Here’s the interesting part: every page that’s indexed contributes to the overall quality of your site or a site section.
So I really look at what pages are performing well versus not. If you have many pages that aren’t contributing…
Bill Scully (44:06)
Thank you.
Adam Gent (44:18)
They’re dragging your site down. I’ve seen sites with millions of pages, where only 16% are getting traffic or impressions. The other two million indexed pages aren’t contributing at all. That’s a problem.
It’s like your shop analogy, Bill. People walk in, the shelves are mostly empty or spoiled, and there’s only one or two shelves of good food. They won’t stay. That’s what I think people miss.
A lot of content gets published without anyone stopping to take stock of what’s contributing and what’s not. Our tool helps people capture which pages are indexed, which aren’t, how frequently they’re crawled, and then analyze which pages are adding value. Often, clients are surprised. Google won’t revisit pages that don’t contribute—there’s no search volume, nobody’s linking to them, the content isn’t as good as competitors, and sometimes they aren’t integrated into social media or other channels.
Google aggregates signals for topicality, quality, and popularity. Even on massive authoritative sites, it takes longer for these effects to happen, but it does happen. In my Brightness.io deck, I show an example: 95% of new pages were indexed, but 5% were actively removed. My guess is it’s due to lack of interest, low visits, or maybe poor internal linking.
As for AI, every page contributes to your site’s overall quality score. When Google does core updates and pages drop out, how will you know what’s appearing in AI if you don’t have click or impression data? That can drop by up to 50% when the num=100 parameter is turned off. You see what I mean?
Bill Scully (47:14)
Nice work.
Adam Gent (47:17)
How do you know anything? Well, indexing and index status is a good leading indicator to understand which pages are appearing in search. That’s what “indexed” means—eligible to appear in search. And if you have too many pages indexed because Google is a greedy goose, and you have a flaw in the system combined with high authority—hundreds of thousands or millions of pages not contributing—it drags down the rest of your site. You stop appearing in AI, in Google’s AI features, and it could be because you’re not taking stock of what is indexed and what’s contributing to your overall search health.
That’s where I want to take my tool. In the next few months, I’m going to launch a service to help people understand this because there’s clearly something there. I’m helping people who, as you’ve said, often don’t take stock or don’t have the time, the data, or sometimes even the capability to understand all of this. So that’s where I want to take Indexing Insight—helping people like you quickly analyze and understand where the gaps are, which pages should be reviewed, rethought, or simply reconsidered. Sometimes it’s about looking at the data and admitting, “Yeah, this isn’t contributing.”
Bill Hunt (48:50)
All right—
Bill Scully (48:52)
Right. Well, I want to be conscious of your time. I know you’ve got a hard stop.
Adam Gent (48:57)
Yes. Yeah.
Bill Scully (49:00)
This is a great conversation. I think we’re going to have to ask you to come back. I have a lot more questions.
Adam Gent (49:05)
I mean, I loved it. I loved this conversation. This was great. Thank you for having me on, by the way, Bills. I really appreciate it. You’re the first people to ask me on a podcast night after my Brightness talk, and I really appreciate that.
Bill Scully (49:11)
Yeah, you must. And… yeah, so why don’t you tell us where people can find you, find your tool. I want to give a shout-out for his newsletter. His newsletter is one of the three I absolutely have to read because it’s unique and based on data. The last three, I think, you did hit—
Adam Gent (49:20)
Sure.
Bill Scully (49:36)
—you talked about the massive deindexing in, was it May? I’m looking for my notes.
Adam Gent (49:44)
Yeah, there was an update, and something happened. It changed all the metrics. Everything got kicked out faster. I’m not sure what happened, but something happened. There is evidence, by the way, that every core update, Google actively removes pages.
Bill Scully (49:58)
And I think you’re one of the few people who actually talked about that. There wasn’t a lot of chatter about it, and I think it’s because people aren’t looking at Google Search Console the right way.
Adam Gent (50:05)
No—yeah, yeah. I caught it. I caused a bit of a ruckus with Google.
Bill Scully (50:15)
Yeah. Sorry, I didn’t mean to jump in there. Yeah—tell us everything about you and where people should go and learn.
Adam Gent (50:21)
Sure. So yeah, I’m the co-founder of Indexing Insight—indexinginsight.com. It’s a tool that helps you monitor Google indexing at scale. I’ve talked about some features during this podcast, but basically it’s an advanced version of the page indexing report. It gives you historic data for your pages, lets you see which pages are actively removed, and we try to make it as easy as possible to get the most out of your indexing data from the inspection API.
We’re two to three times faster than any other tool—at least as far as I can tell—when pulling that data. Nobody else seems able to get it as fast as we can, and it’s proven really useful for a lot of people.
There’s also a newsletter: newsletter.indexinginsight.com. Completely free. I share thoughts on indexing, interesting findings, topics, data points—things the Bills have said they really enjoy.
And yeah, that’s it. You can find me on LinkedIn—Adam Gent or Indexing Insight. Message me if you’ve got questions, or email support@indexinginsight.com if you have more. More than happy to help.
Bill Scully (51:40)
Great, thanks. Bill, final thoughts?
Bill Hunt (51:40)
Perfect. No, this was great. It’s fantastic getting somebody who’s into this every day and looking at it from multiple viewpoints. So, it was great. I really appreciate you having me on. And again, this is something I’m very passionate about, so it’s good to see other people are too. I really appreciate what you’re doing to increase awareness in this area.
Bill Scully (51:49)
Yeah.
Adam Gent (52:06)
Awesome. Thank you very much. I really appreciate you bringing me on. Thank you so much, guys.
Bill Scully (52:10)
And Bill, are you still doing the bi-weekly series on Searching for Journal?
Bill Hunt (52:14)
I am. Yep, they just renewed it for me to do another 12 to 15 articles.
Bill Scully (52:20)
That’s great. All right. Well, if you’ve made it to the end, we hope you’re finding value in our conversations. If you’re enjoying the show, we’d really appreciate it if you could take a moment to write a review. We also want to hear from you—your feedback motivates us to keep doing this. For all episodes and transcripts, visit internationalwebmastery.com. And until next time, thank you—and thank you, guys.
Bill Hunt (52:45)
Yep, thanks.
Adam Gent (52:46)
Thank you. Bye.