In a recent Google Search Off the Record podcast, John Mueller and Martin Splitt spoke with Allan Scott from the Google “Dupes Team” about how Google handles duplicate content.
Clustering and Canonicalization’s 40ish Signals
The first four minutes of the podcast are spent explaining and simplifying clustering and canonicals, which are key processes that Google uses to manage duplicate or similar content across the web efficiently. John explained that clustering groups pages the systems believe are the same, and then canonicalization selects the best version for searchers from those in the cluster.
Allan is not a fan of the term canonicalization, canonicalization is often blamed for the “wrong page being presented,” he frequently reminds Google engineers that canonical processing is a singular step after clustering URLs and not a magical black box that sorts it out.
Clustering is the initial step, where Google’s systems group URLs that appear to represent the same or nearly identical content. This step helps consolidate resources by treating similar pages as a single unit, reducing redundancy in crawling and indexing.
Once the cluster is formed, the canonicalization process determines the “canonical” URL—the best or most authoritative version of the page to display in search results.
Allan indicated that the duplicate team might use “something like 40 variables” of different weights, such as rel=”canonical,” redirects, sitemaps, and others, to determine which version is canonical. Search Engine Lands’ coverage listed many of the variables.
Together, clustering and canonicalization help searchers find the most relevant and accurate version of content while improving Google’s ability to crawl, store, and rank pages efficiently.
Ensure Every Signal Points in the Right Direction
The key reminder came near the end of the podcast at [29:42], when Martin said, “It is the job of the website owner to make sure that every signal points in the right direction. If you want one specific URL to appear in search results, then make sure that we can understand that it is the URL that should. Listening to the podcast, it became even more important to confirm that their signals and error management protocols are dialed in and that redirects, rel=canonical, and sitemaps are correct so that pages do not end up in the black hole.
Challenge of Conflicting Signals
A significant challenge for the Dupes Team is applying weighting logic to detect duplicates, especially when a webmaster sends conflicting signals. [06:11] The two most common would be 301 versus rel=”canonical.” Allan indicated those are both very strong signals, but what if your signals conflict? What’s going to happen is the system will start falling back on lesser signals like sitemaps or PageRank.
In Hreflang Builder, we have this specific error code of 301CAN for this scenario when a web page 301 redirects to another that has a canonical, either back to the original redirecting page or to yet another page. Unfortunately, too many web teams have told me Google should be able to understand what we mean, and this podcast confirms that they don’t and won’t.
Marauding Black Holes
The phrase of the episode was “black hole,” and it was used a total of nine times to refer to the duplicate page nightmare of “error pages,” which Allan affectionately referred to as “marauding black holes.” This reference starts at [16:21], with various error scenarios and an even bigger problem when these errors are served from a 200 status error page. The critical issue is referenced at [17:12] where Martin and Allan talk about a common e-commerce situation with a 302 redirect to a temporary out-of-stock notice page serving a 200 header, which will put those pages into the black hole and “never coming back out due crawl not really like dups.”
The Localization Iceberg
In the episode, there are seven references to the “localization Iceberg,” with Allan stating that many teams are dealing with the complexities of the “giant mass underneath” and the tiny sliver above the surface. Localization handling is highly complex and involves multiple teams “across the stack” with indexing, clustering, deduping, and serving.
What was specifically interesting to me was a response to Martin’s question about what I call aspirational hreflang where a site owner incorrectly uses hreflang tags to indicate a language other than the actual language. [07:16] What should be an apparent duplicate signal that the dupes team may not have noticed, with Allan indicating that would be a question for the “serving team.”
Hreflang is a Separate System
As I have stated forever, hreflang is a separate system, and the many hacks, workarounds, and deviations from standards end up not working for people because it is processed independently. During the episode [11:10] Allan references hreflang as a separate system that “sites on top of clustering and dups systems/
“So sitting on top of all of this talk about clustering, which is the dups system on its own, there’s hreflang, which is basically a separate system where, if you put in the annotations, we will try to substitute them.”
Hrelang’s Expansion in 2025
Allan also mentioned [11:10] that a project is “attempting to increase the reach of that specifically.” We want to serve more hreflang variants. We want to utilize that more, but we need to put in place mechanisms that will determine basically how much we can trust it on a given site. We’re doing some crawl and verification, basically, to determine, you know, is this site serving its map correctly, and if so, then we’re going to try to serve that more often without necessarily having to verify it as much as we currently do.
Note: I have advocated that your hreflang MUST be correctly implemented. Like everything, the more it can be trusted, the more likely it will be honored. Contact us to do an hreflang audit to confirm your implementation is perfect.
Potential X-Default Confusion
Another example of the localization iceberg is Allan’s explanation of X-default, which indicates that it is a signal and is “not inconsequential,” which means it is important and significant [12:55]. Allan explains how rel=”canonical” and X-default are distinct tools. The canonical indicates that multiple versions of a language page should be in the same cluster and without a specific location preference, which should be retrieved and shown to users.
I’m afraid I have to disagree that most websites get it correct. My research shows that it is one of the top 5 mistakes websites make, especially with decentralized hreflang management and different CMS implementations where the deployment market sets their site as the X-default when it has been set to another page in multiple other markets. In our hreflang errors course, I show numerous companies that have 4 to 10 different markets set as their X-Default, which is not correct.
Have a Conversation with Google Bot
I loved the section on dual signals discussion referencing rel=canonical and X-default; Allan made his point by having a conversation with conversation with Google’s systems. For example, if you have a page in your sitemap or linked that 301 redirects to another page and that page has a canonical back to the redirecting page that is a loop, what is a system to do? My most common Google System chat involves people who want their UK page to be indexed and use hreflang but listen to supposed experts who tell them to canonicalize it to the US.
He Google, I am glad you found my UK page (because I have it listed in a hreflang entry, my XML sitemap, and a link to it from many other UK pages). While I want it indexed to bring me significant SEO traffic, please respect my rel=canonical tag and not index it and show people in the UK the US page. Thank you. If you have this conversation with yourself, you will quickly see the error in the recommendation.
Related but Separate – Crawling vs. Indexing vs. Duplicate vs. Internationalization
In most recent podcasts, a key takeaway has been how many facets and interconnected systems it takes for Google to work. In this episode, the main focus was on the “Dupes Team,” whose job is to develop automation to review all similar documents in a cluster and pick the best one to present. Alan references an overlap with his team with crawling, serving, and internationalization, requiring alignment in their activities.
For example, there were references to rel-canonical at [25:09], where Allan mentioned trying to get the crawling team to integrate canonical handling into the crawl. He stated that when the crawler encounters a page with a canonical tag, there is no implicit directive to crawl the canonical destination if it has not been crawled before. This concerns me because, on the Internationalization episode Gary mentioned that when the crawler encounters hreflang, a crawling dependency is “triggered ” to verify the alternate page.
It was an information and nuggets packed episode that you should take the time to listen to.
