In this episode of the International Web Effectiveness Podcast, Bill Hunt and Bill Scully share their takeaways from the recent Google Search Off the Record Podcast. They break down important topics like managing duplicate content, tackling localization challenges, and using hreflang correctly.
The discussion covers how Google handles crawling and rendering, the concept of the localization iceberg, and common struggles e-commerce sites face with duplicate pages. They also highlight future hreflang updates, explain common x-default mistakes, and offer practical tips for keeping SEO strategies on track.
Key Takeaways
- Google faces challenges with localization and duplicate content.
- Understanding clusters of pages is crucial for SEO.
- Google Crawling and Rendering teams play a significant role in search results.
- Error pages can lead to black holes in indexing.
- Webmasters must ensure all signals point in the right direction.
- The localization iceberg represents hidden complexities in SEO.
- E-commerce sites must manage duplicate pages carefully.
- Correct hreflang implementation is vital for international SEO.
- Future enhancements to hreflang could improve its effectiveness.
- X-default usage is often misunderstood and misapplied.
Catch the Full Episode
YouTube: https://youtu.be/uB3Lnt2NgbI?si=EO3eSU7gx3zscSKp
Spotify: https://open.spotify.com/episode/7q002Cjk7ebbU4y0gTM5Tu?si=iCu3cSCMRkGRQN8X2j3CeA
Transcript
Bill Scully (00:00)
Welcome to episode 20 of the International Web Effectiveness Podcast. Bill Hunt and I, Bill Scully, will be discussing the strategies and techniques necessary to improve your web presence and manage global websites. In our last episode, we explored why your perfect hreflang may not be working. We covered some valuable insights and interesting discussions, and I even learned a few things myself.
In this episode, we’ll dive into Google’s recent Search Off the Record podcast on handling duplicates, which I found really interesting. One of the key takeaways for me was how Google faces significant challenges with localization and near-duplicates. They also discussed issues like “black holes” and error pages, which can wreak havoc on your site. Bill, you had a chance to listen to this too—actually, you saw it before I did. What are your thoughts?
Bill Hunt (01:05)
I thought it was interesting. I really like when they bring in specialists from different areas because they all bring unique insights. The first four minutes were focused on explaining the process. John Mueller gave a quick definition, saying, “Think of it like this: We use clusters, which are groups of pages the system believes are similar or the same.” These clusters are then handed off to a canonicalization process, which is handled by the DUPES team. Alan Scott, who’s from the DUPES team, walked us through the process.
During a crawl, Google detects pages that are identical or nearly identical and groups them into a cluster. Then, the DUPES team steps in to determine which version is the best. It’s like a shelf full of college textbooks with 15 editions and different covers. The team has to decide which version to pull off the shelf. Alan mentioned they use about 40 variables to make this decision. John joked that SEOs might start a spreadsheet to track these variables. While he didn’t list all of them, Search Engine Land did a great job breaking down the ones that have been discussed, such as rel canonical, redirects, site maps, and link authority.
The DUPES team applies weights to these variables to make a final decision. This is particularly important in international contexts. As you mentioned earlier, Google may come across 25 nearly identical English-language pages—perhaps with minor differences like currency or country names in title tags—but the content is essentially the same. Which one should they prioritize? This is where waiting logic comes in, and I’ve been wanting to write an article on this for weeks now.
Once the DUPES team applies its logic, they can then move on to applying hreflang. Alan pointed out that hreflang sits on top of canonicalization or deduplication. He introduced yet another team, the Serving team, which works with hreflang signals to serve the correct version of a page. Alan suggested we try to get someone from the Serving team to talk about how they take data from the database and present it to the interface. I found that really interesting.
Bill Scully (05:26)
And to complicate matters further, they also discussed the Rendering team, which handles issues related to JavaScript. All these teams—the DUPES team, Crawl team, Serving team, and Rendering team—show just how complex getting search results right can be.
Bill Hunt (05:37)
Exactly. They referenced a previous episode with the Rendering team, discussing how pages with delayed rendering are held in the cluster until they can be processed. It’s a chain of systems that need to work together. Honestly, I’ve been thinking that Google must be just as complicated as the rest of us in trying to get things right.
There was another interesting point Alan raised about the crawl team. He’s trying to get them to integrate the canonical tag into their part of the process. This is a significant challenge for the DUPES team when they receive conflicting signals, like a 301 redirect pointing to a page with a canonical tag. In cases like these, Google ignores both the redirect and the canonical tag, relying instead on one of the 40 other signals, which may be less significant—like sitemaps or PageRank.
Bill Scully (07:55)
Yeah, I think Alan mentioned these were weaker signals.
Bill Hunt (08:06)
Exactly. And he also brought up issues like HTTP vs. HTTPS, incorrect or missing URLs in sitemaps, and other sitemap issues. This highlights the importance of ensuring that all signals point in the right direction.
Bill Scully (08:13)
Right.
Bill Hunt (08:25)
Right. He said it’s explicitly the webmaster’s job to ensure all signals point in the right direction and to fix any errors. I thought that was great because we’ve been telling people this. In fact, this is our 20th episode, and I think we’ve mentioned something similar in every episode—keeping your house clean by fixing these errors. They have real implications when you don’t address them. He emphasized that it’s on you to ensure all signals point in the right direction.
Another topic they mentioned was the “localization iceberg.” But before I get into that, let me backtrack a bit. The most frequently referenced topic, at least from my abbreviated notes, was the black hole. They mentioned the “marauding black hole” about nine times, or more specifically, its impact on error pages.
About a third of the way through, they discussed error scenarios. A significant problem arises with error pages like 404s. Sometimes, a custom 404 page may display “Page Not Found,” but the server might return a 200 status. If you have a thousand pages like this, all returning a 200 for error pages, Google treats them as one cluster, which can severely impact your site’s performance.
They also discussed how pages caught in these black holes are unlikely to escape. He said, “We’ll put these pages into a black hole, and they’re never coming back out,” due to crawl issues. Other potential duplicates might still recover, but not pages in these black holes. So, you’re hurting your site by generating these negative signals. He made it clear that this happens when error pages are involved.
An interesting point was made about e-commerce sites. Many of them, when a product is out of stock, either redirect the page to a category page or an alternative product, or they display a page saying the item is temporarily out of stock and ask users to submit their email for notifications. This results in a 200 status for a page that’s effectively empty. When you have, say, 400 pages with identical content, Google can treat them as duplicates because they have the same checksum value. If Google identifies them as duplicates, these pages will be stuck in a perpetual black hole.
It’s crucial to understand this issue and read the article or listen to the podcast to ensure I’ve captured it correctly. This is a serious problem for e-commerce sites. I’m curious if you have any thoughts on this, as it’s a nuanced error we encounter quite often.
Bill Scully (12:46)
Sorry about that. Can you hear me now?
Bill Hunt (12:48)
Yep.
Bill Scully (12:49)
Yeah, it was pretty scary when they started discussing that black hole phenomenon, especially with pages that have the same checksum, like out-of-stock pages. These pages could disappear from search results and be very hard to bring back. He also mentioned issues like JavaScript not working correctly, where the pages appear identical on the first crawl. If JavaScript can’t render properly, Google clusters these pages together as duplicates, which can be even more alarming.
Bill Hunt (13:30)
Yeah. I saw someone else mention a related problem with rendering this week. They detected an issue where pages are served as placeholders until the content is fully rendered. It’s another sign that Google needs to be able to render the page properly, without relying on complex JavaScript, especially on e-commerce sites.
Bill Scully (13:59)
Mm-hmm.
Bill Hunt (14:00)
It’s especially interesting when the word “localization” came up. Martin described it as just the tip of the iceberg, with the real complexity lying beneath the surface. What’s under the surface involves multiple teams working across the entire stack, from indexing to clustering, deduplication, and serving. This complexity is why it’s so difficult for many companies.
Bill Scully (14:39)
Yeah, the iceberg analogy really resonated. The surface issue, which everyone focuses on, is just a fraction of the true complexity.
Bill Hunt (14:42)
That was the second most referenced topic, with seven mentions of the “localization iceberg.” It’s fascinating because, as Martin pointed out, the visible part above the surface is nothing compared to the complexity beneath. He noted that the true challenge involves managing multiple teams across various processes like indexing, clustering, and deduplication.
He also mentioned aspirational hreflang issues, where pages that haven’t been fully localized yet are flagged as localized. For instance, pages might be marked as Greek, but they’re actually still in English. Ellen’s team doesn’t see that as an obvious duplicate signal, but it’s still a concern. These pages get caught up in duplicate detection systems, and it’s likely a system processing error, possibly related to hreflang issues.
Martin clarified that hreflang is handled by a separate system, which was a crucial point. It’s important to get everything right because if it’s processed by a more precise system, mistakes can result in pages being flagged as duplicates. This is why it’s essential to make sure the full cluster of pages is correct and processed properly.
Bill Scully (16:41)
Yeah.
Bill Hunt (16:47)
His team has to break it down, find the needle in the haystack within the cluster. Then, the rendering engine does its job, and hreflang is applied at this point, which is handled by the serving team. This starts to tie the pieces together. If you think about our previous episode, we were talking about these chains. So, if the other signals have been applied to a page—let’s say all things are considered, and all 40 elements were reviewed—but you still have these duplicate language pages, especially the same-language pages, stuck in the cluster, it’s going to take the hreflang processing to potentially remove them.
Again, this ties back to what Martin mentioned a couple of times—this is why the canonical version still shows in Search Console. It may not show all the others and might still be flagged as a duplicate canonical until the hreflang is fully processed, effectively removing it from duplicate status. That may clarify the question people ask: why is this flagged as a duplicate when I’m using hreflang? It’s because it hasn’t been processed yet.
I thought it was interesting that they discussed this, and he specifically pointed out that sitting on top of clustering in the duplicate system is the hreflang system. So, it’s step three in that process. He mentioned that if they understand the annotations, they’ll substitute them during the serving process. That led to the next point.
Bill Scully (18:32)
So…
Bill Hunt (18:44)
I’ve seen where a page has been marked as a duplicate, yet it’s still being served.
Bill Hunt (18:47)
That’s likely because hreflang has been applied, but the system hasn’t been fully integrated into Search Console yet. As we discussed in “Why Your Perfect Hreflang Isn’t Working,” pages might not have been indexed yet. If pages are still considered duplicates, it could be because the hreflang was introduced late or due to errors in processing.
Bill Scully (18:45)
Yeah.
Bill Hunt (19:12)
It’s taking more time. For example, one site I reviewed, after deploying their hreflang correctly, it’s taken 49 days, but they’re finally starting to see pages indexed. We’re running tests to check if they’re showing, ranking, and delivering traffic. However, Google is still tagging them as duplicate canonical because the system hasn’t been updated. There’s a lag in hreflang processing. This highlights why you need to get your hreflang right during a migration or a new market launch.
Bill Hunt (19:50)
An even more critical reason to get your hreflang act together is a reference Alan made about a project—he didn’t provide the code name, and it might not be out by the end of the year. He cryptically said they want to serve more hreflang variants. Gary has been working on this, asking for input. In the show notes, we’ll link to the article we contributed, where we gave around 20 potential variations of hreflang that could be used, some more complicated than others.
Bill Hunt (20:43)
What Alan said was that they need to put mechanisms in place to determine how much they can trust the hreflang on a given site.
Bill Scully (20:57)
Yeah.
Bill Hunt (21:00)
So, they’re doing crawl verification to check if the site is serving its hreflang mapping correctly. If so, they’ll try to apply it more without re-verifying every time. What I took from this is that you’ve got to get it right and be trusted to implement it correctly. Google needs to trust your site and its hreflang implementation. If they can, they’ll apply it more often, without constantly re-checking.
Bill Scully (22:10)
Yeah, go ahead.
Bill Hunt (22:10)
Two other things. First, there was a discussion about the x-default tag. His explanation could be confusing. He tried to equate it with rel canonical, which is possible, but the issue is that not everyone uses it on every page. The misleading part is when he said it should be in the same cluster without a specific location preference, which isn’t always true. I’ve asked for clarification on this. He mentioned that most sites get the x-default correct, but I actually think this is one of the top five mistakes people make. If someone tells me they have a decentralized multinational organization where each market or region manages its own system, or if multiple agencies are involved, I can almost guarantee their x-default will be deployed incorrectly.
Bill Hunt (23:20)
This is especially true if they’re using separate CMS systems that don’t communicate with each other. These tools don’t check across CMS builds, so if the hreflang or XML sitemaps are built by different systems, the x-default will be incorrect. You’ll often see it when the agency is only concerned with a few markets, which creates an issue. So, make sure your x-default is consistent across markets and regions. It’s crucial because it serves as a fallback if no other version is available. You can’t pick and choose, and I wish Google would clarify that. They should spoon-feed us errors, as we discussed earlier in the podcast. Google expects your setup to be correct.
Bill Scully (24:30)
Yeah, he said it’s a signal to serve when there’s no clear match for a user’s locale or language preference. He mentioned it’s a valuable tool, but it isn’t widely used or properly implemented. The people who use it correctly typically know how to, but, in your experience, it’s not clear to most people how to implement it.
Bill Hunt (25:02)
Right. I’ve sent them what I thought was a clear explanation, and we have articles that clarify it. Last two points: One, I love how Alan described having a conversation with Google. We do this all the time. I recently had a case where someone suggested a strategy to Google, and it made no sense because it contradicted other signals they had. This illustrates why you should “talk” through the situation to Google and identify any errors in the recommendations.
Bill Hunt (25:52)
Alan had a few similar conversations with Google. Essentially, it’s about explaining your situation out loud and recognizing when there’s a conflict in your approach. It’s helpful to address issues before they become bigger problems.
Bill Hunt (26:12)
Finally, related but separate: Gary mentioned in the internationalization session that the canonical and hreflang systems are separate, which I found fascinating. Alan has been trying to get the crawling team to integrate canonical handling into the crawl, but that hasn’t happened yet. Meanwhile, during crawling, hreflang creates a dependency, meaning it tells Google to crawl certain pages to verify their alternatives. It’s just interesting how these systems interact.
Bill Scully (27:37)
Let me ask a question. How could they possibly think like a search engine if they’re doing it with tags?
Bill Hunt (27:48)
What do you mean?
Bill Scully (27:49)
Using hreflang tags? They would have to crawl everything before seeing the tags. It really makes a case for using a hreflang sitemap.
Bill Hunt (27:58)
Exactly.
Bill Hunt (27:49)
During crawl, right? So when they crawl, they’re in…
Bill Scully:
Right.
Bill Hunt:
Exactly. Then you get, so they’re, you know, as they’re crawling, they’re finding these URLs. And if they find it within a hreflang tag, as opposed to a regular tag, you know, they should trigger that differently than if it wasn’t. What I thought was interesting is that there is no implicit directive to crawl the canonical destination. And he specifically said if it hasn’t been crawled before.
So, this is perfect again, back to our point on new migrations. But at the end of the day, it’s just saying these are separate systems that may not be completely interconnected in a perfect scenario. The more we can ensure that we’re adhering to the optimal standards for each of these systems and phases, the more likely things are to work as expected.
I thought that was a really interesting takeaway.
Bill Scully (28:58):
Yeah, great conversation. The podcast was really interesting. The concept of clusters of pages, and I guess they could even be on any domain, not just .com.
I can’t imagine the scale that Google is working with. Think of all the articles that are published on other websites. You used to be able to pay and get your article on 60 different websites, all the same exact article.
Bill Hunt (29:34):
And they negated that.
Bill Scully (29:38):
Yeah, right, right.
Bill Hunt (30:02):
It sounds like they cluster them, they find them. We’ve worked with people over the years. I remember one site that was writing brilliant articles but had a clunky internal workflow. They were also licensed by, I think it was Yahoo. So, Yahoo would get the feed, publish the articles, and then post them instantly.
Bill Scully (29:38):
Yeah, right, right.
Bill Hunt (30:18):
The problem was there was a lag of about four days on their side. Yahoo was getting the first mover advantage, ranking for the article concepts instead of the people who actually wrote them.
Bill Scully (30:18):
Which just goes to show that maybe indexation time is one of the factors they use for determining which page to show.
Bill Hunt (30:31):
If you go back to one of their podcasts, we’ve got to find it. See, you always get me to remember something that we need to extrapolate on. The reason Google says you should use a date in your sitemap is because it helps them understand who had it first.
Bill Hunt (30:52):
They like that date. When it is produced, they can get it from the server. But John Mueller, and I think someone else at Google, said it’s to establish the relevance of the content.
Bill Scully (31:06):
I don’t remember that.
Bill Hunt (31:23):
John said they could potentially use that date as a tiebreaker.
Bill Scully (31:28):
Okay.
Bill Hunt (31:30):
That was his only rationale for using the date.
Bill Scully (31:34):
Not to put a black hat on, but could you force an earlier date in your sitemap?
Bill Hunt (31:42):
Absolutely, but it’ll still bind to the date they found it. You can say you wrote it nine years ago, but it still matters when they fetched it, when they first saw it. Especially with crawl hints from services like Cloudflare. They know who had it first.
Bill Scully (31:47):
Well, that’s true. Yeah, they probably locked that in.
Bill Hunt (32:12):
Backdating is the same thing as the people who write scripts to change the date on all their articles to the current month so it seems fresh. They know that. They can check the checksum. This is why we used to game XML sitemaps by rewriting them, adding or removing a URL to change the checksum, signaling that something had been altered.
Bill Scully (32:16):
To date it every day so it’s fresh.
Bill Hunt (32:36):
But anyway, I think this was good. This is why I like listening to these. I always pick out eight or nine sound bites that help piece together the puzzle. Nothing material is going to change your business other than making sure you’re fixing your errors and aligning your signals. But I’m looking forward to what improvements and enhancements they make to Hreflang.
Bill Scully (33:06):
Yeah, cool. Well, I want to thank you and Alan Scott from the duplicate team. This was a great conversation. Probably a good place to break. What do you think?
Bill Hunt (33:23):
Sounds good.
Bill Scully (33:25):
Any final thoughts?
Bill Hunt (33:28):
Nope. Check out the connected articles and please ask us questions. We’ve had a couple of these now, prompted by people posting something on Facebook or sending us a note. Please keep doing that, and check out some of our courses. Let us know if you like what we’re talking about, and send us ideas for topics you want us to cover.
Bill Scully (33:54):
Yeah, that’s great. We definitely want to hear from you. Your feedback helps motivate us to keep making these podcasts. Follow International Webmastery on LinkedIn, Facebook, and X. If you find value in our conversations and enjoy the show, it really helps get the show in front of others if you take a moment to rate and review the podcast on your favorite platform. Until next time, take care.
Bill Hunt (34:18):
We’ll see you.