On July 25th, Google posted a new Search Off the Record Podcast with Lizzi, Martin, and Gary focused on internationalization and hreflang.
This is a must-listen-to episode if you have a multilingual website or are involved in international SEO because it offers some interesting insights into how Google processes local language sites, hreflang, and searcher intent in the local vs. global context.
My top takeaways:
- My favorite comment was from Gary, who stated, ” The language is absolutely not a tell for what country you are targeting.”
- Gary indicated that a crawling dependency is “triggered” when they encounter hreflang to verify the alternates, resulting in faster indexing of the new language pages. That means hreflang is critical to get right during site migrations.
- Excellent and straightforward discussion of why hreflang gets complicated and is error-prone due to the complexity of varied elements like domains, URL structures, and localization.
- The LDCP algorithm, “Language Demotion Country Promotion,” is mentioned. LDCP is used to boost a webpage that targets a specific country, typically via a ccTLD, but it can also demote a page if it does not match the language of the query.
- Following the LDCP, there was a discussion on the ranking contribution value of ccTLDs currently and in the future. This was because many ccTLDs were converted to generic TLDs for branding (AI, CO, IO, TV), making it difficult to equate a ccTLD representing a specific country.
- There was an interesting reference to AGL (Attribute Grammar Language), which Google uses to understand the context and identify the user’s intent within a language nuance. This AGL algorithm intersects with personalization, impacting which pages are presented in the SERPS.
- Gary discussed the potential for using automation, favoring it because it can be more accurate than human-initiated annotations.
There are a few comments in the podcast that can be confusing without the actual context and nuance of the conversation, resulting in a few articles that misstate or take a comment out of context and I will try to clarify them.
Favorite Statement:
[00:07:45:04] – Gary
The language is absolutely not a tell for what country you are targeting.
Thank you! A common reason I hear for not using hreflang is that Google can understand language. Yes, they can detect language, and they do it very well. The problem is they don’t know what market it is targeting. I have been screaming this as loud as I can for years. People still do not understand that correct hreflang elements minimize the potential that near identical pages will be considered duplicates and not added to the index.
Martin addressed this as a specific challenge for e-commerce sites where product pages are nearly identical across language versions, with maybe a difference in price, currency, and tax rates. Gary confirmed that this is what hreflang does; it helps prevent de-duplication issues.
Second Favorite Statement:
Martin starts the podcast with the standard conversation starter related to the confusion around why hreflang is so complex and why everyone seems to hate it. Gary explains that the problem is the lack of coordination and uniformity that many companies have to work with. Essentially, every deviation from consistency adds a new level of complexity to hreflang implementations.
[00:01:28:19] – Martin …But everyone keeps telling me it’s [hreflang] very complicated, and it’s so easy to make a mistake, and it’s so complex. I’m not seeing this complexity, so I’m wondering, where does that come from?
[00:01:54:24] – Gary I think it comes when your site is big or you are managing multiple properties, and the multiple properties each have their own URL structures because, in a utopic view, you could have like example.com/dch/ and then, for English, you would just have en and us and it would work. It would be so simple. But then let’s say that in another jurisdiction, you need to have a separate domain, like exemple.es, for example. I don’t know what you are selling, but maybe you need a different domain name. Then, maybe on a different domain name, you can’t follow that pattern anymore, so you absolutely need to be able to annotate everything separately, like every language and country variation. As soon as you have to vary things, that’s where you start introducing errors.
Shameless Plug: I discuss this and many other challenges in detail in our free course on Demystifying Hreflang.
Automatic Detection vs. Hreflang Annotations
The conversation about Google’s ability to automate hreflang, eliminating the need for annotations, followed the discussion that HTML Lang attributes were not reliable and that Google could automatically learn that two sets of pages had the same content as far back as ten years ago and should be able to to a better job than human edited tags, especially with the wonders of machine learning and AI.
[00:14:17:10] – Gary – Like, even when I worked on Hreflang, we already had something that was automatically learning that to set two pages, different versions of the same content, we could already do that. This was, what, almost ten years ago. Wow. I’m old now with the advancements that we have with AI and all that weirdo stuff.
I agree with Gary that Google may be able to detect it, but can you ensure accuracy? The key challenge to automation is the lack of signals on many sites that can help identify the market. I have used Nike as an example of this problem. They have 78 sites, 38 in English and 22 with the same $ currency symbol, so how would automation make the distinction for each market? There is nothing on these pages that indicates what market they represent, and without hreflang, most would not even be indexed due to the perception of duplication.
There was some brief banter about Regex and detecting URL patterns. This option is only viable for those using a language and market attributes path structure. If you take an example like Nike.com, they only use a country folder and not a country/language folder structure. What if a site used /fr/ only? Is that site targeting France or, more broadly, the French language? What about a /uk/ is that United Kingdom or Ukranian language? For more structured sites and those with distinct market information in the pages, I agree it should be relatively easy to detect them. As someone working on hreflang since the beginning, I have noticed that this rarely happens.
What if the automation believes Canadian English pages are Australian? How can we adjust it? If automation is adopted, there must be a way to submit overrides when it makes mistakes.
Potential For Confusion
This segment of the podcast concerns me because some may conclude that you can have incorrect hreflang elements without any negative impact. If you take it in the proper context, you will understand that is not what is being said. Even if you think it is okay, why would any reputable company not want it to be correct?
[00:10:19:28] – Martin – Okay. I have another question. This actually does come back to Hreflang again. Say, I have an Hreflang annotation on the English version of a page, and it says, “This is the German version of this page,” but then, for some reason I made a mistake, and the actual content on the page isn’t German. Is that going to cause problems or not?
[00:10:43:21] – Gary – Not from Hreflang perspective as far as I remember. I worked on the parsing implementation plus the promotion implementation of Hreflang. Back then, it didn’t cause problems, but that was like 2016 or something.
[00:11:03:01] – Gary Since then, we changed so many things that I would have to check whether it causes problems. I think it wouldn’t because, like when I spelled out LDCP, then I said the language demotion country promotion. So, for example, if someone is searching in German and your page is in English, then you would like get a demotion in the search.
I have read a few recap articles using this statement: “if you get hreflang wrong, you won’t get penalized.” That is NOT what this says. Gary states two points:
- From a hreflang standpoint, it may not be a problem, but that was nearly 10 years ago, and he would need to confirm.
- He then says that while hreflang may not be a problem, you would get a demotion under the LDCP algorithm.
First, this does not mean you can incorrectly tag the language and country of your pages using hreflang. I heard this once before from a content creation agency that was trying to leverage what I call “Aspirational Hrelang,” where by accident or on purpose, you use the incorrect language or country designation.
If it is incorrectly tagged, Gary clarifies that the content would not be in the local language and would be subject to an LDCP demotion. The LDCP algorithm, “Language Demotion Country Promotion,” is applied to boost a webpage that has been determined to target a specific country, typically via a ccTLD, but it can also demote a page if it does not match the language of the query. Martin gives an example of a coding site he created on a .de Germany domain but the content is completely in English. The site would receive a slight boost for Germany, but that would be counteracted by the fact that it is in English if a related query were done in English. I have seen firsthand that Google does, in fact, either ignore, demote, and, in some cases, not index pages without a correct hreflang annotation via an LDCP demotion.
Without hreflang there would not be any protection from the duplicate cluster process dropping the pages from the index. In my Aspirational Hreflang article, I show examples of where the hreflang language was set incorrectly, and nearly all incorrectly tagged pages had been removed from the index. Once the language was corrected to English in the hreflang annotation, the pages started to get reindexed. This is due to the hreflang alternate page desgination being applied after Google has already used any duplication flags on the pages.
Introducing Attribute Grammar Language
All three hosts discussed the challenge of understanding the search query and presenting the best results. They used the query “How to spell banana.” They agreed that the SERPs should return English results since the query was written in English. But what if you did that query in Germany, and what if your previous search history indicates that you understand German? Does that change the results presented? Would the result be presented in German?
[00:12:26:01] – Gary – So that can also happen. But, if you break away from personalization, then generally we will try to push down languages that are not in the current AGL language, AGL language being the interface language that you are searching in.
AGL Language refers to “Attribute Grammar Language.” AGL is logic that provides a structured way to parse and understand user queries, intent, and multilingual content. By applying AGL, Google ensures that users receive the most relevant information, even if some of it originates in another language, enhancing the overall search experience. The implications of this are significant. With the adoption of MUM functionality and user personalization, it can create a richer experience. As I always remind people, if there is a metric for it in GSC, it should not be too far off of a wider adoption.
Bill Scully and I discussed this recently on episode three of our Podcast, explaining that Google had just added 8 additional languages to “Translated Results,” which automatically translated the title link and meta description into the user’s local language. This makes a website published in one language available to a searcher in another language. If the searcher clicks on the link of a translated result, the web page itself will also be automatically translated.
Why the HTML Lang Attribute is not a viable signal
Martin asked an interesting question about why Google is not using the HTML Lang attribute. I covered this a few months ago in my post on alternatives to Hreflang. It is not used because it is often incorrect. Gary gives some historical context and validation of my statement.
[00:13:01:00] – Gary – It is, and that’s because, if you think back, earlier days, I remember that I think Joomla just came with the Lang attribute set to English, and there was no way to override it, even if you change the template to German or whatever. It was just like basically yelling that, “Hey, I’m English, I’m English, I’m English, I’m English!” And then you looked at the page, and it was 100% German. So it was like you can’t trust that.
Potentially Confusing Stats
During the discussion, Gary stated that unlike the HTML Lang attribute, which is required, hreflang is not and that most sites don’t need it. Lizzi stated she checked the Web Almanac before the session and read that 9% of sites had Hreflang in 2022. Gary felt that number was very high and would be closer to 1 or 2 percent.
I flagged this statistic and response as potentially problematic because people may make incorrect assumptions based on their perspective, as evident by the two statements below, which interpret that percentage and Gary’s reaction to it very differently.
“This relatively low adoption rate might be a factor in Google’s consideration of alternative methods for detecting language and regional targeting.”
Source Search Engine Journal
The higher-than-anticipated adoption rate underscores the importance of hreflang in the current multilingual web landscape.
PPC.Land
Two clarifications need to be made:
- The stats were only for those websites that used HTML tags or HTTP header implementations (not hreflang XML sitemaps). Hreflang Builder found that 31% of the most localized websites had used XML sitemaps
- Per Web Almanac, the 9.6% of websites using hreflang was from a sample of 5,428,235 desktop websites, meaning 521,110 websites had hreflang on the home page, which is why Gary thought it was a lot of websites.
- The total number of active websites in 2022 was 198 million, so the sample is only 2.7% so if the ratio holds only 0.26% of all websites may actually have hreflang implemented.
- In a 2019 Hreflang Builder study of the most localized websites, found that only 60% were using hreflang in any form at the time. This number is 95% currently.
Why do Hreflang Tags Require Self-Referencing?
The three hosts were asking about the need for self-referencing, as Google should be able to understand the page it was on, and therefore, it was unnecessary. This is the deductive logic many SEOs use to justify not adding it. As a person in the discussions back in the day, there were a few reasons it was included.
- The one that still lingers is that a site owner could simply copy and paste the entire cluster into all of their pages. By forcing self-referencing, it would ensure that the complete cluster is there.
- It was a way to ensure the canonicalization of the URL in the cluster. Not many sites were using the canonical or using it correctly, and since hreflang was typically an independent action, it helped ensure that the canonical version of a page was mapped to the canonical of other versions.
- Many websites used relative URLs and/or ccTLDs, and the hreflang was processed in a secondary routine that the full cluster needed to be captured to enable that process to validate fully.
Similar Pages are Alternates and not Canonicals
During the discussion about a GSC “bug,” Google may show another market page as not indexed due to a different canonical being selected with hreflang in place. Gary does an excellent job of explaining this non-issue, stating that GSC only reports on canonicals and reiterating that similar pages referenced in hreflang are NOT canonicals of that page but alternates.
[00:27:09:28] – Gary – I think it’s also important to mention that Search Console only reports canonicals. Basically, the vast majority of the Hreflang clusters, when we are talking about similar languages, they are not canonical. Basically, you are blind to what’s happening on the non-canonical pages of a cluster. But that was a tough product decision because we don’t store information about the alternate names, we just put them in the dupe cluster and just say that these are localized alternate names that we can show in search results when the user query plus settings imply it. For example, we could swap out the URL for Nile.com to Nile.ch if the language on it was similar and the user was from Switzerland, for example.
If you get those “Duplicate, Google chose another canonical” error messages, it may still be an error, and Google may not detect your hreflang. I just worked with a brand that implemented hreflang in the page, but due to faulty JavaScript, the hreflang was not being indexed and applied, resulting in multiple same-language markets dropping out. Their SEO consultant told them this was a “normal” action by Google. The company tested with XML sitemaps, and the canonical errors disappeared and their pages were reindexed.
Which hreflang method is better?
I will admit I have always been biased toward hreflang XML sitemaps for the very reason Gary gives in the podcast,
- It is not always feasible to put it into the HTML
- If you have “billions” of pages that block, with a lot of entries, it impacts server overhead, making XML sitemaps more efficient.
It was interesting the conversion of “why an implementation method might be better than another” had to do with ingesting the URLs, and this stood out as a critical reason why hreflang must be considered both for new product pages and during site rebuilds.
[00:20:58:05] – Gary – Like, if you publish something, like you want to get that weird non-GMO scented candle online as fast as possible and sell it on multiple markets, then you probably want that annotation to be seen as fast as possible by Googlebot because there is a dependency crawl triggering when we discover Hreflang, like we want to verify that.
If you have not done so, you may want to ensure that hreflang elements are part of your site migration acceptance testing to ensure your hreflang is correct when migrating.
