Building Hreflang XML Sitemaps at Scale with AI
▼ Summary
– The author used Google Gemini to build a custom Python script to automate mapping thousands of URLs into hreflang XML sitemaps for over a dozen multilingual websites, a task that previously required specialized software or days of manual work.
– The project began by asking for an approach rather than a final script, leading to a multi-step strategy that included crawling websites and using semantic AI for fuzzy matching of translated pages.
– Iteration was crucial; the author provided specific examples of mismatched URLs to refine the script, such as adding a URL flattener for directory reorganizations and implementing “concept traps” to prevent incorrect matches.
– A key breakthrough came when the author pointed out direct translations in slugs, prompting Gemini to create a “Combined Semantic Signature” that dynamically translated phrases to bridge language gaps.
– The author recommends treating AI as a collaborator by discussing architecture first, providing concrete examples of failures, embracing an iterative loop, and using Google Colab to run Python code without local installations.
As AI tool adoption accelerates across industries, I’ve witnessed both brilliant automation that solves genuine operational headaches and pointless implementations that add complexity without real value. My philosophy is simple: only deploy AI where it delivers measurable time savings and solves tangible problems.
That philosophy recently guided me through a complex international SEO project involving over a dozen websites spanning three distinct businesses, eight regional domains, and nine languages including three English dialects, Italian, Japanese, Spanish, Thai, French, and Korean.
The task was straightforward conceptually but brutal in execution: map thousands of URLs into accurate hreflang XML sitemaps. Traditionally, this would have required expensive specialized software or painstaking spreadsheet work spanning days. Instead, I turned to Google Gemini to build a custom Python script that automated the entire process.
Here’s how the project evolved from a simple prompt into a sophisticated automation tool, and what it revealed about effectively leveraging AI for technical SEO.
Where AI delivers maximum impact
I reserve AI for practical, time-sensitive tasks that genuinely accelerate my workflow. These include:
- Generating regex patterns when I need a quick solution without researching syntax from scratch.The hreflang project fits squarely into that final category.Scaling hreflang mappingThe challenge was unambiguous: connect thousands of URLs across more than a dozen multilingual websites into coherent hreflang XML sitemaps. Rather than tackling this manually, I used Google Gemini to build a custom Python solution.Phase 1: Asking for strategy, not just codeA common mistake when using generative AI for coding is demanding a finished product before understanding the problem. Typing “Write a Python script to create an hreflang sitemap” produces generic, brittle code that breaks against real-world data.Instead, I asked for an approach. I explained the scenario: multiple regional domains, years of organic growth creating mismatched URL slugs, translated subfolders, and appended revision years.Gemini proposed a multi-step, data-driven strategy:
- Crawl the websites to collect live URLs and metadata.Phase 2: Crawling and data collectionFollowing the strategy, I used Screaming Frog to spider all regional websites, generating a unified CSV containing live URLs, status codes, title tags, and H1s. This tool worked perfectly for the task.A critical reminder: your AI output is only as good as your crawl data. The old adage “garbage in, garbage out” applies here. An AI script will fail to map an obvious exact match if the target URL returns a 404 or 301 redirect. Filter your CSV to include only indexable content before feeding it to the script.Phase 3: The Google Colab sandboxGoogle Colab offers a free, cloud-based Jupyter notebook environment where you can write and execute Python code without local installations or environment variables. Accessible through Google Drive, the free version provided sufficient capacity for this project.I uploaded the CSV to Colab, and Gemini provided the initial Python script. The script used a domain-mapping routine to assign language codes, clean URLs, and generate an XML tree. The initial output was far from perfect.Phase 4: The iteration where real work happensExpecting AI to deliver a flawless, edge-case-proof script on the first attempt is unrealistic. Think of AI tools as interns: you must check their work. That comparison holds true.The real value of AI emerges through iteration. As we ran the script, we encountered several unmatched URLs, leaving pages orphaned rather than grouped with their international counterparts.Here’s how I iteratively trained the AI to handle the nuances of human-managed websites.The directory flattening problemThe U.S. site had recently reorganized its blog into topical folders, while the Mexican and Italian sites hadn’t yet been reorganized.I prompted Gemini with these specific mismatched examples. It responded by adding a URL flattener function to the script, stripping the topical folders behind the scenes so translated slugs could align cleanly.The aggressive semantic trapTo prevent the AI from mixing up different topics, we implemented concept traps. Initially, they were too strict. A UK article about the manufacturing sector wouldn’t match an Italian article because the U.S. title was slightly more generic.I instructed Gemini to loosen the traps for generic industries while keeping them strictly enforced for critical acronyms like “SEO” versus “SEM.” This gave the AI breathing room to match creative translations.The translated slug epiphanyThe biggest breakthrough came while auditing the Mexican blog orphans. For example, the Spanish URL /detras-de-escenas-historias is a direct translation of the English /behind-the-scenes-stories. I pointed this out to Gemini.Instead of forcing me to hard-code hundreds of manual matches, Gemini updated the script to create a “Combined Semantic Signature.” It dynamically translated core operational phrases in the slugs, effectively bridging the language gap for the semantic matching model and connecting dozens of orphaned pages almost instantly.Lessons from building an AI-assisted SEO toolThe project reinforced a simple lesson: AI works best when treated as a collaborator rather than a shortcut.Be the strategist, let AI be the coder. Don’t just demand a final product. Discuss architecture, edge cases, and logic first. Treat AI like a junior developer that needs clear architectural direction.Provide concrete examples. When a script fails, don’t just say “It’s broken.” For this project, I provided either exact URLs that failed and the URLs they should have matched with, or groups of URLs with mismatches. AI needs concrete patterns to fix its logic.Embrace the iterative loop. Expect to run the code, identify anomalies, and feed them back into the prompt. Each iteration makes the tool significantly smarter.Leverage Google Colab. You don’t need to be a Python expert to use Python for SEO. Colab bridges the technical gap, allowing you to run complex data science libraries directly in your browser.By the project’s end, we had a robust, highly customized Python script that could process a massive CSV and generate a cross-referenced hreflang XML sitemap in minutes.AI isn’t going to replace technical SEOs anytime soon. However, SEOs who know how to collaborate with AI to build custom, scalable, and useful tools will have a significant advantage.
