The Web Grows a Second Layer, Approaching a Third

The past few weeks have been loud in the world of web development and search. Google launched the Open Knowledge Format (OKF) , and shortly after, its developers introduced the Agentic Resource Discovery (ARD) specification. Meanwhile, your LinkedIn feed is probably split between those hailing markdown as the future of the web and those insisting you should ignore the noise entirely.

As usual, the reality is more nuanced than either extreme.

What we’re witnessing is the emergence of a parallel machine-readable infrastructure for the web. This includes MCP/WebMCP, OKF, ARD, LLMs.txt, and more. For SEOs, the key is understanding what each layer actually does, rather than lumping everything under “AI SEO” or chasing a silver bullet. That understanding will guide smarter decisions about where to invest time.

The Layer Cake: Six Distinct Pieces

At least six distinct concepts are being discussed under the “making your site AI-ready” umbrella. They operate at different layers and solve different problems:

1. Crawlable HTML Pages: This remains the absolute foundation. Nothing has changed. Everything else builds on top of this.For ecommerce, there’s another layer worth naming separately: the product feed. It may well be the future of retail discovery.Each layer does something different. And the list keeps growing. It’s ballooning.

What OKF Actually Is (And Isn’t)

Google published the OKF spec quietly, attached to a rebrand of Dataplex into Knowledge Catalog. The format is disarmingly simple: a directory of markdown files, each with a small YAML header declaring a type, title, description, resource, and tags. The files link to each other like any markdown document. That’s it.

As Google’s own blog puts it, OKF is “just markdown, just files, just YAML frontmatter.”

SEO Suganthan Mohanadasan offers a clear breakdown: OKF is one floor in a stack that includes sitemap.xml (which URLs exist), LLMs.txt (which pages you most want read), and OKF (the library itself). They stack; they don’t compete.

The confusion comes not from what OKF is, but what it does and where it sits in the agentic and search ecosystem.

OKF is not a retrieval system. It doesn’t replace crawling. I don’t see a future where AI systems stop ingesting massive amounts of HTML, or where search and RAG aren’t complex, multi-step pipelines using both self-reported and “unbased” signals.

Any self-reported system can and will be gamed. Thinking you can just throw markdown files on your site and become the preferred choice for retrieval is far-fetched. OKF is a higher-signal source among many. It may reduce parsing cost and improve signal quality, but it doesn’t replace existing pipelines.

Let’s be honest: OKF was built for data teams, not marketing sites. It arrived as a way to share internal knowledge (table schemas, runbooks, metric definitions) between AI agents inside organizations. Pointing it at a public website feels like yet another repurposing.

Francois Vanderseypen makes the most precise point: a directed graph of markdown files is a web of documents, not a knowledge graph (in its purest sense). A real knowledge graph has explicit, queryable, typed relations. OKF leaves what a link implies entirely up to the producer, and an LLM still has to infer the semantics each time it reads it.

This gets to the heart of how I understand the web and SEO. OKF doesn’t change the stack. It adds one more input. It’s not a shortcut. There are no shortcuts.

The Schema.org Parallel (And Why It Matters)

Structured data followed a predictable arc: adoption, ranking boost, widespread use and gaming, platform learning, and reduced dependency as a ranking signal. FAQ schema had its moment in SERPs, then Google discontinued the rich result. The platforms learn from the signals, fold the lessons into the algorithm, and the explicit markup becomes less necessary.

OKF and LLMs.txt may follow the same path. They’re most valuable early, as clear signals in a world where AI systems are still learning to parse the web. Over time, if the formats work, the systems learn. Explicit markup becomes redundant or remains a verification layer.

In ecommerce, schema and feed alignment have become increasingly important. This is another call for co-ownership of the product feed between SEO and paid teams.

Jarno van Driel’s deep dive on product variants in Search Engine Journal illustrates this well: For years, Google Search and Google Merchant Center had conflicting structured data requirements, forcing publishers to duplicate markup. Schema.org evolves to close gaps, but it’s slow, complex, and implementation is often a mess. Structured data has never been a plug-and-play ranking lever. OKF won’t be either.

Should You Convert Your Site to Markdown?

A big fat no from me. That doesn’t mean I won’t test it and apply it carefully.

John Mueller said it on the podcast: “When it comes to things like a search engine or probably also in generic LLM system, having a website that uses normal HTML for the pages is critical. Because a search engine or crawler can just go to that page. It can recognise all of the other links that are within the website.”

The structural information in HTML (nav links, footers, header hierarchies, internal links) is how crawlers understand your site’s shape. Markdown files strip all of that out. You’d be breaking discovery to marginally improve machine readability of individual pages.

I recently saw research suggesting “Your navigation might be eating your LLM reading budget.” Interesting findings, but please don’t remove your navigation to “save some tokens.”

Jono Alderson makes this point brilliantly: “A page is not just a container for words. It’s an editorial artifact.” Hierarchy, emphasis, placement, what comes first, what’s prominent, what’s tucked in a footnote… these aren’t pretty decorations for humans. “They are signals about meaning.” When you flatten a page into markdown, you don’t just remove clutter. You remove judgment and context. The moment you publish a machine-only representation, you’ve created a second candidate version of reality.

The boring fix still works: Semantic HTML, clear structure, sensible hierarchy, content that exists when the page loads.

Mueller covers the markdown debate extensively, discussing the parallel versions problem, the dynamic rendering lessons we already learned, and why maintaining a shadow version of your site for AI doubles your maintenance burden and creates a debugging nightmare.

The one exception Mueller carves out is developer documentation: “If you have something like developer documentation, where… the user says, how do I use this API? Then if you give the LLM system a Markdown file, it’s a lot easier for it to understand.” I can see a straightforward use case there.

What ARD Is Actually Doing

The Agentic Resource Discovery specification, announced by Google on June 17, 2026, is a different beast entirely. It arrived just days after OKF (not a coincidence) and is already making waves.

The problem ARD solves is coordination. Right now, an agent must be wired to each tool, MCP server, or API it uses before it can do anything. That works for a handful of known services. It stops scaling when the number of available capabilities grows beyond what any team can pre-configure by hand.

ARD moves that discovery out of setup and into runtime. The agent finds what it needs when it needs it, rather than only knowing what it was told about in advance.

It’s built on two primitives:

• Catalogs: An `ai-catalog.json` file hosted on your domain, describing your available capabilities (MCP servers, A2A agents, OpenAPI tools). Domain ownership acts as the cryptographic foundation for identity and trust.
• Registries: Search engines for the agentic web. They crawl catalogs, index them, and return matching capabilities with the metadata needed to verify the publisher before connecting.

If OKF is about packaging knowledge for consumption, ARD is about advertising capabilities for connection. These are parallel efforts at different layers of the emerging agentic stack. Both shipped within inches of each other and have been adopted at lightning speed by major players like Hugging Face and their Discover Tool.

It’s possibly a more pragmatic bet than the formal logic layer that came before it and never reached web scale. Time will tell.

A Gap Worth Watching

Within days of both specs shipping, a contributor opened companion issues on the ARD and OKF repos pointing out something basic was missing: there’s no agreed media type for an OKF bundle. A catalog can list one but can’t actually recognize it as OKF without sniffing the contents. Publishers are already advertising bundles in production using their own interim types, which won’t agree with each other.

On the surface, this looks like a small ask: just a request for a shared label. After diving into this rabbit hole, it turns out that’s normal practice. Waiting for full agreement before anyone ships anything is exactly how a spec dies in committee. Shipping fast and patching as real adoption surfaces is an age-old strategy.

`application/json` itself wasn’t formally registered until 2006, roughly five years after JSON was already in wide, informal use. Nobody worried, because the cost of an unsettled label was low: a parser might reject something or fall back ungracefully.

But OKF is different, because what happens after the fetch is different. The artifact behind the label is a bundle an autonomous agent is meant to ingest, verify, and potentially act on, inside a discovery system built specifically for agent-to-agent and agent-to-tool connection. Get the type wrong here, or leave an agent to infer it, and the risk isn’t a parse error; it’s a system acting on something it shouldn’t have trusted, with no one checking the result first.

I wonder about the risk involved in settling this later rather than sooner. It depends on how fast it gets resolved relative to how fast adoption runs ahead of it.

What This Means If You’re an SEO

A few honest conclusions and my current thinking:

• For most marketing and content sites, not much has changed. HTML, well-structured for humans, is still the right foundation. A contact-us form and a clean site architecture will serve you better than any OKF bundle. Discovery still depends on links, authority, user signals, and indexing.

The Underlying Shift

What all of this points to is a web that’s genuinely growing a second layer (or a third head), one written for machines alongside the one written for browsers and humans.

Sitemap.xml told crawlers which URLs existed. Robots.txt told them where not to go. LLMs.txt, OKF, and ARD are similar infrastructure for agentic systems: navigation hints, content packaging, and capability discovery.

None of it is mandatory today. None of it replaces solid HTML, authoritative content, sensible structure, or the thing that actually sits underneath all of it: a brand worth finding.

But the SEOs who understand what each layer actually does, rather than treating it as a single undifferentiated “AI SEO” category, will make better bets on where to spend their time.

My money is on the second layer: a parallel infrastructure written for machines, not a replacement for what already exists. The third head scenario, where agentic systems fully diverge from the human web, would require a different set of bets than any of us are currently making.

Big thanks to Jarno van Driel, Jono Alderson, Chris Green, Suganthan Mohanadasan, Kristine Schachinger, Gianluca Fiorelli, Victor Pan, Renee Bigelow (and anyone else I’ve missed) for some brilliant discussions on this topic over the last few weeks.