Unlock Big AI Insights with Small LLM Tests
▼ Summary
– SEO professionals are actively developing strategies to influence LLM answers, using tools and debating technical concepts like RAG and vector embeddings.
– A test showed that publishing new branded content on a small website can change ChatGPT’s answers within hours, using RAG to fetch updated information.
– ChatGPT appears to rely on Google’s index rather than Bing’s, as blocking Googlebot prevented it from finding new content while allowing access enabled discovery.
– Updating or removing content from sources cited by LLMs can impact AI results, as seen when a course disappeared from answers after being removed from a listicle.
– Running small, controlled tests is recommended to understand LLM optimization effects before scaling, and changes should be documented and shared across stakeholders.
Understanding how large language models (LLMs) like ChatGPT source and display information is becoming a central focus for modern SEO professionals. While complex theories around vector embeddings and retrieval-augmented generation (RAG) provide valuable background, practical testing often delivers the clearest insights. Simple, controlled experiments can reveal how AI models interact with web content, allowing marketers to optimize effectively without needing deep technical expertise.
Consider the impact of publishing fresh branded content. In one test, a query about an individual’s travel plans initially returned a negative response from ChatGPT. However, just hours after publishing a blog post detailing the trip, the AI’s answer changed to confirm the upcoming visit. This demonstrates that new information on webpages can influence ChatGPT answers in a matter of hours, even for smaller websites. The AI used RAG to fetch the latest result, initially relying on a snippet from the homepage before later accessing the full article. This shows that LLMs actively notice new or updated content, underscoring the value of a consistent brand content strategy.
Another experiment addressed industry speculation about whether ChatGPT relies on Google’s or Bing’s index. By adding a “noindex” tag for Googlebot while allowing Bingbot to crawl a new blog post, it was possible to test which index ChatGPT used. Despite Bing indexing the page within days, ChatGPT failed to locate the article for over a week. Only after removing the “noindex” restriction and allowing Google to crawl the page did ChatGPT promptly find and display the correct URL. This indicates that, at least in this case, ChatGPT appeared to rely on Google’s index rather than Bing’s. Still, implementing tools like IndexNow for Bing remains a low-effort task that may offer additional visibility benefits.
The influence of individual sources on AI-generated answers is another area worth exploring. In a test involving a listicle of certification courses, one outdated course was removed from multiple pages on the same domain. Initially, Google’s AI Overview and ChatGPT found alternative sources to mention the course. However, within a week, the course disappeared from AI results entirely. This suggests that updating or removing content from a source cited by LLMs can directly impact AI answers, though results may vary depending on query specificity and available alternative sources.
When planning optimization efforts, starting with small-scale tests is advisable. Create a minimum viable product (MVP) approach, test one change at a time, track outcomes carefully, and scale only after observing clear effects. For example, if you aim to shift how a product is described in ChatGPT, begin by requesting updates on a few key referring pages. This helps gauge the difficulty of influencing external sources and whether such optimization requires paid partnerships or can be achieved organically.
Larger organizations can also run meaningful LLM tests by coordinating carefully across teams. One agency, for instance, updated its global footer tagline and observed ChatGPT incorporating the new messaging within 36 hours. To ensure reliable results, limit tests to one variable per page, use consistent tracking methods, and share findings company-wide. Tools that analyze brand sentiment can also highlight areas for improvement, providing test ideas that extend beyond traditional SEO.
Given the rapid evolution of AI search, no single strategy guarantees success. The most reliable approach involves combining trusted industry insights with hands-on experimentation. Document your tests, learn from both expected and unexpected outcomes, and continually refine your methods. By taking these steps, you can build a practical understanding of LLM behavior and develop a more resilient, future-ready optimization plan.
(Source: Search Engine Land)
