We set up llms.txt. Not because we believe that a small text file will suddenly open all the doors to ChatGPT, Claude, and the vast world of AI chatbots. But because we wanted to know what happens.
To be precise: what really happens.
Which bots access it? Do they follow the links? Do they also read Markdown versions? Does it generate measurable referral traffic? Or is llms.txt ultimately just the next SEO promise that sounds bigger than it is?
The initial observations are unspectacular—and that’s precisely why they’re interesting. A few bot hits. Isolated signals. No miracles. But also no reason to dismiss the topic too quickly.
Because llms.txt is probably not the big lever for AI visibility. But it could be one of those small building blocks that, when added up, determine how well a brand is understood by AI systems.
In this Article
What is llms.txt—and why is everyone suddenly talking about it?
At its core, llms.txt is a simple text file located in a website’s root directory. In other words, it’s a file accessible at an address like your-domain.com/llms.txt. Its purpose is to provide large language models and AI agents with a kind of curated guide:
- Who is this website or brand?
- What content is important?
- Which pages explain the offering particularly well?
- Where can machines find content they can process without unnecessary design, JavaScript, or layout clutter?
At first glance, this sounds unspectacular. Technically, it is. But that’s exactly what makes the file interesting.
Because llms.txt attempts to solve a problem that didn’t exist in traditional SEO for a long time: websites are no longer just read, indexed, and sorted into blue links by search engines. They are increasingly being processed, summarized, cited, integrated into response systems, or used as context by agents. For this, these systems don’t need a pretty interface, animations, button colors, or a perfectly coordinated design. They need clarity.
A good llms.txt is therefore not a second robots.txt. Nor is it a standard XML sitemap. It is more like a machine-readable content map: compact, carefully curated, and ideally written in such a way that an AI system can more quickly understand which information on a website is truly important (more on this later).
Major tech and AI companies are already experimenting with llms.txt or using the file in specific areas.
- Anthropic has published its own llms.txt and even llms-full.txt files for the Claude documentation. (An llms-full.txt file often contains all the content of a domain)
- Cloudflare uses llms.txt in its developer documentation and links to Markdown or plain-text versions of the content there.
- Google has also used llms.txt files in its Gemini Developer API documentation, among other places.
Is your brand understood by AI systems?
From llms.txt to structured data and bot monitoring: We’ll help you strategically optimize your website for ChatGPT, Claude, Perplexity, and AI Search.
Does llms.txt make ChatGPT, Claude, and others more visible?
The honest answer is: We don’t know for sure yet.
If you search online for user experiences, you’ll mostly find mixed signals.
In Reddit threads, SEO communities, and initial practical tests, you’ll read everything from “completely useless” to “nice to have,” along with individual reports where companies see slight effects or isolated bot traffic.
The problem: Hardly anyone can conclusively prove whether llms.txt was really the cause. Attribution is difficult anyway, especially with AI traffic.
What the discussions about llms.txt reveal
The most interesting insight from countless online discussions isn’t whether llms.txt works or doesn’t work. It’s that many participants are talking about different expectations.
Some expect a new ranking factor. Others think it’s about bot control. Still others see llms.txt as a technical preparation for a more agent-based approach to web usage.
This leads to a lot of confusion.
| Expectation | Fact Check |
| llms.txt brings direct rankings to ChatGPT | There is currently no reliable evidence to support this |
| llms.txt works like robots.txt | No, robots.txt controls crawling; llms.txt provides guidance |
| LLMs automatically read every llms.txt file | This cannot be reliably verified for every LLM; Google indexes some llms.txt files |
| The file is useless | That’s too short-sighted, because it can be useful in the right setup |
| One click on a plugin replaces a strategy | No, automated files are a starting point, but not the gold standard |
Many skeptics rightly argue: If large LLM systems don’t actively retrieve the file at all—or only rarely do so—then it can’t be a direct visibility lever. Others point out that classic SEO fundamentals remain more important: indexable content, clean information architecture, strong entities, clear topic coverage, and trustworthy sources.
That is correct.
But it is only half the truth.
Our observations so far regarding llms.txt
We have tested llms.txt ourselves and are monitoring traffic using tools such as vigIA and Known Agents, among others.
The results so far: no miracles, but some interesting signs.
So far, we’ve seen very little bot traffic to llms.txt. More importantly: In our observations to date, the file has not been visibly used as a starting point for deeper crawling. While the tools show the chronological sequence of individual bot visits, this does not currently indicate that a bot is reading llms.txt and then systematically following the URLs listed there.
That is an important distinction.
It is not enough to simply observe: “A bot was on the website.” The crucial question would be: Did the bot retrieve llms.txt, understand the curated content there, and subsequently derive further actions from it? Unfortunately, we have not yet seen any evidence of this.
Why the direct effects of llms.txt are difficult to measure
Even if llms.txt is used by individual systems, measurement remains difficult. This is because AI visibility works differently from traditional organic search.
In Google Search Console, we see impressions, clicks, CTR, and position. With ChatGPT, Claude, or Perplexity, this is much more fragmented. Some systems search the web in real time. Others rely on their own indexes, partner data, search engines, or cached content.
Possible sources include:
- traditional Google or Bing results
- structured website content
- external mentions
Directories and review platforms - Wikipedia, Wikidata, or specialized media
- Product and documentation pages
- Direct crawls by AI bots
- Existing training data
- Current live searches
So llms.txt is just one possible touchpoint in a much larger system. But it’s the touchpoint you can control 100%.
Why we recommend llms.txt
Despite all the uncertainty, we recommend llms.txt in most cases because the effort involved is manageable and the strategic benefits are clear.
If you know what you’re doing, the initial setup can often be completed in under an hour. With WordPress, it can even be done with a single click using Yoast or Rank Math.
This is not a substitute for SEO. Nor is it a substitute for good content marketing. But it fits well into a modern SEO strategy where many small signals work together.
llms.txt and Markdown: An exciting combination
Before we talk about what a “perfect setup” for llms.txt looks like, we need to take a step back. That’s because much of the current discussion surrounding llms.txt only really makes sense once you understand a second concept: Markdown, or text-based versions of content.
What is Markdown in the context of AI?
Markdown is essentially a highly simplified form of HTML. No design, no animations, no JavaScript. Just structured content: headings, paragraphs, lists, links.
This isn’t particularly appealing to humans. But it’s ideal for machines.
Why?
- No rendering processes required
- No dependency on CSS or JavaScript
- Focus on the actual content
With specialized tools or plugins, it’s possible to create Markdown versions of pages (.md) in just a few clicks. With a few more clicks, you can also link to the .md version from the original page (more on that later).
How llms.txt comes into play here
llms.txt can be used exactly for this purpose: as a directory for Markdown versions of your content. Instead of linking to the standard URLs, links consistently point to the corresponding .md version. This is how Anthropic does it, for example.
llms.txt is therefore not used in isolation, but as part of a new infrastructure for machine-readable content.
Server Load & Bot Traffic: Can llms.txt and Markdown Help?
One argument that comes up time and again in practice is the issue of server load.
Today, websites receive a significant portion of their traffic from bots. Some sources already estimate this to be more than 50%. Depending on the setup, the size of the site, and its popularity, this can place a noticeable strain on server resources and slow down the website—even for human users.
So the idea is: If bots can retrieve simple, streamlined content instead of complex pages, that could theoretically save resources.
This isn’t a guarantee. But it’s a plausible idea.
Important to note:
- If your site has slow loading times due to poor-quality images, lack of caching, or bad code, Markdown won’t solve the problem.
- If your server is heavily loaded due to many bot requests, it can be one component of a solution. Blocking unimportant or harmful bots can also be considered. To do this, it’s necessary to first analyze the existing bot traffic in detail. If you need help with this, feel free to contact us.
- If you have a lot of content per page, it might also make sense to
What our tests with .md Markdown have shown so far
We’re currently seeing very little traffic to the Markdown versions. However, our analyses so far have revealed a few visits, particularly from BingBot and OAI-SearchBot. We estimate the share to be about 1 %.
What’s particularly interesting is the sequence of events: the bots first visited the respective original page and then accessed the corresponding .md version. This suggests that the technical setup is fundamentally working.
In our setup, the original page includes a reference to the Markdown version in the code using `rel=“alternate”`:
<link rel="alternate" type="text/markdown" href="https://weventure.de/example-page.md" />
This means that every relevant original page (e.g., a blog post) contains a machine-readable reference to its own Markdown version. This Markdown version, in turn, stores key metadata such as title, description, original URL, date, author, language, and category. The Markdown file is therefore not isolated but remains neatly linked to the actual URL.
The Perfect Setup: Using llms.txt Correctly
Anyone who takes llms.txt seriously shouldn’t simply treat the file as just another mandatory technical file. Its true value doesn’t come from the fact that there’s an automated list stored somewhere under /llms.txt. Its value comes from selection, context, and connections.
The quick solution: One-click via plugins
Getting started with WordPress is now easy. Tools like Yoast SEO or Rank Math can automatically generate an llms.txt file. This is convenient, fast, and a sensible first step for many websites.
Especially if you just want to test whether AI bots are accessing the file at all, this kind of setup is often sufficient. Technically, then, this isn’t a major project.
However: In our tests, the automatically generated solution wasn’t the strongest option.
The problem isn’t that it’s wrong. The problem is that it often doesn’t make enough decisions. It lists content, but it doesn’t really prioritize it. It doesn’t clearly explain which pages are particularly important. It rarely reflects a company’s strategic logic.
A manually curated file is therefore better. You can also make manual adjustments in Yoast. However, in our view, the gold standard is: create it yourself, review it, and update it regularly.
The Gold Standard: Sitemap, Company Context, and AI-Powered Curation
The most pragmatic approach is as follows:
- Export the sitemap
- Select the most important URLs
- Add company information
- Organize services, topic clusters, authors, case studies, and key resources
- Use ChatGPT, Claude, or another AI tool to format the content into a clean llms.txt file and customize it to your preferences
- Review and edit manually
A good llms.txt file should not attempt to map the entire website. That’s what the sitemap is for. It should highlight the content that is truly essential for understanding the brand.
What companies can specifically include in llms.txt
Large companies use llms.txt as a flexible context format for AI systems. Here are a few examples:
| Real-world example | What can be learned from this |
| Izzy Wheels | The file can summarize the business model, key offerings, contact information, and product catalogs. |
| Google Gemini Developer API Docs | llms.txt can serve as an entry point to extensive documentation and link to many machine-readable detail pages. |
| Anthropic / Claude Docs | The file can link exclusively to Markdown versions, thereby creating a clear, AI-readable layer of documentation. |
| Weather.com | llms.txt can provide guidance on how to interpret certain URL structures or content. |
| Zapier | The file may contain technical information, such as instructions on how to use APIs. |
| Microsoft GenAIScript | The file may contain specific instructions for code generation or agent-based use. |
llms.txt doesn’t have to contain only links. It can provide context, priorities, notes, and a machine-readable structure.
Example: Structure of llms.txt
A useful structure might look something like this (using one of our own examples):
# WEVENTURE Performance GmbHWEVENTURE is a Berlin-based digital marketing agency for measurable digital growth. We combine SEO, GEO, paid media, tracking, CRO, content and AI visibility to help companies improve discoverability, lead generation and marketing performance.
## Entity Summary
Organization: WEVENTURE Performance GmbH
Location: Berlin, Germany
Market: DACH
Primary audience: B2B companies, ecommerce businesses, startups, SMEs and enterprise teams
Core expertise: SEO, GEO, SEA, social media advertising, tracking, CRO, content strategy, accessibility
Website: https://weventure.de/
## What WEVENTURE should be associated with
WEVENTURE should be understood as a digital marketing agency specializing in measurable growth, AI search visibility, performance marketing and strategic SEO/GEO consulting.
This may be followed by central service areas, such as:
## Core Services– SEO & GEO: https://weventure.de/digital-marketing/seo
– SEA: https://weventure.de/digital-marketing/sea
– Social Media Advertising: https://weventure.de/digital-marketing/sma
Or, if Markdown versions are available:
- SEO & GEO: https://weventure.de/digital-marketing/seo.md– SEA: https://weventure.de/digital-marketing/sea.md
– Social Media Advertising: https://weventure.de/digital-marketing/sma.md
Both are possible. With Claude Docs, Anthropic demonstrates that a pure Markdown setup in llms.txt can be quite useful.
In addition, an llms.txt file can include:
- Key blog posts / guides
- Case studies
- Contact / About us
- Author or expertise pages
llms.txt is part of a broader AI visibility strategy
llms.txt is not a ranking factor, not a traffic channel, and not a shortcut to AI systems. It is a building block. And just as with SEO, the same principle applies here: the impact only emerges through synergy.
Markdown versions of your pages can be a useful addition because they make content more machine-readable. But they, too, are only one part of the whole.
Crawlability & Technical Accessibility
- clean HTML structure
- strong internal linking
- no unnecessary barriers for bots
- clear URL structure
Indexing & Basic SEO
- indexable pages
- relevant keywords & topic coverage
- clear information architecture
- structured content
Many AI systems rely directly or indirectly on traditional search engines. If you’re not visible there, you’re less likely to appear in AI systems either.
Content Quality & Citationworthiness
Content is still king. AI systems favor content that:
- provides clear answers
- is well-structured
- is written in an easy-to-understand manner
- offers genuine value
Feel free to read our article on E-E-A-T for more information.
Entities & Brand Understanding
LLMs rely heavily on context and entities.
This means:
- clear positioning
- consistent brand messaging (& NAP consistency)
- recurring themes
- mentions on other pages (off-page SEO)
The clearer a system understands who you are and what you stand for, the more likely you are to be included in its responses.
Structured data
Structured data is one of the most important tools for ensuring that content is not only presented attractively but also clearly understood. Using Schema.org and JSON-LD, a website can very specifically mark up what a particular page is about.
This does not replace good content. But it reduces room for interpretation.
Bot Monitoring & Control
Allow good bots, block bad bots. Done. Right?
In reality, it’s more complicated.
Because not every unknown bot is automatically worthless. And not every bot that seems irrelevant at first glance should be immediately blocked.
A good example is PetalBot, which “visits” our site at regular intervals. At first glance, you might think: Huawei, Chinese bot, Petal Search—do we even need this for the German market, or is it actually dangerous? Better to block it?
It’s not that simple.
Because Petal Search comes pre-installed on Huawei devices by default. If a significant portion of users in Germany use Huawei smartphones—and that’s probably around 7%—this bot suddenly plays a role in visibility and discoverability. So blocking it isn’t a good decision.
Before excluding bots, you should first understand:
- Which bot is accessing your site?
- How often?
- Which URLs?
- With what crawl depth?
- Does it cause server load?
- Does it potentially bring visibility?
- Is it part of a relevant search, AI, or platform ecosystem?
- Does it crawl cleanly or aggressively?
Server logs or bot management solutions help identify patterns. Only then should you decide whether to allow, limit, or block a bot.
Control can then be implemented in various ways, e.g., robots.txt (rather unreliable) and server rules. If you need help with this, contact us today.
Conclusion: llms.txt isn't a magic bullet, but it's a step in the right direction
llms.txt won’t catapult your website to the top of ChatGPT, Claude, or Perplexity overnight. There is currently neither solid evidence of this nor a universally established standard that all major systems visibly rely on.
Nevertheless, it would be wrong to dismiss the topic as mere hype.
Because llms.txt forces us to ask a question that is becoming central in the AI era: What content should a machine learning system understand first when processing our website?
That is precisely why we recommend llms.txt as a pragmatic building block: quick to set up, easily curatable, and particularly powerful when combined with Markdown versions, structured data, clean internal linking, bot monitoring, and high-quality content.
Anyone who simply uploads a file and waits for more AI traffic will likely be disappointed. However, those who view llms.txt as part of a broader AI visibility strategy create better conditions for their brand to be correctly categorized, recommended, and, in the best case, cited by AI systems. If you need support in implementing your AI strategy, feel free to contact us for a no-obligation consultation.
FAQ about llms.txt
What is llms.txt?
llms.txt is a text file located in a website’s root directory that is designed to provide AI systems and large language models (LLMs) with curated guidance. It can consolidate important pages, company information, documentation, or machine-readable content.
Is llms.txt an official standard?
No, llms.txt is currently a proposed standard, but not an established web standard like robots.txt or sitemap.xml. That is precisely why you shouldn’t expect any guaranteed results.
Does llms.txt improve my ranking in ChatGPT and similar platforms?
No, there is currently no reliable evidence to support that. llms.txt is not a ranking factor, but rather a signal that helps AI systems understand the context, which, when combined with other factors, can improve visibility.
Should I create an llms.txt file?
Yes, in many cases it’s worth it. The effort involved is manageable, the risk is low, and the file can be effectively integrated into a broader AI visibility strategy.
What is llms-full.txt?
llms-full.txt is a more comprehensive version of llms.txt. While llms.txt is intended to be curated and compact, llms-full.txt can contain significantly more content and serve as a larger context store for AI systems.
Do I need both llms.txt and llms-full.txt?
Not necessarily. For most websites, a well-structured llms.txt file is sufficient to start with. llms-full.txt may be useful if you plan to provide a large amount of content, documentation, or technical information for AI systems.
What should be included in an llms.txt file?
Useful content includes company information, key service pages, important blog posts, documentation, contact pages, author profiles, usage guidelines, and optional links to Markdown versions.
What should not be included in an llms.txt file?
Don’t just include the entire sitemap unfiltered. A good llms.txt file intentionally prioritizes content and explains which items are particularly important.
What is Markdown in the context of llms.txt?
Markdown is a stripped-down, text-based version of a page. It primarily contains structured content such as headings, paragraphs, lists, and links—without any design, JavaScript, or visual elements.
Why are Markdown versions interesting for LLMs?
They can make content easier to read because machines have less rendering overhead to process. This can be particularly useful for large amounts of content, documentation, or highly dynamic websites.
Does my llms.txt file need to link to Markdown files?
No. An llms.txt file can also contain regular URLs. Markdown versions are optional, but they can be useful if you want to make your content more machine-readable or establish an interconnected AI ecosystem on your site.
How do I link an original page to a Markdown version?
For example, using a rel=”alternate” link in the HTML code of the original page:
<link rel="alternate" type="text/markdown" href="https://deine-domain.de/example-page.md" />
Can Markdown reduce server load?
It depends. If a lot of bot traffic hits complex pages, lean Markdown versions can theoretically be more resource-efficient. However, Markdown won’t solve problems caused by images, JavaScript, caching, or code.
Should I block AI bots via robots.txt?
Don’t make sweeping generalizations. Some bots may seem irrelevant at first glance, but they can still boost visibility. Analyze first, then decide.
How do I measure whether AI bots are visiting my website?
Through server logs, bot analytics, or specialized tools like vigIA or Known Agents. What matters isn’t just whether a bot was there, but which pages it visited and whether it crawled deeper.
Does WEVENTURE help with llms.txt?
Yes. WEVENTURE provides support for analysis, strategy, technical implementation, and monitoring related to llms.txt, Markdown versions, AI bot traffic, and AI visibility.
Does WEVENTURE also create a complete AI visibility strategy?
Yes. In addition to llms.txt, this includes GEO, SEO, structured data, content strategy, entities, internal linking, bot monitoring, and technical accessibility.
