What do you think about llms.txt, guys? What is it exactly and how do I create my own llms.txt for my website??
I’m not a coder; is this complicated to do? Also ONE MORE THING does it improve SEO and ranking?
Could you add the latest updates about it here? That would be great!
Basically, llms.txt is a simple text file that helps large language models like ChatGPT find and understand your website’s most important content.
On september 3, 2024, Jeremy Howard shared a proposal. Here’s the link, you can check it out:
As for how to create your own llms.txt file, you don’t need to know any code, but it might take a bit of time.
Since I did this recently, I will show you step by step following the documentation I created for myself.
-First; define your objectives clearly.
Identify primary pages (products, blogs, key resources) and their importance for AI models.
-Secondly; map your content
Create a sitemap-like list or outline the URLs and descriptions of important content.
-Then, format it in Markdown
Use simple Markdown syntax for clarity and ease of parsing by AI.
Here’s a basic example of what an llms.txt might look like:
# Your Website Name
> Brief summary of your site, including its primary purpose.
Additional details or notes.
## Core Content
- [Homepage](https://example.com): Our main page, describing core services.
- [Products](https://example.com/products): List of key offerings with descriptions.
## Optional
- [Documentation](https://example.com/docs): Detailed docs, useful but secondary for LLM context.
However, I’m also not entirely sure will it make my website appear more often in ChatGPT’s search results. If so, it is basically the new generation ‘Sitemap’…
@GodlessDobby many LLM models like ChatGPT, Claude, Grok, Google Gemini, etc., need to train their datasets on content.
Your content needs to be in a format that can be easily discoverable, indexable, and understandable by AI engines (technically LLMs).
So yeah, in this case, llms.txt will probably make your website appear more often in ChatGPT’s search results. 
By the way, don’t bother creating it by yourself, there are free tools that quickly convert your website to llms.txt format for llms.
Free LLMs.txt GeneratorThx, this tool pretty cool!!
So is llms.txt a new standard?
What do you think? And is llms.txt useful even if it’s not a standard?
Yes!
Even without universal adoption, llms.txt can provide practical advantages for website owners, AI models, and developers.
Should You Use It?
You can see many companies in that list. This page helps us monitor which websites are implementing llms.txt.
Soon, I bet there might be an announcement from major players like ChatGPT that this has become standardized.
Have you seen these?
This shows that it’s not just hype. Therefore, it’s a good idea to implement llms.txt on your website.
Below are a few tools you can use to quickly convert your website into a format that LLMs can understand:
- LiveChatAI’s free llms.txt generator at “Free LLMs.txt Generator”
- SiteSpeakAI’s generator at “SiteSpeakAI - llms.txt Generator”
- Firecrawl’s llms.txt Generator at “Generate llms.txt”
I missed them. That was a big surprise. I feel like I have to be quick, something big is coming. Thanks for the tools, @fatih2555 & @ecesanan I’ll get right to work!
After I convert my website into markdown text for LLMs, what do I do next?
How do I implement llms.txt on my website?
In your website’s root directory, create a file named llms.txt and place it there (e.g., https://yourwebsite.com/llms.txt).
If your website uses a CMS (WordPress, Shopify, etc.), just upload the file via FTP or a file manager.
If you want an easy way to generate and update llms.txt, you can embed the llms.txt tool on your website.
It gives you a starting point for your llms.txt file quickly.
<div class="livechat-embed">
<iframe style="height: 135px; width: 100vw; margin-top: 3rem;"
allowtransparency="true"
frameborder="0"
id="llms-txt-generator"
scrolling="yes"
src="https://app.livechatai.com/free-tools/en/llm-txt-generator"
allow="clipboard-write">
</iframe>
<a href="https://livechatai.com" rel="dofollow">Powered by LiveChatAI</a>
</div>
I’m thankful, many thanks, I’ve got a lot to learn.
I am just still not get why robots.txt or sitemap doesn’t replace it.
Can someone explain the differences between robots.txt and sitemap and llms.txt?
Sure let me explain: llms.txt works together with other common website files (robots.txt and sitemap.xml). Each has its own role:
robots.txt tells search engines which pages they’re allowed to visit or not.
sitemap.xml lists all your pages to help search engines index your website fully, but it can include pages not useful for AI.
llms.txt is different—it clearly highlights your site’s key pages specifically for AI models, helping them quickly find and accurately understand your most important content.
In other words, llms.txt doesn’t replace the others, it’s an extra tool that makes your website content clearer and easier for AI systems to use when answering people’s questions or creating responses.
| Feature | Robots.txt | Sitemap.xml | LLMs.txt |
|---|---|---|---|
| Purpose | Controls crawler access to content | Lists all pages for search indexing | Provides AI-optimized, structured content |
| Audience | Web crawlers (SEO-focused) | Web crawlers (indexing-focused) | AI models and reasoning engines |
| Content | Permissions for crawling and indexing | List of all pages | Simplified and structured content tailored for AI readability |
| Use Case | SEO and access management | SEO and indexing of all content | AI chatbots, LLM context generation, automation tasks |
Hi, in the bookstore where I work, we recently added a LiveChatAI to our website to support and answer questions about books, shipping, returns, etc.
Can you help me understand how to use llms.txt to improve the accuracy of our chatbot’s answers?
Hey nikki, that’s definitely a good decision - I’m sure it will work.
You’ll just add the text from your generated llms.txt file (via the generator) as a data source in LiveChatAI.
Additionally, you can also use these resources to improve the quality of the chatbot’s answers:
