Add llms.txt discovery indexes to docs.fedoraproject.org

cverna commented

2026-09-12 20:08:41 +00:00

Propose generating llms.txt (https://llmstxt.org/) discovery indexes for the English Fedora Docs site as part of the existing build.

This is a v1, index-only proposal: it publishes a curated, grouped map of the site that links to the existing HTML pages. It deliberately does not generate per-page Markdown, translations, or a merged corpus.

Example

# Fedora Documentation

> Official documentation for Fedora Linux, Fedora CoreOS, and related projects.
> Each component has its own llms.txt with a full page index.

## Getting started
- [A Beginner's Guide to Fedora](/en-US/beginners-guide/llms.txt): Last updated: April 2026
- [Quick Docs](/en-US/quick-docs/llms.txt): This is a collection of short HOW-TO and FAQ-style documentation for Fedora users. Use the navigation menu on the left to find an area of interest, and under that, solutions to your specific problem.
- [Fedora User Docs](/en-US/fedora/llms.txt): The Fedora documentation provides extensive information on how to install and use the Fedora operating system and the software packaged by the Fedora Project. Fedora is a large project. Therefore,...
- [Fedora Linux Releases](/en-US/releases/llms.txt): Fedora creates two major OS releases every year, targeted for the fourth Tuesday in April and October. We don’t follow a strict "ship on this date!" policy, nor do we wait until every single possible...
- [Fedora Accounts](/en-US/fedora-accounts/llms.txt): Fedora Accounts and CentOS Accounts is the primary source of accounts for Fedora Project and CentOS contributors. The majority of applications and services used by the Fedora Project and CentOS...

I have a PoC --> cverna/docs-fp-o → llms-txt (https://forge.fedoraproject.org/cverna/docs-fp-o/src/branch/llms-txt).

Why should we do this?

llms.txt adoption is concentrated in software documentation. Shipping a valid index is a low-cost, forward-looking convenience for AI agents and tooling that opt in.
The generation is a post-Antora step writing regular files under /antora/output, which rsync and Apache already carry, so we don't need infra changes for this.

It would be generated on every English build:

/llms.txt	Domain-root copy of the site index (conventional probe target)
/en-US/llms.txt	Site index: components grouped, each linking to its component file
/en-US/<component>/llms.txt	Per-component page index from the navigation order

Assited-by: OpenCode

Propose generating llms.txt (https://llmstxt.org/) discovery indexes for the English Fedora Docs site as part of the existing build. This is a v1, index-only proposal: it publishes a curated, grouped map of the site that links to the existing HTML pages. It deliberately does not generate per-page Markdown, translations, or a merged corpus. Example ``` # Fedora Documentation > Official documentation for Fedora Linux, Fedora CoreOS, and related projects. > Each component has its own llms.txt with a full page index. ## Getting started - [A Beginner's Guide to Fedora](/en-US/beginners-guide/llms.txt): Last updated: April 2026 - [Quick Docs](/en-US/quick-docs/llms.txt): This is a collection of short HOW-TO and FAQ-style documentation for Fedora users. Use the navigation menu on the left to find an area of interest, and under that, solutions to your specific problem. - [Fedora User Docs](/en-US/fedora/llms.txt): The Fedora documentation provides extensive information on how to install and use the Fedora operating system and the software packaged by the Fedora Project. Fedora is a large project. Therefore,... - [Fedora Linux Releases](/en-US/releases/llms.txt): Fedora creates two major OS releases every year, targeted for the fourth Tuesday in April and October. We don’t follow a strict "ship on this date!" policy, nor do we wait until every single possible... - [Fedora Accounts](/en-US/fedora-accounts/llms.txt): Fedora Accounts and CentOS Accounts is the primary source of accounts for Fedora Project and CentOS contributors. The majority of applications and services used by the Fedora Project and CentOS... ``` I have a PoC --> cverna/docs-fp-o → llms-txt (https://forge.fedoraproject.org/cverna/docs-fp-o/src/branch/llms-txt). Why should we do this? - llms.txt adoption is concentrated in software documentation. Shipping a valid index is a low-cost, forward-looking convenience for AI agents and tooling that opt in. - The generation is a post-Antora step writing regular files under /antora/output, which rsync and Apache already carry, so we don't need infra changes for this. It would be generated on every English build: ``` /llms.txt Domain-root copy of the site index (conventional probe target) /en-US/llms.txt Site index: components grouped, each linking to its component file /en-US/<component>/llms.txt Per-component page index from the navigation order ``` Assited-by: OpenCode

cverna referenced this issue from docs/docs-fp-o

2026-09-13 07:19:48 +00:00

WIP: add-antora-build-workflows #53

hricky commented

2026-09-13 09:20:21 +00:00

Owner

AIUI, this aims to help LLMs and AI agents spend less time crawling the Docs site. I’m not sure if it will work, though, since the site is protected against data scraping by Anubis. @kevin is probably more familiar with the subject.

AIUI, this aims to help LLMs and AI agents spend less time crawling the Docs site. I’m not sure if it will work, though, since the site is protected against data scraping by [Anubis](https://github.com/TecharoHQ/anubis). @kevin is probably more familiar with the subject.

cverna commented

2026-09-13 10:25:46 +00:00

Author

@hricky wrote in #64 (comment):

AIUI, this aims to help LLMs and AI agents spend less time crawling the Docs site.

Not it is mostly to save tokens when working with AI agents and the Fedora docs. Instead of having to get the full HTML page, convert it to markdown and then find the links, an agent can directly get that information from the llms.txt.
It is not a huge optimization but it is also a small cost to implement.

I’m not sure if it will work, though, since the site is protected against data scraping by Anubis. @kevin is probably more familiar with the subject.

It is not that difficult for Agents or tools to by-pass Anubis, which is good. Here the use case is a legitimate Agent or Contributor using a coding Agent for accessing Fedora Docs.

@hricky wrote in https://forge.fedoraproject.org/docs/tickets/issues/64#issuecomment-1356550: > AIUI, this aims to help LLMs and AI agents spend less time crawling the Docs site. Not it is mostly to save tokens when working with AI agents and the Fedora docs. Instead of having to get the full HTML page, convert it to markdown and then find the links, an agent can directly get that information from the llms.txt. It is not a huge optimization but it is also a small cost to implement. > I’m not sure if it will work, though, since the site is protected against data scraping by [Anubis](https://github.com/TecharoHQ/anubis). @kevin is probably more familiar with the subject. It is not that difficult for Agents or tools to by-pass Anubis, which is good. Here the use case is a legitimate Agent or Contributor using a coding Agent for accessing Fedora Docs.

hricky commented

2026-09-13 12:32:21 +00:00

Owner

AFAICS this proposal modifies the Containerfile by adding an additional 700-line Python script (likely LLM generated), the python3-beautifulsoup4 module and some config. It also modifies build-scripts/rebuild-site.py to execute the build-scripts/gen_llms.py script.
I don't know if it is not a huge optimization, but it's also a small cost to implement, and I'm not against it. I do wonder, however, if it's reasonable to run it every single hour just to save tokens for a hypothetical legitimate Agent or Contributor using a coding Agent for accessing Fedora Docs.

AFAICS this proposal modifies the Containerfile by adding an additional 700-line Python script (likely LLM generated), the `python3-beautifulsoup4` module and some config. It also modifies `build-scripts/rebuild-site.py` to execute the `build-scripts/gen_llms.py` script. I don't know if it is not a huge optimization, but it's also a small cost to implement, and I'm not against it. I do wonder, however, if it's reasonable to run it every single hour just to save tokens for a hypothetical legitimate Agent or Contributor using a coding Agent for accessing Fedora Docs.

cverna commented

2026-09-19 09:25:00 +00:00

Author

@hricky wrote in #64 (comment):

I do wonder, however, if it's reasonable to run it every single hour just to save tokens for a hypothetical legitimate Agent or Contributor using a coding Agent for accessing Fedora Docs.

How often do we rebuild and publish the docs? Running the new script really doesn't take much time compared to the already quite long process of pulling all the docs git repos.

@hricky wrote in https://forge.fedoraproject.org/docs/tickets/issues/64#issuecomment-1356571: > I do wonder, however, if it's reasonable to run it every single hour just to save tokens for a hypothetical legitimate Agent or Contributor using a coding Agent for accessing Fedora Docs. How often do we rebuild and publish the docs? Running the new script really doesn't take much time compared to the already quite long process of pulling all the docs git repos.

hricky commented

2026-09-25 09:01:10 +00:00

Owner

@cverna wrote in #64 (comment):

How often do we rebuild and publish the docs? Running the new script really doesn't take much time compared to the already quite long process of pulling all the docs git repos.

The English versions of the Fedora Docs sites are rebuilt hourly for both the prod and stg branches.

I have researched the topic and have some questions and thoughts. Has anyone actually requested llms.txt support? The indexing would only help with discovery. Agents would still need to fetch and parse full HTML pages. We're adding a 700+ line script, a new dependency, and ongoing maintenance for an unproven standard with no evidence of adoption. "Small cost" additions accumulate over time. We protect the docs from scraping for good reasons. Publishing a machine-readable index may make it easier to circumvent those protections, even if that's not the immediate intent.

The real question is: what problem are we actually solving here? If contributors are struggling to use Fedora Docs with AI tools, that's worth discussing. But absent that feedback, this feels premature.

@cverna wrote in https://forge.fedoraproject.org/docs/tickets/issues/64#issuecomment-1358439: > How often do we rebuild and publish the docs? Running the new script really doesn't take much time compared to the already quite long process of pulling all the docs git repos. The English versions of the Fedora Docs sites are rebuilt hourly for both the `prod` and `stg` branches. I have researched the topic and have some questions and thoughts. Has anyone actually requested `llms.txt` support? The indexing would only help with discovery. Agents would still need to fetch and parse full HTML pages. We're adding a 700+ line script, a new dependency, and ongoing maintenance for an unproven standard with no evidence of adoption. "Small cost" additions accumulate over time. We protect the docs from scraping for good reasons. Publishing a machine-readable index may make it easier to circumvent those protections, even if that's not the immediate intent. The real question is: what problem are we actually solving here? If contributors are struggling to use Fedora Docs with AI tools, that's worth discussing. But absent that feedback, this feels premature.

cverna commented

2026-09-25 10:00:56 +00:00

Author

@hricky wrote in #64 (comment):

@cverna wrote in #64 (comment):

How often do we rebuild and publish the docs? Running the new script really doesn't take much time compared to the already quite long process of pulling all the docs git repos.

The English versions of the Fedora Docs sites are rebuilt hourly for both the prod and stg branches.

👍

I have researched the topic and have some questions and thoughts. Has anyone actually requested llms.txt support?

Well I requested support in this ticket 😄, if we decide to go this way I have plans to make a skill (https://forge.fedoraproject.org/ai-ml/skills-library) to instruct agents to use these indexes.

The indexing would only help with discovery. Agents would still need to fetch and parse full HTML pages.

Yes, this is for me the middle ground. We make it easy and cheap for Agents to find which page to actually fetch. As a first step I don't think it is useful to convert and serve all the docs pages as markdown files.

We're adding a 700+ line script, a new dependency, and ongoing maintenance for an unproven standard with no evidence of adoption.

Well, it is not (yet) widely adopted but there is evidence of adoption https://llmstxt.site/

"Small cost" additions accumulate over time. We protect the docs from scraping for good reasons. Publishing a machine-readable index may make it easier to circumvent those protections, even if that's not the immediate intent.

I don't think this has any impact or relation with scraping. These could/would still be protected by Anubis.

The real question is: what problem are we actually solving here? If contributors are struggling to use Fedora Docs with AI tools, that's worth discussing. But absent that feedback, this feels premature.

Well the current way to work with Fedora's Docs and AI is not ideal. What I am seeing the most is folks cloning the git repos (see example bellow). I agree that having llms.txt file will not radically improve things, but it's a small steps towards making it easier for Agents to do work in Fedora.

https://github.com/hroncok/llm-review/blob/main/src/llm_review/workspace.py#L20,
ai-ml/skills-library – skills/fedora-packaging/assets/sync-guidelines.sh
Line 14 in ai-ml/skills-library@9e6b5c6
REPO_URL="https://forge.fedoraproject.org/packaging/guidelines.git"

@hricky wrote in https://forge.fedoraproject.org/docs/tickets/issues/64#issuecomment-1360153: > @cverna wrote in #64 (comment): > > > How often do we rebuild and publish the docs? Running the new script really doesn't take much time compared to the already quite long process of pulling all the docs git repos. > > The English versions of the Fedora Docs sites are rebuilt hourly for both the `prod` and `stg` branches. 👍 > > I have researched the topic and have some questions and thoughts. Has anyone actually requested `llms.txt` support? Well I requested support in this ticket 😄, if we decide to go this way I have plans to make a skill (https://forge.fedoraproject.org/ai-ml/skills-library) to instruct agents to use these indexes. > The indexing would only help with discovery. Agents would still need to fetch and parse full HTML pages. Yes, this is for me the middle ground. We make it easy and cheap for Agents to find which page to actually fetch. As a first step I don't think it is useful to convert and serve all the docs pages as markdown files. > We're adding a 700+ line script, a new dependency, and ongoing maintenance for an unproven standard with no evidence of adoption. Well, it is not (yet) widely adopted but there is evidence of adoption https://llmstxt.site/ > "Small cost" additions accumulate over time. We protect the docs from scraping for good reasons. Publishing a machine-readable index may make it easier to circumvent those protections, even if that's not the immediate intent. I don't think this has any impact or relation with scraping. These could/would still be protected by Anubis. > > The real question is: what problem are we actually solving here? If contributors are struggling to use Fedora Docs with AI tools, that's worth discussing. But absent that feedback, this feels premature. Well the current way to work with Fedora's Docs and AI is not ideal. What I am seeing the most is folks cloning the git repos (see example bellow). I agree that having llms.txt file will not radically improve things, but it's a small steps towards making it easier for Agents to do work in Fedora. * https://github.com/hroncok/llm-review/blob/main/src/llm_review/workspace.py#L20, * https://forge.fedoraproject.org/ai-ml/skills-library/src/commit/9e6b5c6101ae9da4ebd336ffcf0affdb6496c383aa9741dc2034a53c6052a552/skills/fedora-packaging/assets/sync-guidelines.sh#L14

pbokoc added the

Contributors' Guidance

technical support measures

labels

2026-09-30 05:51:05 +00:00

pbokoc commented

2026-09-30 05:53:44 +00:00

Owner

I think we'll need to do something like this eventually, so why not now.

Question, though - is there a way to validate the output? How do we tell whether the generated llms.txt makes sense? I'm concerned about possible future changes - this seems like something that would sit in our build pipeline with no way for us to tell if it still works or not.

I think we'll need to do something like this eventually, so why not now. Question, though - is there a way to validate the output? How do we tell whether the generated llms.txt makes sense? I'm concerned about possible future changes - this seems like something that would sit in our build pipeline with no way for us to tell if it still works or not.

hricky commented

2026-09-30 08:10:32 +00:00

Owner

@cverna wrote in #64 (comment):

Well the current way to work with Fedora's Docs and AI is not ideal. What I am seeing the most is folks cloning the git repos (see example bellow). I agree that having llms.txt file will not radically improve things, but it's a small steps towards making it easier for Agents to do work in Fedora.

In my experience, cloning Git repos is the way to work with Agents in the Fedora Project as a whole, not only with the docs. Let's try to add llms.txt to the Docs site and see if it improves things at least a bit.

@pbokoc wrote in #64 (comment):

Question, though - is there a way to validate the output? How do we tell whether the generated llms.txt makes sense? I'm concerned about possible future changes - this seems like something that would sit in our build pipeline with no way for us to tell if it still works or not.

I would suggest first adding a single llms.txt file, and then implementing the skill @cverna mentioned, to confirm that everything works as expected. To test this, I can migrate the Fedora Docs Build Repository to the Documentation Team Staging org. Then I can try to set up a CI/pipeline that generates the Docs site and a preview with the llms.txt file included. For this, we can try to use our runnerhost machine that is already set up and running.

@cverna wrote in https://forge.fedoraproject.org/docs/tickets/issues/64#issuecomment-1360185: > Well the current way to work with Fedora's Docs and AI is not ideal. What I am seeing the most is folks cloning the git repos (see example bellow). I agree that having llms.txt file will not radically improve things, but it's a small steps towards making it easier for Agents to do work in Fedora. In my experience, cloning Git repos is the way to work with Agents in the Fedora Project as a whole, not only with the docs. Let's try to add `llms.txt` to the Docs site and see if it improves things at least a bit. @pbokoc wrote in https://forge.fedoraproject.org/docs/tickets/issues/64#issuecomment-1361578: > Question, though - is there a way to validate the output? How do we tell whether the generated llms.txt makes sense? I'm concerned about possible future changes - this seems like something that would sit in our build pipeline with no way for us to tell if it still works or not. I would suggest first adding a single `llms.txt` file, and then implementing the skill @cverna mentioned, to confirm that everything works as expected. To test this, I can migrate the [Fedora Docs Build Repository](https://forge.fedoraproject.org/docs/docs-fp-o) to the [Documentation Team Staging](https://forge.stg.fedoraproject.org/docs) org. Then I can try to set up a CI/pipeline that generates the Docs site and a preview with the `llms.txt` file included. For this, we can try to use our `runnerhost` machine that is already set up and running.

pbokoc commented

2026-10-01 10:38:12 +00:00

Owner

Oh hey, FYI

https://antora.zulipchat.com/#narrow/channel/282400-users/topic/Generation.20of.20an.20llms.2Etxt.20file.20at.20root/near/524780912

Oh hey, FYI https://antora.zulipchat.com/#narrow/channel/282400-users/topic/Generation.20of.20an.20llms.2Etxt.20file.20at.20root/near/524780912

👍 1 ❤️ 1

cverna commented

2026-10-02 10:05:55 +00:00

Author

Interesting, I didn't think about checking existing Antora plugins. Might be interesting to try https://github.com/cerbos/antora-llm-generator