docs/tickets

Monitoring and incident reporting for docs site builds (prod and stg) #53

New issue

Open

opened 2026-06-29 18:23:43 +00:00 by jflory7 · 3 comments

jflory7 commented

2026-06-29 18:23:43 +00:00

Owner

Summary

When changes are pushed to docs-fp-o, the production docs site sometimes rebuilds successfully while the staging site does not. This has happened multiple times with no automated detection, sometimes going unnoticed for weeks. There is no documented process for where or how to report these failures.

Context

This issue was discussed in the #docs:fedoraproject.org Matrix room where the following points were raised:

@hricky observed that after recent pushes to docs-fp-o, production rebuilt but staging did not, and that this has happened before.
@darknao indicated that reporting in the Matrix room is fine, but @hricky noted that messages can get buried in a busy room.
@jflory7 suggested either docs/tickets or docs/docs-fp-o as reporting locations, and raised the idea of automated alerting in Matrix when builds break.

Proposed actions

Document where to report docs site build failures. Decide on a canonical location (e.g. docs/tickets, docs/docs-fp-o, or Matrix) and document it so contributors know where to go.
Investigate why staging sometimes fails to rebuild when production rebuilds successfully from the same push.
Add monitoring or alerting to detect when prod or stg builds fail or diverge. Ideally, post a notification to the #docs:fedoraproject.org Matrix room.

#49 — Shared Forgejo Actions workflow for Fedora Docs components (CI for PRs, different scope)
#19 — Outdated docsbuilder.sh pulls Antora container w/ EOL nodejs

Assisted-by: Claude Opus 4.6 (1M context)

## Summary When changes are pushed to `docs-fp-o`, the production docs site sometimes rebuilds successfully while the staging site does not. This has happened multiple times with no automated detection, sometimes going unnoticed for weeks. There is no documented process for where or how to report these failures. ## Context This issue was discussed in the [#docs:fedoraproject.org](https://matrix.to/#/#docs:fedoraproject.org) Matrix room where the following points were raised: - @hricky observed that after recent pushes to `docs-fp-o`, production rebuilt but staging did not, and that this has happened before. - @darknao indicated that reporting in the Matrix room is fine, but @hricky noted that messages can get buried in a busy room. - @jflory7 suggested either `docs/tickets` or `docs/docs-fp-o` as reporting locations, and raised the idea of automated alerting in Matrix when builds break. ## Proposed actions 1. **Document where to report docs site build failures.** Decide on a canonical location (e.g. `docs/tickets`, `docs/docs-fp-o`, or Matrix) and document it so contributors know where to go. 2. **Investigate why staging sometimes fails to rebuild** when production rebuilds successfully from the same push. 3. **Add monitoring or alerting** to detect when prod or stg builds fail or diverge. Ideally, post a notification to the [#docs:fedoraproject.org](https://matrix.to/#/#docs:fedoraproject.org) Matrix room. ## Related issues - #49 — Shared Forgejo Actions workflow for Fedora Docs components (CI for PRs, different scope) - #19 — Outdated docsbuilder.sh pulls Antora container w/ EOL nodejs --- Assisted-by: Claude Opus 4.6 (1M context)

jflory7 added this to the 2026 Q3 Team Goals milestone

2026-06-29 18:23:43 +00:00

jflory7 added the

labels

2026-06-29 18:23:43 +00:00

pbokoc added this to the Docs Team Meeting Planner project

2026-07-13 16:59:35 +00:00

jflory7 commented

2026-07-14 14:22:28 +00:00

Author

Owner

Discussed in 2026-07-14 Fedora Docs Team meeting.

@hricky volunteered to take on this issue, as it connects to the CI/infrastructure work he is already doing on docs/tickets#49. @pbokoc described the core problem: the site sometimes fails to build silently, and the OpenShift error logs are only accessible to a few people (currently @pbokoc and @darknao). The ideal solution would be a watchdog connected to OpenShift that alerts in #docs:fedoraproject.org on Matrix and publishes error logs publicly. @hricky confirmed that Forgejo Actions build failure alerts have recently been set up and are now posting to #docs:fedoraproject.org, and the CI work in docs/docs-fp-o PR#46 may address some log visibility. However, @pbokoc clarified that the OpenShift logs currently only report fatal errors, not content warnings like broken xrefs or missing images — so there is still a gap between what CI catches and what a full production build reports.

Follow-up:

@hricky: Assigned to work on this issue
@hricky: Will test whether the Forgejo Actions CI can provide full site build error reporting
@hricky: Reach out to @gwmngilfen, who offered to help with monitoring efforts
If automated monitoring is in place, a manual reporting procedure may not be needed

_{Assisted-by: Claude Opus 4.6 (1M context)}

_Discussed in [2026-07-14 Fedora Docs Team meeting](https://discussion.fedoraproject.org/t/fedora-docs-team-meeting-2026-07-14/196628)._ --- @hricky volunteered to take on this issue, as it connects to the CI/infrastructure work he is already doing on [docs/tickets#49](https://forge.fedoraproject.org/docs/tickets/issues/49). @pbokoc described the core problem: the site sometimes fails to build silently, and the OpenShift error logs are only accessible to a few people (currently @pbokoc and @darknao). The ideal solution would be a watchdog connected to OpenShift that alerts in `#docs:fedoraproject.org` on Matrix and publishes error logs publicly. @hricky confirmed that Forgejo Actions build failure alerts have recently been set up and are now posting to `#docs:fedoraproject.org`, and the CI work in [docs/docs-fp-o PR#46](https://forge.fedoraproject.org/docs/docs-fp-o/pulls/46) may address some log visibility. However, @pbokoc clarified that the OpenShift logs currently only report fatal errors, not content warnings like broken xrefs or missing images — so there is still a gap between what CI catches and what a full production build reports. **Follow-up:** - @hricky: Assigned to work on this issue - @hricky: Will test whether the Forgejo Actions CI can provide full site build error reporting - @hricky: Reach out to @gwmngilfen, who offered to help with monitoring efforts - If automated monitoring is in place, a manual reporting procedure may not be needed <sub>_Assisted-by: Claude Opus 4.6 (1M context)_</sub>

hricky was assigned by jflory7

2026-07-14 14:22:29 +00:00

hricky commented

2026-07-15 09:01:11 +00:00

Member

Proposal: Improve Build Workflow for Fedora Documentation Site

Update and refactor the Containerfile (renamed from Dockerfile).
Create a justfile for local build automation that simplifies the workflow, with recipes for:

Container image building
Docs site generation
Local preview

Document the local workflow and the building process of the site in OpenShift in README.md and/or dedicated documentation pages.
Build the CI workflow to produce the generated documentation site. Defer the decision on artifact storage and deployment to a follow-up discussion once the build workflow is stable.

## Proposal: Improve Build Workflow for Fedora Documentation Site 1. Update and refactor the `Containerfile` (renamed from `Dockerfile`). 2. Create a `justfile` for local build automation that simplifies the workflow, with recipes for: - Container image building - Docs site generation - Local preview 3. Document the local workflow and the building process of the site in OpenShift in `README.md` and/or dedicated documentation pages. 4. Build the CI workflow to produce the generated documentation site. Defer the decision on artifact storage and deployment to a follow-up discussion once the build workflow is stable.

gwmngilfen commented

2026-07-15 16:08:30 +00:00

So, monitoring. Infra use Zabbix to do our monitoring, and I've been working with the COPR folks to enable them to have their own space inside that. They have their own checks, triggers, a Matrix room for alerts, and a UI dashboard to look at.

I think we could perhaps do the same here, if it makes sense. The key questions I have to answer are:

How is the build failures detected? Is there an API to hit, a script to run, or a file to read?
Where does the target for (1) live? I think this is an OpenShift app, right?

These matter because all the Zabbix config lives in the Infra Ansible repo, which in practice means I need either a host (eg, proxy01 or koji02) to attach checks too. However, I am working on how to store OpenShift monitoring for individual apps (as we have quite a few) so that would work too.

Thoughts?

So, monitoring. Infra use [Zabbix](https://zabbix.fedoraproject.org) to do our monitoring, and I've been working with the COPR folks to enable them to have their own space inside that. They have their own checks, triggers, a Matrix room for alerts, and a UI dashboard to look at. I *think* we could perhaps do the same here, if it makes sense. The key questions I have to answer are: 1. How is the build failures detected? Is there an API to hit, a script to run, or a file to read? 2. Where does the target for (1) live? I think this is an OpenShift app, right? These matter because all the Zabbix config lives in the Infra Ansible repo, which in practice means I need either a host (eg, proxy01 or koji02) to attach checks too. However, I am working on how to store OpenShift monitoring for individual apps (as we have quite a few) so that would work too. Thoughts?