ai-ml/packaging-guide-ramalama
2
4
Fork 2
Code Issues Pull requests Releases Activity Actions

📝 docs: Add project docs for contributors, users, and AI agents

Browse source 
The `ai-ml/packaging-guide-ramalama` repo had code and data but not any
documentation explaining how to run the pipeline, how to contribute, or
how AI agents should interact with the project. These three files fill
that gap.

`GUIDE.md` walks a user from a fresh clone through corpus staging to a
built RAG vector store and a live query — the full pipeline end to end.
It is written to eventually move to the AI/ML SIG docs site once the
project stabilizes. `CONTRIBUTING.md` codifies the PR workflow, DCO
sign-off expectations, and AI-assisted contribution disclosure per
Fedora policy. `AGENTS.md` gives coding agents the project layout,
RamaLama API constraints, and safety guardrails they need to contribute
without inventing flags or committing secrets.

All three docs reflect the current state of the data-prep pipeline and
encode review feedback: one-sentence-per-line convention, corpus
terminology defined on first use, Fedora Packaging Guidelines
acknowledged as the primary and heaviest-weight data source, and the AI
disclosure exemption scoped strictly to superficial language help.

Assisted-by: Claude Opus 4.8
Signed-off-by: gtfrans2re <francoisgonothitoure@gmail.com>
Signed-off-by: Justin Wheeler <jwheel@redhat.com>
This commit is contained in:
Francois Gonothi Toure 2026-07-03 17:05:52 -04:00 • committed by Justin Wheeler
commit 2a0ce36d2b
Signed by: jflory7
GPG key ID: 7748B15FA8FA4C7E
6 changed files with 763 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

192
AGENTS.md Normal file
View file

@ -0,0 +1,192 @@
# AGENTS.md
Guidance for AI coding agents working on **packaging-guide-ramalama**.
This is a RamaLama-based Retrieval-Augmented Generation (RAG) assistant that helps identify deviations from the Fedora RPM Packaging Guidelines.
Think of this as a README written specifically for coding agents.
For human-facing docs, see [README.md](README.md) and [CONTRIBUTING.md](docs/CONTRIBUTING.md).
## Project overview
- **Purpose:** an assistant that reviews RPM spec files and packaging questions against the Fedora Packaging Guidelines using a local RAG pipeline.
- **Core tech:** [RamaLama](https://github.com/containers/ramalama) (RAG build and run), [Docling](https://github.com/docling-project/docling) (document conversion and chunking, used internally by RamaLama), and small open-weight GGUF models.
- **Language scope:** the corpus (the collection of documents the RAG system ingests and retrieves from) is scoped to Python, C, and shell packaging, the foundational languages for OS-level development in Fedora.
- **Artifacts:** RAG vector-store images are published to Quay and consumed by the app and CLI.
## Primary data source
The [Fedora Packaging Guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/) are the central data source for this project.
They are the rules the assistant checks packaging work against, so they are the obvious and most important source in the corpus.
The AsciiDoc source is cloned from the Fedora Forge and staged with the heaviest weight in the dataset.
Everything else (the RPM Packaging Guide, the in-tree RPM Reference Manual, and real-world spec files) is supporting material around these guidelines.
> Possible follow-up: it may be worth adding a git submodule for specific source content, such as the Fedora Docs packaging/guidelines sources, so the exact upstream revision is pinned in-tree.
> This has not been done yet and should be evaluated against the current fetch-and-stage approach.
## Repository layout
- `scripts/` - pipeline: `fetch_sources.py` -> `stage_corpus.sh` -> build
- `data/sources.yaml` - the annotated source list; single source of truth for what the corpus is built from
- `data/source/` - the staged corpus that `ramalama rag` ingests (gitignored, rebuilt on demand)
- `data/raw/` - untouched fetched material (gitignored)
- `data/MANIFEST.json` - per-source provenance: resolved URL, timestamp, and git commit hash (gitignored)
- `tests/` - pytest suite for the fetch pipeline (network and git mocked)
- `docs/` - guides (GUIDE, CONTRIBUTING)
## Setup commands
On Fedora Linux, install the system dependencies first.
RamaLama and Podman are not Python packages and must be installed separately.
```bash
# System dependencies (Fedora Linux)
sudo dnf install python3 python3-pip git podman ramalama
# Python dependencies for the pipeline
pip install -r requirements.txt
# Development dependencies (adds pytest)
pip install -r requirements-dev.txt
```
If RamaLama is not packaged for your Fedora release, it can also be installed with `pip install ramalama`.
See https://github.com/containers/ramalama for other installation options.
## Data staging (this is the current focus of the project)
The corpus is **not** committed.
It is a build artifact, rebuilt from `data/sources.yaml` by a script.
This keeps the repo lean and the pipeline reproducible across environments ([12-Factor App](https://12factor.net/), factor III: config in the environment).
```bash
# List declared sources without fetching
python scripts/fetch_sources.py --list
# Fetch a single source
python scripts/fetch_sources.py --source guidelines
# Fetch everything and print a corpus summary
bash scripts/stage_corpus.sh
```
Source URLs live in `sources.yaml` but can be overridden per source with an environment variable `PGR_SOURCE_<KEY_UPPER>_URL`, so locations stay out of the code.
Every run rewrites `data/MANIFEST.json` for traceability.
## Build and run
The RAG flow is two steps.
First, **build** an OCI image from the staged corpus.
Then, **run** queries against it.
```bash
# Build the RAG vector-store image from the staged corpus
ramalama rag --chunk-size N data/source quay.io/gtfrans2re/packaging-guide-ramalama
# Run a query (prompt is passed via stdin)
ramalama run --rag quay.io/gtfrans2re/packaging-guide-ramalama <model>
```
The full pipeline (fetch -> stage -> build) is documented in [docs/GUIDE.md](docs/GUIDE.md).
> **Important - RamaLama RAG API:** the flags `--model`, `--corpus`, and `--query` do **not** exist.
> Building uses `ramalama rag --chunk-size N <corpus_dir> <oci_image>`.
> Querying uses `ramalama run --rag <oci_image> <model>` with the prompt on stdin.
> Do not invent flags.
> **Chunk size:** the right value is still being determined for this corpus.
> Unlike the editorial mini-project (which used flattened article HTML and needed a very small chunk size), this corpus mixes AsciiDoc, Markdown, PDF, HTML, and raw spec files.
> Mentors have asked to evaluate smaller models with larger chunk sizes (256 or 512) once GPU access is available, since a corpus of strictly RPM guidelines may benefit from larger chunks.
> Always test a build end to end after changing the chunk size.
## Code style
- **Python:** follow PEP 8, and keep functions small and readable.
This is a small project, so prefer clarity over cleverness.
- **Shell scripts:** POSIX-friendly bash, and quote variables.
- **Prose in Markdown and AsciiDoc:** use the One Sentence Per Line convention.
Put each sentence on its own line rather than wrapping prose at a fixed column width.
Rendered output is unaffected (a single line break does not show up to the reader), but git diffs become far easier to review because a change to one sentence does not reflow the rest of the paragraph.
Two consecutive line breaks still start a new paragraph.
- **All files:** use plain ASCII only.
Do not introduce ambiguous Unicode characters such as en dashes, em dashes, curly quotes, or arrows.
Use `-` for dashes, straight quotes, and `->` for arrows.
The Forge review tooling flags these.
To scan a file for ambiguous Unicode before committing:
```bash
python3 -c "
import sys
with open(sys.argv[1], encoding='utf-8') as f:
for i, line in enumerate(f, 1):
for ch in line:
if ord(ch) > 127:
print(f'Line {i}: {repr(ch)} (U+{ord(ch):04X})')
" AGENTS.md
```
If it prints nothing, the file is clean (pure ASCII).
## Testing and verification
The pipeline has a pytest suite under `tests/`.
Every network and git interaction is mocked, so the suite runs airgapped with no access to Forge, GitHub, dist-git, or rpm.org.
Run it before finishing any change to the fetch pipeline:
```bash
# Install dev dependencies (includes pytest)
pip install -r requirements-dev.txt
# Run the suite (airgapped, no network needed)
pytest -q
```
Also verify what is relevant to your change:
```bash
# Confirm the source list parses and lists correctly
python scripts/fetch_sources.py --list
# Confirm the staging pipeline runs end to end (needs network)
bash scripts/stage_corpus.sh
# Confirm the manifest was written
cat data/MANIFEST.json
```
When you change `fetch_sources.py`, add or update a corresponding test.
Test-driven development here is deliberate: it keeps AI-assisted changes honest and guards the pipeline against regressions.
Scan any changed file for ambiguous Unicode before committing (see Code style above).
## Contribution workflow (agents must follow this)
- **Never push directly to `main`.**
Create a branch and open a pull request.
- **Sign off commits** with DCO (`git commit -s`).
This is encouraged as a best practice, though the FPCA already provides the legal basis.
- **Disclose AI use.**
Add an `Assisted-by: <model> <version>` trailer to commits per the [Fedora AI-Assisted Contributions Policy](https://communityblog.fedoraproject.org/council-policy-proposal-policy-on-ai-assisted-contributions/).
- **Authentication:** push over HTTPS following the [Fedora Forge HTTP auth docs](https://docs.fedoraproject.org/en-US/forge-documentation/using_http_auth/).
- **Branch naming:** `feat/...`, `fix/...`, `docs/...`, `chore/...`.
- **Commit messages:** imperative mood, with a concise summary line.
- **One logical change per PR.**
Reference related issues (e.g. `Closes #123`).
## Security and safety notes
- Do **not** commit secrets, API tokens, or Quay/robot credentials.
- The published Quay images are public, so do not embed private data in the corpus.
- The corpus is built from public, appropriately licensed sources only.
Each source's license is recorded in `sources.yaml`; do not add sources whose licensing is unclear.
- Keep changes to the RAG config (chunk size, model strings) deliberate and tested, since they directly affect output quality.
## Good first pointers for agents
- `data/sources.yaml` is the heart of the data-prep stage.
Adding or adjusting a source is usually an edit there plus, if it is a new source type, a matching fetcher in `scripts/fetch_sources.py`.
- When editing docs, keep repo URLs pointing at `forge.fedoraproject.org/ai-ml/packaging-guide-ramalama`.
Leave upstream links (RamaLama, Docling, rpm.org) and Quay image paths as they are unless explicitly migrating them.
## Remember
The Fedora Packaging Guidelines are the heaviest-weight source in the corpus and must stay that way unless mentors decide otherwise.

179
docs/CONTRIBUTING.md Normal file
View file

@ -0,0 +1,179 @@
# Contributing to packaging-guide-ramalama
Thank you for your interest in contributing!
This project is an AI assistant that helps identify deviations from the Fedora RPM Packaging Guidelines, and contributions of all kinds are welcome: bug reports, documentation improvements, code, data-source suggestions, and ideas.
This guide explains *how we work together* on this project.
For technical setup and build details, see [README.md](../README.md) and [docs/GUIDE.md](GUIDE.md).
For guidance aimed at AI coding agents, see [AGENTS.md](../AGENTS.md).
---
## Table of Contents
- [Code of Conduct](#code-of-conduct)
- [Getting Started](#getting-started)
- [Branching and Pull Request Workflow](#branching-and-pull-request-workflow)
- [Data Sources and the Corpus](#data-sources-and-the-corpus)
- [AI-Assisted Contributions](#ai-assisted-contributions)
- [Commit Message Guidelines](#commit-message-guidelines)
- [Developer Certificate of Origin (DCO)](#developer-certificate-of-origin-dco)
- [Review Process](#review-process)
- [Reporting Issues](#reporting-issues)
---
## Code of Conduct
This project follows the [Fedora Code of Conduct](https://docs.fedoraproject.org/en-US/project/code-of-conduct/).
All contributors are expected to abide by it in all project spaces, both in how they treat others and in what they contribute.
Because this is a RAG tool, the content we add to the corpus matters too.
Any material used for the corpus or for training should be legal and ethical for us to use, and should align with our Code of Conduct values.
We avoid introducing harmful, biased, or improperly licensed content into the data.
---
## Getting Started
1. **Fork** the repository on the Fedora Forge.
2. **Clone** your fork locally:
```bash
git clone https://forge.fedoraproject.org/<your-username>/packaging-guide-ramalama
cd packaging-guide-ramalama
```
3. Set the upstream remote:
```bash
git remote add upstream https://forge.fedoraproject.org/ai-ml/packaging-guide-ramalama
```
4. Install dependencies and follow the setup steps in [docs/GUIDE.md](GUIDE.md).
For authenticating with the Fedora Forge over HTTPS, follow the upstream guide:
[Using HTTP auth on the Fedora Forge](https://docs.fedoraproject.org/en-US/forge-documentation/using_http_auth/).
---
## Branching and Pull Request Workflow
Please do not push directly to the `main` branch.
All changes go through a branch and a pull request, so they can be reviewed before merging.
1. Sync your fork with upstream.
2. Create a descriptive branch (for example, `docs/contributing-cleanup` or `feat/data-prep-staging`).
3. Make your changes and commit them.
4. Push your branch to your fork.
5. Open a pull request against `main` on the Forge.
Keep each pull request focused on one logical change.
This makes review easier and faster.
---
## Data Sources and the Corpus
This project builds its RAG corpus from public, appropriately licensed sources, not from files committed to the repo.
The [Fedora Packaging Guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/) are the primary and heaviest-weight source; everything else supports them.
If you want to add or change a source:
1. Edit `data/sources.yaml`, which is the single source of truth for the corpus.
Fill in every field, including `license` and a `notes` rationale explaining why the source belongs in the dataset.
2. If the source needs a new fetch mechanism (a type not already handled), add a matching fetcher in `scripts/fetch_sources.py`.
3. Add or update a pytest test for any new or changed fetch behavior.
4. Run `bash scripts/stage_corpus.sh` and confirm the source stages cleanly and appears in `data/MANIFEST.json`.
Please do not commit fetched documents (the contents of `data/raw/` and `data/source/`).
They are build artifacts and are gitignored.
---
## AI-Assisted Contributions
You may use AI assistance when contributing to this project, in line with the [Fedora AI-Assisted Contributions Policy](https://communityblog.fedoraproject.org/council-policy-proposal-policy-on-ai-assisted-contributions/).
Two things matter most:
- **Accountability.**
You are always the author and are fully accountable for your contribution, including its quality, licensing, and correctness.
AI is a tool, not the author.
- **Transparency.**
When a significant part of a contribution comes from an AI tool, you must disclose it.
The recommended method is an `Assisted-by:` trailer in the commit message, and you may also note it in the pull request description or the document itself.
Example commit trailer:
```
Assisted-by: Claude Opus 4.8
```
Routine use of AI for grammar, spelling, or minor language clarification does not require disclosure.
This exemption only covers superficial language help.
Once AI meaningfully shapes the substance of a contribution (its code, its structure, or its wording), that crosses the line into significant use and should be disclosed with an `Assisted-by:` trailer.
When in doubt, disclose.
---
## Commit Message Guidelines
The commit history is our reasoning audit trail.
A good commit message describes *what* changed; an excellent one explains *why* the change was made.
More context is better than less, especially as AI tools take on more of the routine summarizing.
**Subject line:** `<gitmoji> <component>: <summary>`
- **Gitmoji** - a single emoji from [gitmoji.dev](https://gitmoji.dev/) categorizing the change type.
- **Component scope** - a thematic name grouping the change (for example, `docs`, `ci`, `scripts/fetch`, `data/sources`).
Parenthetical sub-scopes may be used for hierarchy.
- **Summary** - a concise description of what changed, in the imperative mood.
**Body:** explain the reasoning, context, and *why* behind the change.
Connect it to the problem it solves, note any trade-offs or alternatives considered, and include reference links where relevant.
**Trailers:** include the following where applicable.
- `Assisted-by: <model> <version>` when AI tools were used, per the Fedora AI-Assisted Contributions Policy.
- `Signed-off-by:` via DCO sign-off (`git commit --signoff`), encouraged as a best practice (see below).
- GPG-signed commits (`git commit --gpg-sign`) are also encouraged.
---
## Developer Certificate of Origin (DCO)
We do not strictly enforce DCO sign-off in this project.
The [Fedora Project Contributor Agreement (FPCA)](https://docs.fedoraproject.org/en-US/legal/fpca/) already provides the legal basis it would otherwise cover.
However, we highly encourage a DCO sign-off as a best practice.
Our work may not always be presented in a Fedora context, and being explicit is better than being implicit.
Add a sign-off with the `-s` flag:
```bash
git commit -s -m "your commit message"
```
This appends a `Signed-off-by:` line certifying that you have the right to submit the contribution under the project's license, per the [Developer Certificate of Origin (v1.1)](https://developercertificate.org/).
---
## Review Process
A single approved review is sufficient for a pull request to be merged.
Any of the project maintainers can review and approve:
- [@dominikkawka](https://forge.fedoraproject.org/dominikkawka)
- [@cybette](https://forge.fedoraproject.org/cybette)
- [@jflory7](https://forge.fedoraproject.org/jflory7)
When you open a pull request, write a clear description that explains *why* the change is being made, not just what it does.
Reference any related issue (for example, `Closes #123`), respond to reviewer feedback, and push updates to the same branch.
---
## Reporting Issues
Found a bug or have a feature request?
Open an issue on the [Forge issue tracker](https://forge.fedoraproject.org/ai-ml/packaging-guide-ramalama/issues).
The repository provides issue templates (bug report and feature request) to guide you through the details we need.
Please use the relevant template when opening an issue.
---

392
docs/GUIDE.md Normal file
View file

@ -0,0 +1,392 @@
# Complete Pipeline Guide - packaging-guide-ramalama
A step-by-step guide to run the pipeline from a fresh clone through to a built RAG vector store for the Fedora RPM Packaging Guidelines.
This project reuses the architecture validated in the editorial mini-project (`editorial-guide-ramalama`).
A script stages a corpus from public sources, and `ramalama rag` ingests it (running Docling internally) to build an OCI image containing the vector database.
> **Where this guide lives.**
> This is an end-user guide for replicating the pipeline.
> It currently lives in the repo for convenience while the project is under active development.
> The plan is to move it to the AI/ML SIG docs site, in a new section for the RamaLama experimentation projects, once it stabilizes.
> **Status note.**
> The data-preparation stage (Steps 0-5) is implemented and working end to end.
> The model-selection and build parameters (Steps 6-8) are being finalized.
> Mentors have asked to evaluate smaller models with larger chunk sizes once GPU access is provisioned, so the exact model strings and `--chunk-size` value below are starting points, not final values.
> **Setup approach (follow-up).**
> This guide currently documents a local Python virtual-environment setup with per-OS notes.
> Standardizing on a Linux container image is being considered as a follow-up, which would simplify setup and remove the cross-OS matrix.
---
## Prerequisites
| Tool | Min Version | Check |
|------|-------------|-------|
| Python | 3.10+ | `python --version` |
| Git | any | `git --version` |
| pip | 23+ | `pip --version` |
| RamaLama | latest | `ramalama --version` |
| Podman | latest | `podman --version` |
### Install RamaLama (if not already installed)
```bash
# On Fedora / RHEL-based systems
sudo dnf install python3 python3-pip git podman ramalama
# Or install RamaLama via pip
pip install ramalama
```
> Full RamaLama installation options: https://github.com/containers/ramalama
---
## Step 0 - Clone the Repo
```bash
git clone https://forge.fedoraproject.org/gtfrans2re/packaging-guide-ramalama.git
cd packaging-guide-ramalama
git remote add upstream https://forge.fedoraproject.org/ai-ml/packaging-guide-ramalama.git
```
Confirm the structure looks right:
```bash
ls -la
```
Expected top-level contents:
~~~
AGENTS.md
data/
docs/
scripts/
tests/
.env.example
.gitignore
LICENSE
README.md
requirements.txt
requirements-dev.txt
~~~
---
## Step 1 - Set Up the Python Environment
### 1.1 Create a virtual environment
```bash
python -m venv venv
```
### 1.2 Activate it
```bash
# Linux / macOS
source venv/bin/activate
# Windows
venv\Scripts\activate
```
### 1.3 Upgrade pip and install dependencies
```bash
pip install --upgrade pip
pip install -r requirements.txt
```
### 1.4 Verify key packages
```bash
python -c "import requests; print('requests ok')"
python -c "import yaml; print('pyyaml ok')"
python -c "import dotenv; print('dotenv ok')"
python -c "import tqdm; print('tqdm ok')"
```
---
## Step 2 - Configure Environment Variables
The pipeline reads configuration from the environment (12-Factor App, factor III) so source locations are never hard-coded.
### 2.1 Create your `.env` file
```bash
cp .env.example .env
```
The defaults in `data/sources.yaml` work out of the box, so `.env` can stay empty.
Uncomment a line in `.env` only if you need to override a source, for example to point the guidelines at a fork:
```env
PGR_SOURCE_GUIDELINES_URL=https://forge.fedoraproject.org/<you>/guidelines.git
```
`.env` is gitignored and must never be committed.
---
## Step 3 - Review the Source List
`data/sources.yaml` is the single source of truth for the corpus.
Review it before fetching:
```bash
python scripts/fetch_sources.py --list
```
Expected output:
~~~
Sources declared in sources.yaml:
[guidelines ] heavy enabled Fedora Packaging Guidelines (AsciiDoc source)
[rpm_guide ] medium enabled RPM Packaging Guide (AsciiDoc source)
[rpm_docs ] medium enabled RPM in-tree Reference Manual (rpm-software-management/rpm)
[rpm_web ] light enabled RPM reference documentation landing page (rpm.org)
[spec_examples ] medium enabled Real-world RPM spec files (src.fedoraproject.org)
~~~
The sources, and why each is included:
| Source | Weight | Notes |
|--------|--------|-------|
| Fedora Packaging Guidelines | heavy | The primary source: 84 AsciiDoc pages, git-cloned so the manifest records the exact commit. Hosted on Fedora Forge after the 2025 Git Forge migration. |
| RPM Packaging Guide | medium | Structured walkthrough (Adam Miller et al.), organized around C, Python, and shell examples. We clone the AsciiDoc source repo so provenance is captured and Docling reads clean markup. |
| RPM in-tree Reference Manual | medium | The maintained technical docs from `rpm-software-management/rpm`, staged from the repo's `docs/` directory (Markdown manual plus man-page sources). |
| rpm.org landing page | light | A lightweight rendered companion to the in-tree manual. |
| Real-world spec files | medium | Example `.spec` files from `src.fedoraproject.org`, spanning simple to complex across Python, C, and shell. Fetches are validated (404 -> missing, non-spec response -> anomaly). |
> **Language scope:** the dataset is scoped to Python, C, and shell, the foundational languages for OS-level development in Fedora.
---
## Step 4 - Stage the Corpus
The staging script fetches every enabled source into `data/source/<key>/` and writes a provenance record to `data/MANIFEST.json`.
### 4.1 Fetch everything
```bash
bash scripts/stage_corpus.sh
```
### 4.2 Or fetch a single source
```bash
python scripts/fetch_sources.py --source guidelines
```
### 4.3 What to expect
~~~
============================================================
Fetching [guidelines] Fedora Packaging Guidelines (AsciiDoc source) (weight=heavy)
============================================================
Cloning into 'data/raw/guidelines/repo'...
Staged 84 file(s) -> data/source/guidelines/
...
Corpus summary (data/source/):
guidelines 84 files
rpm_docs N files
rpm_guide N files
rpm_web 1 files
spec_examples 6 files
Provenance: data/MANIFEST.json
~~~
### 4.4 Verify the corpus and provenance
```bash
# Count staged files per source
find data/source -type f | awk -F/ '{print $3}' | sort | uniq -c
# Inspect the manifest (URLs, timestamps, git commit hashes)
cat data/MANIFEST.json | python -m json.tool | head -40
```
### Troubleshooting
| Error | Cause | Fix |
|-------|-------|-----|
| `guidelines` stages 0 files | Wrong URL, or cloned an empty stub | Confirm the Forge URL in `sources.yaml`; the old `pagure.io/packaging-committee` repo is a decommissioned stub |
| A spec package 404s | Package retired or renamed in rawhide | Recorded under `missing` in the manifest; swap the name in `sources.yaml` |
| A spec is recorded as an anomaly | Name/spec mismatch or a meta package | Expected safeguard; the response did not look like a spec, so it was not staged |
| `ConnectionError` | No internet / blocked domain | Check connectivity to `forge.fedoraproject.org`, `github.com`, `src.fedoraproject.org` |
| SSH clone rejected | SSH is disabled on the Forge instance | Clone/push over HTTPS with a Forge API token |
---
## Step 5 - Understand the Corpus Layout
There is **no separate Docling ingestion step**.
`ramalama rag` runs Docling internally when it builds the vector store, and Docling parses each source format (AsciiDoc, Markdown, PDF, HTML, spec files) directly.
So the corpus is simply the staged source files:
~~~
data/source/
guidelines/ 84 AsciiDoc pages (flattened filenames preserving structure)
rpm_guide/ RPM Packaging Guide AsciiDoc pages
rpm_docs/ RPM in-tree Reference Manual (Markdown)
rpm_web/ rpm.org landing page (HTML)
spec_examples/ example .spec files, grouped by language
~~~
That whole directory tree is what `ramalama rag` ingests in one build.
### Known limitations (tracked for follow-up)
- **rpm.org landing page** (`rpm_web`) is a single rendered page, kept only as a light companion.
The authoritative technical depth now comes from the cloned in-tree Reference Manual (`rpm_docs`), so this is intentional rather than a gap.
- **Spec examples** use the reviewer-endorsed curated set.
It can be expanded later based on model performance.
---
## Step 5b - Run the Tests (optional but recommended)
The fetch pipeline has a pytest suite.
Every network and git interaction is mocked, so it runs airgapped with no external access.
```bash
pip install -r requirements-dev.txt
pytest -q
```
All tests should pass.
If you change `scripts/fetch_sources.py`, add or update a matching test in `tests/`.
---
## Step 6 - Pull a Model
Use HuggingFace GGUF format.
Start with a small model so you can validate the build on modest hardware before scaling up on the server.
```bash
# A small model for local iteration (example)
ramalama pull hf://ibm-granite/granite-4.0-micro-GGUF
ramalama list
```
> **Hardware note.**
> Training-scale work needs a dedicated GPU (mentors target 24-32 GB VRAM; 12 GB can suffice for 1B-4B models).
> Local iteration with a small model and a small context window is enough to verify the pipeline works end to end before moving to the GPU host.
---
## Step 7 - Build the RAG Vector Store
Build an OCI image from the staged corpus.
Use your own Quay namespace for testing; org-owned "releases" under the Fedora namespace come later.
```bash
ramalama rag --chunk-size 512 data/source quay.io/gtfrans2re/packaging-guide-ramalama
```
> **Chunk size is under evaluation.**
> For a corpus of strictly RPM guidelines, mentors have suggested larger chunk sizes (256 or 512) may improve retrieval, and want this evaluated alongside smaller models once GPU access is available.
> Test any change to `--chunk-size` with a full end-to-end build.
> If a build overflows the embedder's token limit on an atomic block (a large table or code block that Docling will not split), lower the chunk size and rebuild.
### Verify the image
```bash
podman images | grep packaging-guide-ramalama
```
---
## Step 8 - Query the RAG Store
```bash
ramalama run --rag quay.io/gtfrans2re/packaging-guide-ramalama hf://ibm-granite/granite-4.0-micro-GGUF
```
This opens an interactive prompt.
Try a packaging question, for example:
~~~
What are the requirements for the %files section in a Fedora spec file?
~~~
~~~
Review this spec file against the Fedora Packaging Guidelines and flag any deviations:
[paste spec content here]
~~~
Press `Ctrl+D` to exit.
> Project success is measured by the model's ability to identify and reason about RPM spec files against the guidelines.
> Evaluation (RAGAS vs. DeepEval) is planned for the next sprint.
---
## Full Pipeline - Quick Reference
```bash
# Activate environment
source venv/bin/activate
# 1. Configure (defaults work; .env only for overrides)
cp .env.example .env
# 2. Review the source list
python scripts/fetch_sources.py --list
# 3. Stage the corpus (fetch all sources, write MANIFEST.json)
bash scripts/stage_corpus.sh
# 4. (Optional) run the tests
pip install -r requirements-dev.txt
pytest -q
# 5. Pull a small model (first time only)
ramalama pull hf://ibm-granite/granite-4.0-micro-GGUF
# 6. Build the RAG vector store from the staged corpus
ramalama rag --chunk-size 512 data/source quay.io/gtfrans2re/packaging-guide-ramalama
# 7. Verify the image
podman images | grep packaging-guide-ramalama
# 8. Query it
ramalama run --rag quay.io/gtfrans2re/packaging-guide-ramalama hf://ibm-granite/granite-4.0-micro-GGUF
```
---
## Getting Help
| Channel | Use for |
|---------|---------|
| Matrix / Element (#ai-ml:fedoraproject.org) | Daily stand-ups, quick blockers |
| Forge Issues | Bug reports, feature requests |
| Francois Gonothi Toure (@gtfrans2re) | Intern; pipeline, data prep, and this guide |
| Justin Wheeler (@jflory7) | Coordination, packaging sources, RamaLama questions |
| Dominik Kawka (@dominikkawka) | GPU access, tech stack advice |
| Carol Chen (@cybette) | Sprint process, scope questions |
---
## References
- Fedora Packaging Guidelines: https://docs.fedoraproject.org/en-US/packaging-guidelines/
- RPM Packaging Guide: https://rpm-packaging-guide.github.io/
- RPM man pages: https://rpm.org/docs/latest/man/
- rpm-software-management/rpm: https://github.com/rpm-software-management/rpm
- Building your first RPM package in Fedora: https://jwheel.org/blog/2017/11/first-rpm-package-fedora/
- src.fedoraproject.org: https://src.fedoraproject.org/
- Docling: https://github.com/docling-project/docling
- 12-Factor App: https://12factor.net/

BIN
scripts/__pycache__/fetch_sources.cpython-314.pyc Normal file
View file

Binary file not shown.

BIN
tests/__pycache__/conftest.cpython-314-pytest-9.1.1.pyc Normal file
View file

Binary file not shown.

BIN
tests/__pycache__/test_fetch_sources.cpython-314-pytest-9.1.1.pyc Normal file
View file

Binary file not shown.