[Gonothi] Start building out the RAG pipeline for Fedora RPM packaging #160
Labels
No labels
state
ready for review
function
analytics
function
communications
function
docs
function
events
function
onboarding
function
operations
good first issue
help wanted
needs
changes
needs
reporter feedback
needs
team vote
needs
triage
scope
bug
scope
improvement
scope
new
state
approved
state
blocked
state
duplicate
state
invalid
state
wontfix
No milestone
No project
No assignees
3 participants
Notifications
Due date
Dependencies
No dependencies set.
Reference
commops/interns#160
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Using Fedora RPM packaging guidelines to build out the RAG pipeline with RamaLama for the main goal of the Outreachy project
Background
Most of the tasks from the previous sprints have been paving the paths to this main part of the project. Here's where we put it all together. Depending on the progress, some of the tasks can be carried forward to the next sprint.
Deliverables
Not an exhausive list, please add to this as needed. Also listed in no particular order, please implement them in the order you deem fit.
ramalama rag(containing the vectordb) and those will be pushed to https://quay.io/repository/fedora/packaging-guide-ramalama under the Fedora namespace (from #149)Outcome
One step closer to the finish line
@gtfrans2re, as Sprint 9 kicks off on Monday, could you post brief daily updates on this issue? A few bullet points at the start or end of each day is great — we want to stay close to your progress, especially during the data prep stage, which involves a lot of thoughtful curation decisions.
I'd also like to ask for an end-of-sprint update by Friday, July 3rd covering:
This issue will be moved to your Sprint 9 project board on Monday.
Assisted-by: Claude Opus 4.6 (1M context)
Data sources and staging plan for the RAG pipeline
Here are some starting points for data prep. Not all of these may end up being useful, but they're worth exploring as potential sources for the packaging guidelines RAG pipeline:
rpmbuild,rpmspec,rpmsign, etc.).docs/directory and README may contain useful reference material beyond what's in the man pages.Language scope: We agreed to focus the training data on Python, C, and shell scripting — the foundational languages for OS-level development in Fedora.
Data staging methodology: Rather than copying static files into the repo, the data prep should be driven by a script that fetches and processes sources. This keeps the repo lean and the pipeline reproducible across environments (see 12-Factor App methodology). Docling can handle much of the data cleaning and processing of various source formats.
End-of-sprint deliverable: @gtfrans2re, please aim to open a first-draft PR on packaging-guide-ramalama with the data prep script and annotated source list by Friday.
On model selection: For the initial data prep and pipeline testing, consider using one of the smallest available models so you can iterate locally without needing GPU hardware. The goal at this stage is to verify the process works end-to-end — ingestion, chunking, vector store creation, and basic retrieval. Once the pipeline is validated locally, we can scale up to a larger model on the server.
Assisted-by: Claude Opus 4.6 (1M context)
Well noted with thanks, @jflory7. I will deliver this tomorrow following your instructions.
Hi @jflory7, here's my end-of-sprint update ahead of Sprint 9 kickoff. I've opened the first-draft data-prep PR on
packaging-guide-ramalama— https://forge.fedoraproject.org/ai-ml/packaging-guide-ramalama/pulls/2.Overall progress so far
The data-prep staging pipeline is built and working end-to-end against all four initial sources. Rather than committing static files, staging is driven by a script (
scripts/fetch_sources.py) that reads an annotated source list (data/sources.yaml) and fetches each source intodata/source/forramalama ragto ingest, following the 12-Factor approach we discussed. The fetched corpus is a rebuildable artifact and is gitignored, so the repo stays lean. Every run writes adata/MANIFEST.jsonrecording each source's resolved URL, retrieval timestamp, and (for git sources) the exact commit hash, so any corpus build is fully reproducible.Current state of work
All four sources stage successfully:
src.fedoraproject.org— a placeholder curated set spanning simple → complex across Python, C, and shell (python-requests,python-setuptools,systemd,bash-completion,ShellCheck).One finding worth flagging: the guidelines source has moved as part of the 2025 Git Forge migration. The old
pagure.io/packaging-committeerepo is now a decommissioned stub, so I've pointed the pipeline at the Forge home (forge.fedoraproject.org/packaging/guidelines), which matches the path from our meeting notes. Source URLs are also env-overridable (PGR_SOURCE_<KEY>_URL), so we're not hard-coding locations.Next steps
CONTRIBUTING.md+ DCO sign-off andAGENTS.md(planned as a separate governance PR per my sync with @dominikkawka).Assisted-by: Claude Opus 4.8
Follow-up: opened a second PR adding project documentation — https://forge.fedoraproject.org/ai-ml/packaging-guide-ramalama/pulls/3.
docs/GUIDE.md- step-by-step run guide (data-prep steps reflect the working pipeline; build/query steps marked as pending model selection and GPU access)docs/CONTRIBUTING.md- branch/PR workflow, DCO sign-off, and AI-disclosure policyAGENTS.md- guidance for AI coding agentsKept as a separate PR from the data-prep script so governance/docs can be reviewed independently. All files are pure-ASCII per the Forge tooling.
Sprint 9 kickoff — responding to your end-of-sprint update
Thanks for the update and for getting both PRs opened before the sprint boundary, Gonothi.
Sample packages: The current set in
sources.yamllooks like a reasonable starting point. The mix of simple and complex packages across Python, C, and shell gives good variety:python-requests(simple),python-setuptools(moderate)zlib(simple),systemd(complex stress test)bash-completion(simple),ShellCheck(moderate)Go ahead and use these — no need to wait on a separate list from me. We can always expand the set later based on model performance.
PR reviews: I've reviewed both PRs and left feedback requesting changes on each. @dominikkawka, I'd appreciate your help reviewing these as well, since you're deeper in the technical details of the pipeline than I am:
pytestunit tests with mocked network responses so tests can run airgapped. Also flagged some questions about data source scope and error handling in the inline comments.Data sources not yet covered: Your pipeline covers four of the six sources I listed in my earlier comment. Two that haven't been picked up yet:
docs/directoryThese aren't urgent — focus on addressing the PR review feedback first, and we can evaluate adding more sources based on model performance.
RAGAS vs DeepEval: You mentioned you'll bring a recommendation between the two. Could you post a brief summary of your findings sometime this week? Even a few bullet points comparing the two would be helpful as we get into sprint planning for Sprint 9.
Blockers update:
Sprint 9 priorities: Address the PR review feedback, get both PRs merged, and start iterating on the pipeline locally with a small model. Please continue the daily updates on this issue.
Assisted-by: Claude Opus 4.6 (1M context)
Decision: RAG evaluation tool for the mini-project
I built a small local evaluation harness (PR #10) and ran both RAGAS and DeepEval against the pipeline with a local judge model (Granite 4 Micro via
ramalama serve), scoring the same pipeline outputs with each tool for a fair comparison.Results (3-sample run, same outputs, same Granite judge)
RAGAS scored higher (0.969 vs 0.889), though both are high and agree closely - two independent methods both rate the pipeline's answers as strongly relevant. Absolute scores aren't 1:1 comparable across tools (different internals), so I treat them as directional. Only answer relevancy is scored so far; faithfulness / context precision / recall need retrieved-context capture, which is a documented next step.
Memory exhaustion during the runs
Running this locally was constrained by hardware. On my laptop, loading the RAG pipeline model and a judge model at the same time repeatedly exhausted memory - free RAM dropped to a few hundred MB and swap filled to ~90%, causing the pipeline to thrash and stall (queries stuck at 1/10). I had to kill leftover
ramalama/llama-servercontainers to reclaim memory, run the pipeline and judge in separate phases, use the lighter Granite model, and limit runs to 3 samples. This is the same memory ceiling behind the CRC and GPU-instance work (#160/#163); a fuller quantitative run is best done once the GPU instance is available.Setup experience - the deciding factor
Despite RAGAS scoring slightly higher, DeepEval was far easier to set up, use, and evaluate with:
RAGAS took roughly four separate fixes to produce a number (3.12 venv, dependency pinning, a separate embeddings model, and a result-parsing correction). DeepEval ran cleanly the first time.
Recommendation
Adopt DeepEval as the primary RAG-evaluation tool for the mini-project. The scores validate the pipeline (both tools agree it performs well), but the setup experience is decisive: DeepEval ran out of the box on our actual stack while RAGAS required substantial workarounds. For a project that values reproducibility and low friction, that matters more than RAGAS's slightly higher score. DeepEval's Pytest-style structure also fits CI/CD, and G-Eval supports our bespoke editorial checks (tone, featured image, Read More tag).
RAGAS remains worth revisiting for its RAG-native metrics once it supports Python 3.14 (or via a pinned 3.12 environment), especially after we capture retrieved contexts.
Next sprint (per #154)
In short: RAGAS scored marginally higher (0.969 vs 0.889), but DeepEval was much easier to set up and use and ran cleanly on our stack where RAGAS needed extensive workarounds - so DeepEval is the practical choice.
Assisted-by: Claude Opus 4.8
@jflory7 wrote in #160 (comment):
Thanks @jflory7 - both of those sources are now covered:
rpm-software-management/rpmdocs/- this actually landed in the PR #2 rework as therpm_docssource. It clones the RPM repo and stages thedocs/subdirectory (the in-tree Reference Manual and man-page sources), which also replaces the earlier shallow rpm.org fetch. It may have been merged after your comment.first_rpm_walkthroughsource (light weight,http_docstype). It captures the learner's perspective alongside the reference material, as you suggested.That brings us to all six of the sources from your earlier comment. One small thing to confirm: since the walkthrough is a personal blog post with no stated open license, I've marked its license as
all-rights-reservedinsources.yamlpending your confirmation that it's fine to ingest. Happy to adjust that field to whatever you'd prefer.@jflory7 wrote in #160 (comment):
Thanks for the updates, @jflory7.
Quay (#163): Good news, this is sorted on my end now. I was able to push to the Fedora namespace, and the mini-project images are up at https://quay.io/repository/fedora/editorial-guide-ramalama?tab=tags. So the Write access is working. Once we have a built RAG image for the packaging project, I'll push those "releases" to the
fedora/packaging-guide-ramalamanamespace the same way.GPU (#161): No problem, thanks for chasing @jasonbrooks on the 12-16GB VRAM. I'm unblocked for local iteration with a small model in the meantime, so I'll keep the pipeline work moving and pick up the scaled runs once the instance is provisioned.
Sprint 9 priorities: On track - addressing the PR review feedback from you and @dominikkawka on, and iterating locally. I'll keep posting daily updates here.
@gtfrans2re wrote in #160 (comment):
All content on my blog is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License (
CC BY-NC-SA 4.0). So, in the scope of the license, you are permitted to use it here without requesting permission for this noncommercial purpose.Thanks for confirming, @jflory7. I've updated
sources.yamlto record the license asCC-BY-NC-SA-4.0for thefirst_rpm_walkthroughsource, and noted in the source's rationale that our use here falls within the noncommercial scope of the license.That resolves the last open question on the data sources - all six from your list are now covered with their licenses recorded in the manifest.
Update: RAG pipeline built, released, and improved
Good progress today on the two remaining deliverables.
Quay releases (#149): done. The RAG image is built from the full corpus and pushed to the Fedora namespace at
quay.io/fedora/packaging-guide-ramalama(tagsv0.1.3andlatest), with matching annotated git tags on the repo.Along the way I found and fixed a significant ingestion bug:
ramalama rag(via Docling) silently skips.adocand.specfiles, so an early build over our 139-file corpus only ingested 37 files. Every AsciiDoc guideline page and every spec example was dropped with no error, and the resulting image could not answer basic guideline questions. The fix converts.adocto HTML/Markdown at staging time (via asciidoctor, with a plain-text fallback) and stages.specas.txt, and records the conversion method in the manifest. The full corpus now ingests correctly.Retrieval quality — with a targeted improvement from @jflory7's resources. Thanks for the allowed-licenses and not-allowed-licenses pages. I added both as sources. Before, when asked about the License field, the model recommended legacy short names (GPL, MIT, BSD). After adding the license pages, it now references the SPDX License List directly. It still tends to give descriptive names rather than exact SPDX identifiers (e.g. "Apache License, Version 2.0" instead of
Apache-2.0), which is a grounding limitation of the small model (granite-4.0-micro, 3B) rather than a corpus gap. This is a concrete case for testing a larger model once the GPU (#161) is available.Chunk size finding:
--chunk-size 512overflows the embedder's 512-token input limit (a property of the embedding model, not our hardware), so a larger embedder would be needed to raise it, not a bigger GPU.--chunk-size 256builds cleanly and is what the current release uses.Output type (#150): the implementation is in PR #4 (a shared review layer producing a prioritized, blockers-first issue summary that can expand into a full fedora-review style checklist, delivered via both a CLI and the Streamlit app, with unit tests). I'll do the final end-to-end validation against the live image tomorrow before marking it complete.
Minor:
zlibnow 404s in the spec examples (retired/renamed in rawhide); I'll swap it forlibpngorxz.Assisted-by: Claude Opus 4.8
Sprint wrap-up before break
Quick status on where #160 stands.
Done:
eval/, actual scored evaluation is next-sprint work)quay.io/fedora/packaging-guide-ramalama(v0.1.3, latest), with matching git tagsDeferred (by agreement): the last deliverable, aligning with @jflory7 and packagers on the desired output type and implementing it. The implementation groundwork is already in PR #4 (a prioritized, blockers-first issue summary that can expand into a full fedora-review style checklist, via CLI and the Streamlit app). As discussed with Justin in yesterday's meeting, the best path is to align this with Justin and other packagers directly on what output is most useful, which I'll take up when I return from the break my mentors suggested, in about two weeks.
Carried forward for when I'm back:
zlibspec example forlibpngorxzOverall: the pipeline is working end-to-end and released. The main remaining quality gap is small-model grounding, which is the case for the GPU in #161. Stepping away for the suggested break; picking these up on return.
@gtfrans2re Closing this issue to mark the RAG pipeline deliverable as complete.
What shipped (Sprint 9):
.adoc/.specfiles — converted at staging time)quay.io/fedora/packaging-guide-ramalama(v0.1.3,latest)This work all landed in Sprint 9, before my PTO and your paternity leave in Sprint 10. Now that Sprint 10 is wrapping up, I'm closing this to keep things tidy. The remaining work — output type alignment with packagers, scored RAG evaluation, and the
zlibspec swap — will be tracked as new issues in Sprint 11 when you return next week.Great work getting the pipeline to end-to-end and released before the break. Looking forward to picking up the next phase together.
Assisted-by: Claude Opus 4.6 (1M context)
Thanks for the thorough wrap-up and for closing this out, @jflory7 - the summary captures Sprint 9 well.
Small note for the record: I'm actually back as of this week and already picking things up (just addressed the outstanding review feedback on PRs #2 and #4 today). Happy to carry the remaining items - output-type alignment with packagers, scored RAG evaluation, and the zlib spec swap - into Sprint 11 as new issues.
One good update since the break: the GPU instance (#161) is now provisioned, and I have access (NVIDIA A10G, 24GB), so the scaled model testing that motivates the evaluation work is unblocked. Details are on #161.
Looking forward to the next phase.