feat: add local RAG evaluation harness (RAGAS vs DeepEval) #10

Merged

jflory7 merged 1 commit from gtfrans2re/editorial-guide-ramalama:feat/rag-eval-harness into main

2026-07-11 04:25:28 +00:00

gtfrans2re commented

2026-07-07 19:37:52 +00:00

Collaborator

Summary

Adds a self-contained evaluation harness under eval/ that runs the RAG pipeline over a guideline-grounded eval set and scores the outputs with both RAGAS and DeepEval using a local judge model (no external API). Supports the #154 task of choosing a RAG evaluation tool.

What's included

eval_set.json - QA pairs grounded in the Fedora writing guidelines
run_pipeline.py - runs the RAG pipeline, saves (question, answer, contexts, ground_truth)
score_ragas.py / score_deepeval.py - score with each tool via a local judge
compare.py - side-by-side comparison
README.md - full setup and reproduction steps
Committed 3-sample results for reference

Result (3 samples, Granite judge)

Concept	RAGAS	DeepEval
Answer relevancy	0.969	0.889

Both tools agree closely. DeepEval ran cleanly on the current stack (Python 3.14); RAGAS needed a separate 3.12 venv, dependency pinning, and a local embeddings model. Full decision writeup in #154.

Notes

Only answer relevancy is scored so far; faithfulness / context precision / recall need retrieved-context capture (documented as a next step in the README).
The .deepeval/ cache is gitignored.

_{Assisted-by: Claude Opus 4.8}

## Summary Adds a self-contained evaluation harness under `eval/` that runs the RAG pipeline over a guideline-grounded eval set and scores the outputs with **both RAGAS and DeepEval** using a **local judge model** (no external API). Supports the #154 task of choosing a RAG evaluation tool. ## What's included - `eval_set.json` - QA pairs grounded in the Fedora writing guidelines - `run_pipeline.py` - runs the RAG pipeline, saves (question, answer, contexts, ground_truth) - `score_ragas.py` / `score_deepeval.py` - score with each tool via a local judge - `compare.py` - side-by-side comparison - `README.md` - full setup and reproduction steps - Committed 3-sample results for reference ## Result (3 samples, Granite judge) | Concept | RAGAS | DeepEval | |---------|-------|----------| | Answer relevancy | 0.969 | 0.889 | Both tools agree closely. DeepEval ran cleanly on the current stack (Python 3.14); RAGAS needed a separate 3.12 venv, dependency pinning, and a local embeddings model. Full decision writeup in #154. ## Notes - Only answer relevancy is scored so far; faithfulness / context precision / recall need retrieved-context capture (documented as a next step in the README). - The `.deepeval/` cache is gitignored. <sub>_Assisted-by: Claude Opus 4.8_</sub>

gtfrans2re added 1 commit

2026-07-07 19:37:52 +00:00

feat: add local RAG evaluation harness (RAGAS vs DeepEval) 996c94f08d

Add a self-contained eval harness under eval/ that runs the RAG pipeline
over a guideline-grounded eval set and scores the outputs with BOTH RAGAS
and DeepEval using a local judge model (no external API).

Includes eval_set.json, run_pipeline.py, score_ragas.py, score_deepeval.py,
compare.py, a README, and the 3-sample results (answer relevancy: RAGAS
0.969 vs DeepEval 0.889). Supports the #154 task of choosing a RAG eval
tool - both agree closely; DeepEval ran with far less setup friction on
the current stack.

Assisted-by: Claude Opus 4.8
Signed-off-by: gtfrans2re <francoisgonothitoure@gmail.com>

gtfrans2re referenced this pull request from commops/interns

2026-07-07 19:41:13 +00:00

[Gonothi] Start building out the RAG pipeline for Fedora RPM packaging #160

jflory7

2026-07-11 03:42:16 +00:00

added the
enhancement
label
requested reviews from jflory7, dominikkawka

jflory7 self-assigned this

2026-07-11 03:42:37 +00:00

cybette was assigned by jflory7

2026-07-11 03:42:37 +00:00

dominikkawka was assigned by jflory7

2026-07-11 03:42:38 +00:00

jflory7 force-pushed feat/rag-eval-harness from 996c94f08d to 07f5414890

2026-07-11 04:19:35 +00:00

Compare

jflory7 approved these changes

2026-07-11 04:25:22 +00:00

jflory7 left a comment

I would really like to imagine what this would look like if we were able to leverage CI to run some test cases and get checks whenever new changes are merged.

In the meantime, this looks and provides some more base scaffolding for these kinds of harnesses. I know there are some similar PRs opened on ai-ml/packaging-guide-ramalama, so we should discuss more in-depth there, since this mini-project is going on the backburner for a while.

I am curious to know more about the challenges with the setup with the four files too.

Merging! 🌊

I would really like to imagine what this would look like if we were able to leverage CI to run some test cases and get checks whenever new changes are merged. In the meantime, this looks and provides some more base scaffolding for these kinds of harnesses. I know there are some similar PRs opened on `ai-ml/packaging-guide-ramalama`, so we should discuss more in-depth there, since this mini-project is going on the backburner for a while. I am curious to know more about the challenges with the setup with the four files too. Merging! 🌊

jflory7 merged commit 07f5414890 into main

2026-07-11 04:25:28 +00:00

gtfrans2re referenced this pull request from commops/interns

2026-07-13 17:37:19 +00:00

[Gonothi]: Create v1.0.0 release for editorial-guide-ramalama #171