commops/interns
16
8
Fork 4
Code Issues 8 Pull requests Projects 7 Activity

[Gonothi] Evaluating RAG systems and other tools #154

New issue
Closed
opened 2026-06-09 12:26:32 +00:00 by cybette · 13 comments
cybette commented 2026-06-09 12:26:32 +00:00
Owner
Copy link

Summary

Learn more about different tools that can help with RAG evaluations, finding the right models for your system, CPU/GPU usage statistics, scaling ML workloads etc.

Background

We are building RAG systems for this internship, and we need a consistent way to evaluate the quality and effectiveness of what has been developed. Also explore other tools that may (or may not) help with the project.

Deliverables

This issue is not about implementing/integrating any of the tools yet. It's about exploration and discovery to determine which ones are actually useful, and if so, what needs to be done to include them in the project (the implementation will be carried out in future Sprints).

Share your findings as comments in this issue as you progress through the list.

Summary

Understand the different tools, the value they provide, and integrate those that can positively impact the project.

### Summary Learn more about different tools that can help with RAG evaluations, finding the right models for your system, CPU/GPU usage statistics, scaling ML workloads etc. ### Background We are building RAG systems for this internship, and we need a consistent way to evaluate the quality and effectiveness of what has been developed. Also explore other tools that may (or may not) help with the project. ### Deliverables This issue is not about implementing/integrating any of the tools yet. It's about exploration and discovery to determine which ones are actually useful, and if so, what needs to be done to include them in the project (the implementation will be carried out in future Sprints). Share your findings as comments in this issue as you progress through the list. - [x] Watch "Ragas: Evals for RAG & Memory" https://www.youtube.com/watch?v=JEDzmM-8rrM - [x] Check out https://github.com/vibrantlabsai/ragas - [x] Connect with Hitendri Bomble (contact info shared via email) - [x] Hitendri's presentation deck at PyData London: https://drive.google.com/file/d/18AcC5Jh6HxcHSr8r33Ws37mcpBuikRoA/view - [x] Check out and compare DeepEval with Ragas https://github.com/confident-ai/deepeval - [x] Metrix: https://docs.alliancecan.ca/wiki/Metrix/en#Cloud_statistics - [x] llmfit: https://www.llmfit.org/ - [x] Ray project: https://github.com/ray-project/ray - [x] GPU usage with KubeRay: https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/gpu.html - [x] Discuss with @dominikkawka about the relevalce of Ray for the project - [x] Docs2DB https://github.com/rhel-lightspeed/docs2db compared with doc2rag in RamaLama - [x] Fedora Docs RAG Database built with docs2db https://github.com/lifto/fedoradocsrag - [x] Application proposal for onboarding matrix bot using docs2db + FedoraDocsRAG https://forge.fedoraproject.org/join/Fedora-Join/issues/337 ### Summary Understand the different tools, the value they provide, and integrate those that can positively impact the project.
👀 1
cybette added the due date 2026-06-22 2026-06-09 12:27:35 +00:00
cybette changed title from [Gonothi] Evaluating RAG systems to [Gonothi] Evaluating RAG systems and other tools 2026-06-09 13:41:59 +00:00
gtfrans2re commented 2026-06-11 08:14:49 +00:00
Collaborator
Copy link

Hello, @cybette! I am sharing some progress on exploring the RAG evaluation tools:

"Evals for RAG & Memory" YouTube video — watched it, and it was a helpful primer on why RAG evaluation needs its own approach versus standard model evals. The framing of separating retrieval quality from generation quality was the part that stuck with me — it maps directly onto issues I saw in my own pipeline, where a wrong answer can come either from bad retrieval or from the model mishandling good context.

RAGAS — spent some time exploring this one. It starts off as an evals framework for RAG systems, and it has since grown to also cover evaluating AI agents. It looks like a strong fit for measuring our editorial assistant — metrics like faithfulness and context relevance would let me quantify how well the retrieved guidelines actually ground each review, rather than just eyeballing the outputs as I've been doing so far.

Arize Phoenix (additional tool: https://github.com/sagecodes/ai-build-and-learn/tree/main/topics/arize-phoenix) — I came across this one and think it could be valuable for the project. It's an LLM observability tool that's also relevant for tracing what the model does with the retrieved context, which complements RAGAS nicely (RAGAS for scoring, Phoenix for seeing why). I've registered for a session on it on June 12 and will report back here with what I learn.

Next, I'll look into the remaining items on the list (DeepEval, the Metrik/llmlit references, and Ray) and add findings as I go. Happy to discuss the relevance of Ray for the project with @dominikkawka when there's a good time.

Assisted-by: Claude Opus 4.8

Hello, @cybette! I am sharing some progress on exploring the RAG evaluation tools: **"Evals for RAG & Memory" YouTube video** — watched it, and it was a helpful primer on why RAG evaluation needs its own approach versus standard model evals. The framing of separating retrieval quality from generation quality was the part that stuck with me — it maps directly onto issues I saw in my own pipeline, where a wrong answer can come either from bad retrieval or from the model mishandling good context. **RAGAS** — spent some time exploring this one. It starts off as an evals framework for RAG systems, and it has since grown to also cover evaluating AI agents. It looks like a strong fit for measuring our editorial assistant — metrics like faithfulness and context relevance would let me quantify how well the retrieved guidelines actually ground each review, rather than just eyeballing the outputs as I've been doing so far. **Arize Phoenix (additional tool: [https://github.com/sagecodes/ai-build-and-learn/tree/main/topics/arize-phoenix](https://github.com/sagecodes/ai-build-and-learn/tree/main/topics/arize-phoenix))** — I came across this one and think it could be valuable for the project. It's an LLM observability tool that's also relevant for tracing what the model does with the retrieved context, which complements RAGAS nicely (RAGAS for scoring, Phoenix for seeing *why*). I've registered for a session on it on June 12 and will report back here with what I learn. Next, I'll look into the remaining items on the list (DeepEval, the Metrik/llmlit references, and Ray) and add findings as I go. Happy to discuss the relevance of Ray for the project with @dominikkawka when there's a good time. <sub>_Assisted-by: Claude Opus 4.8_</sub>
👍 1
gtfrans2re commented 2026-06-14 19:06:53 +00:00
Collaborator
Copy link

Summary: PyData London 2026 — "The Silent Crash" (Hitendri Bomble)

I read through this talk as part of the RAG evaluation exploration. It argues that traditional NLP metrics fail at evaluating RAG systems, and proposes a more robust framework built on LLM-as-a-judge plus adversarial testing.

Main points

  • Metric failure: BLEU, ROUGE, BERTScore, and cosine similarity reward surface-level lexical/topic overlap rather than factual correctness or logical entailment — so they miss critical entity swaps that change legal or factual meaning.
  • Evaluation decomposition: RAG performance should be split into independent retrieval metrics (Context Recall / Precision) and generation metrics (Faithfulness, Relevance) to pinpoint where a failure actually occurs.
  • LLM-as-a-judge: Using capable LLMs with G-Eval (chain-of-thought scoring + logprobs) enables semantic judgments of nuanced criteria like factuality and groundedness.
  • Bias mitigation: Judge LLMs suffer from position, verbosity, and self-preference biases, which need mitigations like swapping input order, cross-model judging, and multi-judge ensembles.
  • Adversarial testing: Robust evaluation requires deliberately "hard" test sets using negations, multi-hop reasoning, and counterfactuals to expose edge-case failures.

Section summaries

Section Summary
The Metrics Graveyard Traditional NLP metrics (BLEU, ROUGE, BERTScore) and cosine similarity are inadequate for RAG because they reward word overlap over factual correctness, and often fail to detect entity swaps that change meaning.
Retrieval vs. Generation Failures must be categorized as either retrieval failures (wrong chunks) or generation failures (hallucinations despite correct chunks). Decomposing them lets you debug the specific pipeline component at fault.
LLM-as-a-Judge Replaces pattern matching with semantic understanding. Tools like G-Eval use chain-of-thought reasoning and logprobs for more accurate, continuous scoring of qualitative criteria.
The Judge Paradox LLM judges are more capable but introduce biases (preferring the first option or longer answers). Mitigation includes cross-model judging and calibrating against human evaluators.
Adversarial Test Sets Production data misses edge cases. Build "hard" suites using negation injection, entity swapping, and multi-hop reasoning to test true robustness.
Practical Tooling & Strategy Frameworks like RAGAS and DeepEval provide off-the-shelf metrics and CI/CD integration. Costs stay manageable via prompt caching, batch APIs, and hierarchical evaluation.

Key concepts

Concept Definition
Faithfulness Measures whether every claim in the generated answer is strictly supported by the retrieved context — effectively catching hallucinations.
Context Recall The percentage of required information chunks successfully retrieved from the corpus to answer a query.
G-Eval An LLM-based evaluation framework using chain-of-thought reasoning and probability weighting (logprobs) to produce reliable scores.
Self-Preference Bias An LLM judge's tendency to rate outputs from its own model family higher than competing models.
Multi-hop Reasoning A test technique requiring the model to synthesize facts across multiple retrieved chunks to reach a correct answer.
Cosine Similarity A geometric measure of the angle between two embedding vectors; indicates topic similarity but cannot distinguish true from false statements.

Relevance to our project

This maps directly onto the editorial assistant: the retrieval vs. generation split is exactly the distinction I'd want when debugging why a draft review goes wrong (did it fail to retrieve the right guideline, or did it have the guideline and still misjudge?). Faithfulness and Context Recall are strong candidate metrics, and both RAGAS and DeepEval (already on the eval shortlist) implement them — so this reinforces the direction in the evaluation issue.

Source: PyData London 2026 — "The Silent Crash" (Hitendri Bomble).

Assisted-by: Claude Opus 4.8

## Summary: PyData London 2026 — ["The Silent Crash" (Hitendri Bomble)](https://drive.google.com/file/d/18AcC5Jh6HxcHSr8r33Ws37mcpBuikRoA/view) I read through this talk as part of the RAG evaluation exploration. It argues that **traditional NLP metrics fail at evaluating RAG systems**, and proposes a more robust framework built on **LLM-as-a-judge** plus **adversarial testing**. ### Main points - **Metric failure:** BLEU, ROUGE, BERTScore, and cosine similarity reward surface-level lexical/topic overlap rather than factual correctness or logical entailment — so they miss critical entity swaps that change legal or factual meaning. - **Evaluation decomposition:** RAG performance should be split into independent **retrieval** metrics (Context Recall / Precision) and **generation** metrics (Faithfulness, Relevance) to pinpoint where a failure actually occurs. - **LLM-as-a-judge:** Using capable LLMs with **G-Eval** (chain-of-thought scoring + logprobs) enables semantic judgments of nuanced criteria like factuality and groundedness. - **Bias mitigation:** Judge LLMs suffer from position, verbosity, and self-preference biases, which need mitigations like swapping input order, cross-model judging, and multi-judge ensembles. - **Adversarial testing:** Robust evaluation requires deliberately "hard" test sets using negations, multi-hop reasoning, and counterfactuals to expose edge-case failures. ### Section summaries | Section | Summary | |---------|---------| | **The Metrics Graveyard** | Traditional NLP metrics (BLEU, ROUGE, BERTScore) and cosine similarity are inadequate for RAG because they reward word overlap over factual correctness, and often fail to detect entity swaps that change meaning. | | **Retrieval vs. Generation** | Failures must be categorized as either retrieval failures (wrong chunks) or generation failures (hallucinations despite correct chunks). Decomposing them lets you debug the specific pipeline component at fault. | | **LLM-as-a-Judge** | Replaces pattern matching with semantic understanding. Tools like G-Eval use chain-of-thought reasoning and logprobs for more accurate, continuous scoring of qualitative criteria. | | **The Judge Paradox** | LLM judges are more capable but introduce biases (preferring the first option or longer answers). Mitigation includes cross-model judging and calibrating against human evaluators. | | **Adversarial Test Sets** | Production data misses edge cases. Build "hard" suites using negation injection, entity swapping, and multi-hop reasoning to test true robustness. | | **Practical Tooling & Strategy** | Frameworks like RAGAS and DeepEval provide off-the-shelf metrics and CI/CD integration. Costs stay manageable via prompt caching, batch APIs, and hierarchical evaluation. | ### Key concepts | Concept | Definition | |---------|------------| | **Faithfulness** | Measures whether every claim in the generated answer is strictly supported by the retrieved context — effectively catching hallucinations. | | **Context Recall** | The percentage of required information chunks successfully retrieved from the corpus to answer a query. | | **G-Eval** | An LLM-based evaluation framework using chain-of-thought reasoning and probability weighting (logprobs) to produce reliable scores. | | **Self-Preference Bias** | An LLM judge's tendency to rate outputs from its own model family higher than competing models. | | **Multi-hop Reasoning** | A test technique requiring the model to synthesize facts across multiple retrieved chunks to reach a correct answer. | | **Cosine Similarity** | A geometric measure of the angle between two embedding vectors; indicates topic similarity but cannot distinguish true from false statements. | ### Relevance to our project This maps directly onto the editorial assistant: the **retrieval vs. generation split** is exactly the distinction I'd want when debugging why a draft review goes wrong (did it fail to retrieve the right guideline, or did it have the guideline and still misjudge?). **Faithfulness** and **Context Recall** are strong candidate metrics, and both **RAGAS** and **DeepEval** (already on the eval shortlist) implement them — so this reinforces the direction in the evaluation issue. _Source: PyData London 2026 — ["The Silent Crash" (Hitendri Bomble)](https://drive.google.com/file/d/18AcC5Jh6HxcHSr8r33Ws37mcpBuikRoA/view)._ <sub>_Assisted-by: Claude Opus 4.8_</sub>
gtfrans2re commented 2026-06-14 19:10:18 +00:00
Collaborator
Copy link

Hello, @cybette! In addition to waiting for Hitendri Bomble's contact information, I sent her a LinkedIn connection request, which she generously accepted. Just wanted to let you know about this, and I will await the details before checking off the corresponding item in the issue checklist.

Hello, @cybette! In addition to waiting for Hitendri Bomble's contact information, I sent her a LinkedIn connection request, which she generously accepted. Just wanted to let you know about this, and I will await the details before checking off the corresponding item in the issue checklist.
gtfrans2re commented 2026-06-14 19:14:07 +00:00
Collaborator
Copy link

RAGAS vs. DeepEval — comparison for our RAG evaluation

I checked out both frameworks (vibrantlabsai/ragas and confident-ai/deepeval) as candidates for evaluating the editorial assistant. Both are open-source (Apache-2.0), Python, and actively maintained. Here's how they compare.

Side-by-side

Aspect RAGAS DeepEval
Tagline "Supercharge your LLM application evaluations" "The LLM evaluation framework" (Pytest-style unit testing for LLMs)
License Apache-2.0 Apache-2.0
Language Python (+ notebooks) Python (≥3.9)
Popularity ~14.4k stars / ~1.5k forks ~15.4k stars / ~1.4k forks
Primary focus RAG evaluation + test-data generation Broad LLM eval (RAG, agents, chatbots, multimodal) as unit tests
Core RAG metrics Faithfulness, answer/context relevance, context precision/recall Faithfulness, Answer Relevancy, Contextual Precision/Recall/Relevancy (plus a built-in RAGAS composite metric)
LLM-as-a-judge Yes (LLM-based + traditional metrics) Yes — G-Eval and DAG (graph-based deterministic judge)
Test data generation Yes — synthetic/production-aligned test set generation Yes — single and multi-turn synthetic datasets
CI/CD integration Via integrations First-class — Pytest-like deepeval test run, plugs into any CI/CD
Local models Works with any LLM via config Explicitly supports metrics running locally (LLM-as-judge or local NLP models)
Integrations LangChain + major observability tools OpenAI, Anthropic, LangChain/LangGraph, LlamaIndex, CrewAI, Pydantic AI, etc.
Scope beyond RAG Mostly RAG (agent/prompt/workflow templates "coming soon") Broad: agentic, multi-turn, MCP, multimodal, bias/toxicity, benchmarking
Optional hosted platform Ragas app / managed offering Confident AI (datasets, tracing, reports) — optional, not required
Latest release (at time of writing) v0.4.3 (Jan 2026) v4.0.x (May 2026)

How each fits our project

RAGAS is the more focused, RAG-native option. Its metrics map cleanly onto what we care about — Faithfulness (are the review's claims grounded in the retrieved guideline?) and Context Recall/Precision (did we retrieve the right guideline chunks?). The built-in test-set generation is appealing since we don't yet have a labeled eval set of draft → expected-review pairs.

DeepEval is broader and more engineering-oriented. The Pytest-style workflow (deepeval test run) and first-class CI/CD fit nicely with the idea of running evals automatically when the corpus or guidelines change (ties into the rebuild-trigger discussion). It also offers G-Eval for custom criteria — useful because our editorial checks (tone, structure, required elements like featured image / Read More tag) are bespoke and don't map to a stock metric. Notably, DeepEval even ships a RAGAS composite metric, so it can run RAGAS-style scoring within its own harness.

My recommendation

They're complementary rather than mutually exclusive:

  • RAGAS for the core RAG quality metrics (Faithfulness, Context Recall/Precision) and for generating an initial test set.
  • DeepEval for custom editorial criteria via G-Eval and for wiring evals into CI/CD.

If we want to start with one, I'd lean DeepEval first — the custom-criteria support (G-Eval) matches our bespoke editorial checks, it runs locally (important given our hardware constraints and the open-weight, local-first direction), and it can reproduce RAGAS metrics anyway. We can layer in RAGAS's test-set generation afterward.

Next step (if useful): stand up a tiny eval harness with ~5–10 of the good/bad reference articles as test cases and run both on the same set to compare scores directly.

Sources: vibrantlabsai/ragas, confident-ai/deepeval.

Assisted-by: Claude Opus 4.8

## RAGAS vs. DeepEval — comparison for our RAG evaluation I checked out both frameworks ([vibrantlabsai/ragas](https://github.com/vibrantlabsai/ragas) and [confident-ai/deepeval](https://github.com/confident-ai/deepeval)) as candidates for evaluating the editorial assistant. Both are open-source (Apache-2.0), Python, and actively maintained. Here's how they compare. ### Side-by-side | Aspect | RAGAS | DeepEval | |--------|-------|----------| | **Tagline** | "Supercharge your LLM application evaluations" | "The LLM evaluation framework" (Pytest-style unit testing for LLMs) | | **License** | Apache-2.0 | Apache-2.0 | | **Language** | Python (+ notebooks) | Python (≥3.9) | | **Popularity** | ~14.4k stars / ~1.5k forks | ~15.4k stars / ~1.4k forks | | **Primary focus** | RAG evaluation + test-data generation | Broad LLM eval (RAG, agents, chatbots, multimodal) as unit tests | | **Core RAG metrics** | Faithfulness, answer/context relevance, context precision/recall | Faithfulness, Answer Relevancy, Contextual Precision/Recall/Relevancy (plus a built-in `RAGAS` composite metric) | | **LLM-as-a-judge** | Yes (LLM-based + traditional metrics) | Yes — G-Eval and DAG (graph-based deterministic judge) | | **Test data generation** | Yes — synthetic/production-aligned test set generation | Yes — single and multi-turn synthetic datasets | | **CI/CD integration** | Via integrations | First-class — Pytest-like `deepeval test run`, plugs into any CI/CD | | **Local models** | Works with any LLM via config | Explicitly supports metrics running **locally** (LLM-as-judge or local NLP models) | | **Integrations** | LangChain + major observability tools | OpenAI, Anthropic, LangChain/LangGraph, LlamaIndex, CrewAI, Pydantic AI, etc. | | **Scope beyond RAG** | Mostly RAG (agent/prompt/workflow templates "coming soon") | Broad: agentic, multi-turn, MCP, multimodal, bias/toxicity, benchmarking | | **Optional hosted platform** | Ragas app / managed offering | Confident AI (datasets, tracing, reports) — optional, not required | | **Latest release (at time of writing)** | v0.4.3 (Jan 2026) | v4.0.x (May 2026) | ### How each fits our project **RAGAS** is the more focused, RAG-native option. Its metrics map cleanly onto what we care about — Faithfulness (are the review's claims grounded in the retrieved guideline?) and Context Recall/Precision (did we retrieve the right guideline chunks?). The built-in **test-set generation** is appealing since we don't yet have a labeled eval set of draft → expected-review pairs. **DeepEval** is broader and more engineering-oriented. The Pytest-style workflow (`deepeval test run`) and first-class CI/CD fit nicely with the idea of running evals automatically when the corpus or guidelines change (ties into the rebuild-trigger discussion). It also offers **G-Eval** for custom criteria — useful because our editorial checks (tone, structure, required elements like featured image / Read More tag) are bespoke and don't map to a stock metric. Notably, DeepEval even ships a `RAGAS` composite metric, so it can run RAGAS-style scoring within its own harness. ### My recommendation They're complementary rather than mutually exclusive: - **RAGAS** for the core RAG quality metrics (Faithfulness, Context Recall/Precision) and for generating an initial test set. - **DeepEval** for custom editorial criteria via G-Eval and for wiring evals into CI/CD. If we want to start with one, I'd lean **DeepEval** first — the custom-criteria support (G-Eval) matches our bespoke editorial checks, it runs locally (important given our hardware constraints and the open-weight, local-first direction), and it can reproduce RAGAS metrics anyway. We can layer in RAGAS's test-set generation afterward. Next step (if useful): stand up a tiny eval harness with ~5–10 of the good/bad reference articles as test cases and run both on the same set to compare scores directly. _Sources: [vibrantlabsai/ragas](https://github.com/vibrantlabsai/ragas), [confident-ai/deepeval](https://github.com/confident-ai/deepeval)._ <sub>_Assisted-by: Claude Opus 4.8_</sub>
gtfrans2re commented 2026-06-16 11:36:10 +00:00
Collaborator
Copy link

Dear mentor, @cybette, I explored llmfit, a terminal tool (TUI) that scans your hardware and tells you which local LLMs will actually run on your machine, and how well. It's directly useful for picking models for the editorial assistant, given our hardware constraints before GPU provision.

What it does:

  • Detects your CPU, RAM, and GPU, then rates each model's Fit (Good / Marginal) for your system.
  • Shows practical specs per model: parameters, quantization, disk size, estimated tok/s, memory %, context length, and a quality Score.
  • Filters by provider, use case (Reasoning / Coding / Chat / General), capabilities, and runtime backend (Ollama, MLX, llama.cpp, vLLM, etc.).

On my machine (Intel Core Ultra 5 125H, 18 cores, ~24 GB RAM available out of the ~32GB total, Intel Arc shared GPU via SYCL), it lists 4317 of 4319 models and marks the realistic ones as "Good" fit — letting me sort by score and immediately see what's runnable vs. what would overflow memory.

Why it's relevant to us:

  • It's a fast, objective way to shortlist local, open-weight models that fit our hardware — exactly the constraint we hit when narrowing to Gemma 4 / Granite 4.
  • The tok/s and memory-% estimates help anticipate whether a model will run smoothly or choke before we even pull it.
  • Useful as the GPU/server hardware lands: we can re-scan and see which larger models become viable.

Screenshot of the llmfit TUI attached below.

Assisted-by: Claude Opus 4.8

Dear mentor, @cybette, I explored **llmfit**, a terminal tool (TUI) that scans your hardware and tells you which local LLMs will actually run on your machine, and how well. It's directly useful for picking models for the editorial assistant, given our hardware constraints before GPU provision. **What it does:** - Detects your CPU, RAM, and GPU, then rates each model's **Fit** (Good / Marginal) for your system. - Shows practical specs per model: parameters, quantization, disk size, **estimated tok/s**, memory %, context length, and a quality **Score**. - Filters by provider, use case (Reasoning / Coding / Chat / General), capabilities, and runtime backend (Ollama, MLX, llama.cpp, vLLM, etc.). **On my machine** (Intel Core Ultra 5 125H, 18 cores, ~24 GB RAM available out of the ~32GB total, Intel Arc shared GPU via SYCL), it lists **4317 of 4319** models and marks the realistic ones as "Good" fit — letting me sort by score and immediately see what's runnable vs. what would overflow memory. **Why it's relevant to us:** - It's a fast, objective way to shortlist local, open-weight models that fit our hardware — exactly the constraint we hit when narrowing to Gemma 4 / Granite 4. - The tok/s and memory-% estimates help anticipate whether a model will run smoothly or choke before we even pull it. - Useful as the GPU/server hardware lands: we can re-scan and see which larger models become viable. Screenshot of the llmfit TUI attached below. <sub>_Assisted-by: Claude Opus 4.8_</sub>
Screenshot from 2026-06-16 07-26-57.png
415 KiB
Screenshot from 2026-06-16 07-26-57.png
gtfrans2re commented 2026-06-16 11:48:33 +00:00
Collaborator
Copy link

I also looked into Ray and the KubeRay GPU guide as options for scaling ML workloads when we move to cluster hardware.

Ray (ray-project/ray) — an open-source distributed compute engine for ML/AI (~42.8k stars, Apache-2.0). It has a core distributed runtime plus AI libraries (Ray Data, Train, Tune, Serve) for parallelizing Python/ML workloads across many CPUs/GPUs and machines. Relevant to us for scaling: things like running benchmarks/evals over many models in parallel, or serving the RAG model across multiple replicas, rather than one-at-a-time on a single laptop.

KubeRay GPU guide (docs.ray.io) — KubeRay is the operator that runs Ray clusters on Kubernetes. The GPU guide covers requesting GPUs for Ray pods (via nvidia.com/gpu resource limits), scheduling GPU vs CPU worker groups, and autoscaling GPU nodes on demand. This maps directly onto our OpenShift/Communishift path — it's how you'd give a Ray workload GPU access on a Kubernetes/OpenShift cluster.

Relevance to the project:

  • Ray is not needed now (single-machine, two small models), but becomes useful once the GPU server/cluster is provisioned — for parallel evaluation runs and scaled model serving.
  • KubeRay is the bridge between Ray and our Kubernetes/OpenShift deployment direction, so it's worth keeping in mind alongside the Communishift work.
  • Verdict: promising for the scaling phase, not an immediate dependency. We will revisit when GPU hardware is provisioned.

Sources: ray-project/ray, KubeRay GPU usage guide.

Assisted-by: Claude Opus 4.8

I also looked into **Ray** and the **KubeRay GPU guide** as options for scaling ML workloads when we move to cluster hardware. **Ray** ([ray-project/ray](https://github.com/ray-project/ray)) — an open-source distributed compute engine for ML/AI (~42.8k stars, Apache-2.0). It has a core distributed runtime plus AI libraries (Ray Data, Train, Tune, Serve) for parallelizing Python/ML workloads across many CPUs/GPUs and machines. Relevant to us for **scaling**: things like running benchmarks/evals over many models in parallel, or serving the RAG model across multiple replicas, rather than one-at-a-time on a single laptop. **KubeRay GPU guide** ([docs.ray.io](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/gpu.html)) — KubeRay is the operator that runs Ray clusters on Kubernetes. The GPU guide covers requesting GPUs for Ray pods (via `nvidia.com/gpu` resource limits), scheduling GPU vs CPU worker groups, and autoscaling GPU nodes on demand. This maps directly onto our OpenShift/Communishift path — it's how you'd give a Ray workload GPU access on a Kubernetes/OpenShift cluster. **Relevance to the project:** - Ray is **not needed now** (single-machine, two small models), but becomes useful once the GPU server/cluster is provisioned — for parallel evaluation runs and scaled model serving. - KubeRay is the bridge between Ray and our Kubernetes/OpenShift deployment direction, so it's worth keeping in mind alongside the Communishift work. - Verdict: **promising for the scaling phase, not an immediate dependency.** We will revisit when GPU hardware is provisioned. _Sources: [ray-project/ray](https://github.com/ray-project/ray), [KubeRay GPU usage guide](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/gpu.html)._ <sub>_Assisted-by: Claude Opus 4.8_</sub>
cybette modified the due date from 2026-06-22 to 2026-06-26 2026-06-18 12:36:10 +00:00
gtfrans2re commented 2026-06-20 03:32:58 +00:00
Collaborator
Copy link

Metrix is the Digital Research Alliance of Canada's monitoring/metrics tool, with a Cloud statistics view that reports resource usage (CPU, GPU, memory, storage) on Alliance cloud projects.

What it offers us: a way to track utilization on Alliance/DRAC cloud resources. This is useful for understanding GPU/CPU consumption and right-sizing if any of our scaled workloads run on Alliance infrastructure. Given my DRAC background, this is a familiar avenue for compute monitoring.

Relevance: more of a compute-monitoring option than a RAG-eval tool, it answers "how much GPU/CPU is my workload using" rather than "how good is my RAG output." Worth keeping in mind for the scaling phase if we use Alliance cloud resources, but not central to the editorial-assistant evaluation work or the main project. Lower priority than the eval frameworks (RAGAS/DeepEval).

Assisted-by: Claude Opus 4.8

[Metrix](https://docs.alliancecan.ca/wiki/Metrix/en#Cloud_statistics) is the Digital Research Alliance of Canada's monitoring/metrics tool, with a Cloud statistics view that reports resource usage (CPU, GPU, memory, storage) on Alliance cloud projects. **What it offers us:** a way to track utilization on Alliance/DRAC cloud resources. This is useful for understanding GPU/CPU consumption and right-sizing if any of our scaled workloads run on Alliance infrastructure. Given my DRAC background, this is a familiar avenue for compute monitoring. **Relevance:** more of a **compute-monitoring** option than a RAG-eval tool, it answers "how much GPU/CPU is my workload using" rather than "how good is my RAG output." Worth keeping in mind for the scaling phase if we use Alliance cloud resources, but not central to the editorial-assistant evaluation work or the main project. Lower priority than the eval frameworks (RAGAS/DeepEval). <sub>_Assisted-by: Claude Opus 4.8_</sub>
gtfrans2re commented 2026-06-20 03:36:39 +00:00
Collaborator
Copy link

Docs2DB (from RHEL Lightspeed) builds a RAG database from documents. Its pipeline is a clear set of CLI steps: ingest → chunk → embed → load, producing a PostgreSQL + pgvector database and a portable SQL dump others can restore. Notably, it also uses Docling internally (it converts documents to Docling JSON on ingest), which is the same converter RamaLama's doc2rag relies on.

How it compares to RamaLama's doc2rag:

Aspect docs2db doc2rag (RamaLama)
Output PostgreSQL + pgvector, portable .sql dump OCI image (vector store) pushed to a registry
Distribution Download + restore a SQL dump Pull a container image (e.g. our Quay images)
Ingestion Docling → chunk → embed → load Docling (HybridChunker) inside ramalama rag
Context model Optional contextual chunking via Ollama/OpenAI/WatsonX Embeds via llama.cpp within RamaLama
Fit for us DB-centric; good if we want a queryable SQL/pgvector store Container-centric; matches our current Quay-image workflow

Takeaway: Both wrap Docling and produce a RAG store. The key difference is the artifact: docs2db ships a Postgres/pgvector dump, while doc2rag ships an OCI image. Our project is already container/Quay-aligned, so doc2rag remains the natural fit, but docs2db is worth knowing as the Postgres/pgvector alternative (and it's directly relevant to the Fedora Docs RAG work below).

Assisted-by: Claude Opus 4.8

[Docs2DB](https://github.com/rhel-lightspeed/docs2db) (from RHEL Lightspeed) builds a RAG database from documents. Its pipeline is a clear set of CLI steps: `ingest → chunk → embed → load`, producing a **PostgreSQL + pgvector** database and a portable **SQL dump** others can restore. Notably, it also uses **Docling** internally (it converts documents to Docling JSON on ingest), which is the same converter RamaLama's `doc2rag` relies on. **How it compares to RamaLama's `doc2rag`:** | Aspect | docs2db | doc2rag (RamaLama) | |--------|---------|--------------------| | Output | PostgreSQL + pgvector, portable `.sql` dump | OCI image (vector store) pushed to a registry | | Distribution | Download + restore a SQL dump | Pull a container image (e.g. our Quay images) | | Ingestion | Docling → chunk → embed → load | Docling (HybridChunker) inside `ramalama rag` | | Context model | Optional contextual chunking via Ollama/OpenAI/WatsonX | Embeds via llama.cpp within RamaLama | | Fit for us | DB-centric; good if we want a queryable SQL/pgvector store | Container-centric; matches our current Quay-image workflow | **Takeaway:** Both wrap Docling and produce a RAG store. The key difference is the **artifact**: docs2db ships a Postgres/pgvector dump, while doc2rag ships an OCI image. Our project is already container/Quay-aligned, so `doc2rag` remains the natural fit, but docs2db is worth knowing as the Postgres/pgvector alternative (and it's directly relevant to the Fedora Docs RAG work below). <sub>_Assisted-by: Claude Opus 4.8_</sub>
gtfrans2re commented 2026-06-20 03:41:35 +00:00
Collaborator
Copy link

FedoraDocsRAG provides a ready-to-use, vectorized database of Fedora documentation (Quick Docs, Sysadmin Guide, CoreOS, Silverblue, and more) for semantic search and RAG-powered Q&A. It's built using docs2db, rebuilt when upstream docs change, and distributed as a downloadable SQL dump.

Usage is simple: download the dump, restore it, and query:

curl -LO https://github.com/Lifto/FedoraDocsRAG/releases/latest/download/fedora-docs.sql

uvx docs2db db-restore fedora-docs.sql

uvx docs2db-api query "How do I install packages on Fedora?"

Relevance to us: It's a concrete, working example of the docs2db pipeline applied to Fedora documentation specifically, essentially a parallel to what we're doing, but for general Fedora docs rather than editorial guidelines, and producing a pgvector DB rather than an OCI image. Good reference for how a Fedora-wide RAG corpus is assembled and kept up to date (the "rebuilt when upstream changes" point echoes our own rebuild-trigger discussion). There's also a Fedora Magazine article walking through it end to end.

Assisted-by: Claude Opus 4.8

[FedoraDocsRAG](https://github.com/Lifto/FedoraDocsRAG) provides a ready-to-use, vectorized database of Fedora documentation (Quick Docs, Sysadmin Guide, CoreOS, Silverblue, and more) for semantic search and RAG-powered Q&A. It's **built using docs2db**, rebuilt when upstream docs change, and distributed as a downloadable SQL dump. Usage is simple: download the dump, restore it, and query: ``` curl -LO https://github.com/Lifto/FedoraDocsRAG/releases/latest/download/fedora-docs.sql uvx docs2db db-restore fedora-docs.sql uvx docs2db-api query "How do I install packages on Fedora?" ``` **Relevance to us:** It's a concrete, working example of the docs2db pipeline applied to **Fedora documentation specifically**, essentially a parallel to what we're doing, but for general Fedora docs rather than editorial guidelines, and producing a pgvector DB rather than an OCI image. Good reference for how a Fedora-wide RAG corpus is assembled and kept up to date (the "rebuilt when upstream changes" point echoes our own rebuild-trigger discussion). There's also a [Fedora Magazine article](https://fedoramagazine.org/how-to-make-a-local-open-source-ai-chatbot-who-has-access-to-fedora-documentation/) walking through it end to end. <sub>_Assisted-by: Claude Opus 4.8_</sub>
gtfrans2re commented 2026-06-20 03:46:19 +00:00
Collaborator
Copy link

This proposal (Fedora-Join#337) is about an onboarding Matrix bot that would answer newcomer questions using a RAG backend built from docs2db + FedoraDocsRAG, for instance, grounding the bot's answers in actual Fedora documentation rather than a bare LLM.

Relevance to us: it's the same building blocks (Docling-based ingestion → vector store → RAG Q&A) applied to a community-onboarding use case, and it shows where this tooling is heading inside Fedora. Two useful angles for our work:

  • It's a real example of a RAG-backed Fedora service with an end-user interface (a Matrix bot), comparable to our Streamlit app as the user-facing layer.
  • If the editorial assistant or the RPM packaging guidelines work later wants a chat/bot interface, this proposal is a good reference architecture to learn from.

Status: exploration only, not something to integrate now, but worth tracking as related work in the Fedora RAG space.

Assisted-by: Claude Opus 4.8

This proposal ([Fedora-Join#337](https://forge.fedoraproject.org/Fedora-Join/Fedora-Join/issues/337)) is about an onboarding **Matrix bot** that would answer newcomer questions using a RAG backend built from **docs2db + FedoraDocsRAG**, for instance, grounding the bot's answers in actual Fedora documentation rather than a bare LLM. **Relevance to us:** it's the same building blocks (Docling-based ingestion → vector store → RAG Q&A) applied to a **community-onboarding** use case, and it shows where this tooling is heading inside Fedora. Two useful angles for our work: - It's a real example of a **RAG-backed Fedora service** with an end-user interface (a Matrix bot), comparable to our Streamlit app as the user-facing layer. - If the editorial assistant or the RPM packaging guidelines work later wants a chat/bot interface, this proposal is a good reference architecture to learn from. **Status:** exploration only, not something to integrate now, but worth tracking as related work in the Fedora RAG space. <sub>_Assisted-by: Claude Opus 4.8_</sub>
gtfrans2re commented 2026-06-25 20:09:46 +00:00
Collaborator
Copy link

@gtfrans2re wrote in #154 (comment):

Summary: PyData London 2026 — "The Silent Crash" (Hitendri Bomble)

I read through this talk as part of the RAG evaluation exploration. It argues that traditional NLP metrics fail at evaluating RAG systems, and proposes a more robust framework built on LLM-as-a-judge plus adversarial testing.

Main points

  • Metric failure: BLEU, ROUGE, BERTScore, and cosine similarity reward surface-level lexical/topic overlap rather than factual correctness or logical entailment — so they miss critical entity swaps that change legal or factual meaning.
  • Evaluation decomposition: RAG performance should be split into independent retrieval metrics (Context Recall / Precision) and generation metrics (Faithfulness, Relevance) to pinpoint where a failure actually occurs.
  • LLM-as-a-judge: Using capable LLMs with G-Eval (chain-of-thought scoring + logprobs) enables semantic judgments of nuanced criteria like factuality and groundedness.
  • Bias mitigation: Judge LLMs suffer from position, verbosity, and self-preference biases, which need mitigations like swapping input order, cross-model judging, and multi-judge ensembles.
  • Adversarial testing: Robust evaluation requires deliberately "hard" test sets using negations, multi-hop reasoning, and counterfactuals to expose edge-case failures.

Section summaries

Section Summary
The Metrics Graveyard Traditional NLP metrics (BLEU, ROUGE, BERTScore) and cosine similarity are inadequate for RAG because they reward word overlap over factual correctness, and often fail to detect entity swaps that change meaning.
Retrieval vs. Generation Failures must be categorized as either retrieval failures (wrong chunks) or generation failures (hallucinations despite correct chunks). Decomposing them lets you debug the specific pipeline component at fault.
LLM-as-a-Judge Replaces pattern matching with semantic understanding. Tools like G-Eval use chain-of-thought reasoning and logprobs for more accurate, continuous scoring of qualitative criteria.
The Judge Paradox LLM judges are more capable but introduce biases (preferring the first option or longer answers). Mitigation includes cross-model judging and calibrating against human evaluators.
Adversarial Test Sets Production data misses edge cases. Build "hard" suites using negation injection, entity swapping, and multi-hop reasoning to test true robustness.
Practical Tooling & Strategy Frameworks like RAGAS and DeepEval provide off-the-shelf metrics and CI/CD integration. Costs stay manageable via prompt caching, batch APIs, and hierarchical evaluation.

Key concepts

Concept Definition
Faithfulness Measures whether every claim in the generated answer is strictly supported by the retrieved context — effectively catching hallucinations.
Context Recall The percentage of required information chunks successfully retrieved from the corpus to answer a query.
G-Eval An LLM-based evaluation framework using chain-of-thought reasoning and probability weighting (logprobs) to produce reliable scores.
Self-Preference Bias An LLM judge's tendency to rate outputs from its own model family higher than competing models.
Multi-hop Reasoning A test technique requiring the model to synthesize facts across multiple retrieved chunks to reach a correct answer.
Cosine Similarity A geometric measure of the angle between two embedding vectors; indicates topic similarity but cannot distinguish true from false statements.

Relevance to our project

This maps directly onto the editorial assistant: the retrieval vs. generation split is exactly the distinction I'd want when debugging why a draft review goes wrong (did it fail to retrieve the right guideline, or did it have the guideline and still misjudge?). Faithfulness and Context Recall are strong candidate metrics, and both RAGAS and DeepEval (already on the eval shortlist) implement them — so this reinforces the direction in the evaluation issue.

Source: PyData London 2026 — "The Silent Crash" (Hitendri Bomble).

Virtually met today with Hitendri, as suggested by @cybette, to discuss RAG evaluation approaches and tools. She recommended testing both RAGAS and DeepEval on the mini-project and shared guidance on evaluation considerations based on her presentation. She also mentioned that she is preparing a GitHub repository with a step-by-step guide for evaluating RAG systems, which she plans to share once completed. Follow-up is available if questions arise during implementation.

Takeaway: Explore and compare RAGAS and DeepEval for the mini-project evaluation workflow, and leverage the upcoming evaluation guide as a reference for best practices.

Assisted-by: Claude Opus 4.8

@gtfrans2re wrote in https://forge.fedoraproject.org/commops/interns/issues/154#issuecomment-821910: > ## [](#summary-pydata-london-2026-the-silent-crash-hitendri-bomble-https-drive-google-com-file-d-18acc5jh6hxchsr8r33ws37mcpbuikroa-view)Summary: PyData London 2026 — ["The Silent Crash" (Hitendri Bomble)](https://drive.google.com/file/d/18AcC5Jh6HxcHSr8r33Ws37mcpBuikRoA/view) > I read through this talk as part of the RAG evaluation exploration. It argues that **traditional NLP metrics fail at evaluating RAG systems**, and proposes a more robust framework built on **LLM-as-a-judge** plus **adversarial testing**. > > ### [](#main-points)Main points > * **Metric failure:** BLEU, ROUGE, BERTScore, and cosine similarity reward surface-level lexical/topic overlap rather than factual correctness or logical entailment — so they miss critical entity swaps that change legal or factual meaning. > * **Evaluation decomposition:** RAG performance should be split into independent **retrieval** metrics (Context Recall / Precision) and **generation** metrics (Faithfulness, Relevance) to pinpoint where a failure actually occurs. > * **LLM-as-a-judge:** Using capable LLMs with **G-Eval** (chain-of-thought scoring + logprobs) enables semantic judgments of nuanced criteria like factuality and groundedness. > * **Bias mitigation:** Judge LLMs suffer from position, verbosity, and self-preference biases, which need mitigations like swapping input order, cross-model judging, and multi-judge ensembles. > * **Adversarial testing:** Robust evaluation requires deliberately "hard" test sets using negations, multi-hop reasoning, and counterfactuals to expose edge-case failures. > > ### [](#section-summaries)Section summaries > Section Summary > **The Metrics Graveyard** Traditional NLP metrics (BLEU, ROUGE, BERTScore) and cosine similarity are inadequate for RAG because they reward word overlap over factual correctness, and often fail to detect entity swaps that change meaning. > **Retrieval vs. Generation** Failures must be categorized as either retrieval failures (wrong chunks) or generation failures (hallucinations despite correct chunks). Decomposing them lets you debug the specific pipeline component at fault. > **LLM-as-a-Judge** Replaces pattern matching with semantic understanding. Tools like G-Eval use chain-of-thought reasoning and logprobs for more accurate, continuous scoring of qualitative criteria. > **The Judge Paradox** LLM judges are more capable but introduce biases (preferring the first option or longer answers). Mitigation includes cross-model judging and calibrating against human evaluators. > **Adversarial Test Sets** Production data misses edge cases. Build "hard" suites using negation injection, entity swapping, and multi-hop reasoning to test true robustness. > **Practical Tooling & Strategy** Frameworks like RAGAS and DeepEval provide off-the-shelf metrics and CI/CD integration. Costs stay manageable via prompt caching, batch APIs, and hierarchical evaluation. > ### [](#key-concepts)Key concepts > Concept Definition > **Faithfulness** Measures whether every claim in the generated answer is strictly supported by the retrieved context — effectively catching hallucinations. > **Context Recall** The percentage of required information chunks successfully retrieved from the corpus to answer a query. > **G-Eval** An LLM-based evaluation framework using chain-of-thought reasoning and probability weighting (logprobs) to produce reliable scores. > **Self-Preference Bias** An LLM judge's tendency to rate outputs from its own model family higher than competing models. > **Multi-hop Reasoning** A test technique requiring the model to synthesize facts across multiple retrieved chunks to reach a correct answer. > **Cosine Similarity** A geometric measure of the angle between two embedding vectors; indicates topic similarity but cannot distinguish true from false statements. > ### [](#relevance-to-our-project)Relevance to our project > This maps directly onto the editorial assistant: the **retrieval vs. generation split** is exactly the distinction I'd want when debugging why a draft review goes wrong (did it fail to retrieve the right guideline, or did it have the guideline and still misjudge?). **Faithfulness** and **Context Recall** are strong candidate metrics, and both **RAGAS** and **DeepEval** (already on the eval shortlist) implement them — so this reinforces the direction in the evaluation issue. > > _Source: PyData London 2026 — ["The Silent Crash" (Hitendri Bomble)](https://drive.google.com/file/d/18AcC5Jh6HxcHSr8r33Ws37mcpBuikRoA/view)._ Virtually met today with Hitendri, as suggested by @cybette, to discuss RAG evaluation approaches and tools. She recommended testing both RAGAS and DeepEval on the mini-project and shared guidance on evaluation considerations based on her presentation. She also mentioned that she is preparing a GitHub repository with a step-by-step guide for evaluating RAG systems, which she plans to share once completed. Follow-up is available if questions arise during implementation. **Takeaway:** Explore and compare RAGAS and DeepEval for the mini-project evaluation workflow, and leverage the upcoming evaluation guide as a reference for best practices. <sub>_Assisted-by: Claude Opus 4.8_</sub>
jflory7 commented 2026-07-01 02:21:27 +00:00
Owner
Copy link

Closing: Evaluation phase complete

The RAG tool evaluation tracked here is done. @gtfrans2re has explored the relevant tools and systems, and the findings from this work inform the implementation phase now tracked in #160 and #161. Closing to reduce project bloat and keep focus on active deliverables.

Assisted-by: Claude Opus 4.6 (1M context)

### Closing: Evaluation phase complete The RAG tool evaluation tracked here is done. @gtfrans2re has explored the relevant tools and systems, and the findings from this work inform the implementation phase now tracked in #160 and #161. Closing to reduce project bloat and keep focus on active deliverables. <sub>_Assisted-by: Claude Opus 4.6 (1M context)_</sub>
👀 1
gtfrans2re commented 2026-07-01 10:51:25 +00:00
Collaborator
Copy link

Thanks for the update, @jflory7! It was great exploring these RAG tools and mapping out what works best for our needs. I am glad the findings provided a solid foundation for the next steps.

Thanks for the update, @jflory7! It was great exploring these RAG tools and mapping out what works best for our needs. I am glad the findings provided a solid foundation for the next steps.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
state
ready for review
 Ready for final reviewer by a mentor. This label is used for tickets in the "In Progress" column on the Forgejo Projects Boards.

  
function
analytics
 Community Social Analytics. One of the two main functions of the Community Operations Team. All issues/tasks should identify a function as different team members are responsible for different function

  
function
communications
 Community Blog, newsletters, outreach content, Magazine coordination, and community messaging

  
function
docs
 Documentation site, playbooks, guides, knowledge base, and contributor reference materials

  
function
events
 Event production, release parties, conferences, streaming, video processing, and community event logistics

  
function
onboarding
 Newcomer experience, mentoring, intern programs, team growth, and contributor onboarding

  
function
operations
 Team governance, repo management, meeting logistics, infrastructure, and internal process

  
good first issue
Good tickets to begin with (if you want to get involved, try working on one of these!)

  
help wanted
Tickets where we need help or need feedback from the wider Fedora community

  
needs
changes
 Needs changes or fixes before closing

  
needs
reporter feedback
 Additional feedback or information from the reporter is needed to advance

  
needs
team vote
 A vote is required from Fedora Community Operations Team members in order to advance

  
needs
triage
 A repository maintainer is needed to triage this further

  
scope
bug
 Something isn't working

  
scope
improvement
 Requests for something we are either repeating or improving

  
scope
new
 Requests for a new task or project for the team

  
state
approved
 Tickets that received approval by the Fedora Community Operations Team

  
state
blocked
 Cannot move forward: blocked by feedback, other tickets, etc.

  
state
duplicate
 This issue or merge request already exists

  
state
invalid
 This doesn't seem right

  
state
wontfix
 This will not be worked on

No labels
state
ready for review
function
analytics
function
communications
function
docs
function
events
function
onboarding
function
operations
good first issue
help wanted
needs
changes
needs
reporter feedback
needs
team vote
needs
triage
scope
bug
scope
improvement
scope
new
state
approved
state
blocked
state
duplicate
state
invalid
state
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
gtfrans2re
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

2026-06-26

Dependencies

No dependencies set.

Reference
commops/interns#154
No description provided.