docs: add GPU deployment runbook (CDI passthrough) #15

Merged

cybette merged 1 commit from gtfrans2re/editorial-guide-ramalama:docs/gpu-deployment into main

2026-08-28 11:07:05 +00:00

gtfrans2re commented

2026-08-12 00:32:22 +00:00

Collaborator

Summary

Documents how to enable GPU acceleration for the RAG pipeline on the deployment host (docs/gpu-deployment.md).

The key point: installing the NVIDIA drivers is necessary but not sufficient. ramalama runs its models in a Podman container, and by default the container can't see the GPU - it prints a "No CDI configuration found" warning and silently falls back to CPU. The runbook covers installing the NVIDIA Container Toolkit and generating a CDI config (nvidia-ctk cdi generate) so Podman can pass the GPU through, plus a container smoke test and how to confirm the GPU is actually used.

The impact is large: on the GPU host, the same vector store build took ~76 minutes on CPU vs ~6 minutes on GPU (about 13x).

Part of #161 (deploy and run the mini-project on the GPU host).

Type of Change

Documentation update

Checklist

My branch is up to date with main
Commits are signed off with DCO (git commit -s)
I did not push directly to main
I checked changed files for ambiguous Unicode characters (see CONTRIBUTING.md)
I ran any relevant checks/tests locally
Documentation is updated where needed

Additional Notes

The steps in this runbook were run and verified on the GPU instance; the CPU-vs-GPU timings above come from that run.

## Summary Documents how to enable GPU acceleration for the RAG pipeline on the deployment host (`docs/gpu-deployment.md`). The key point: installing the NVIDIA drivers is necessary but not sufficient. `ramalama` runs its models in a Podman container, and by default the container can't see the GPU - it prints a "No CDI configuration found" warning and silently falls back to CPU. The runbook covers installing the NVIDIA Container Toolkit and generating a CDI config (`nvidia-ctk cdi generate`) so Podman can pass the GPU through, plus a container smoke test and how to confirm the GPU is actually used. The impact is large: on the GPU host, the same vector store build took ~76 minutes on CPU vs ~6 minutes on GPU (about 13x). ## Related Issue Part of [#161](https://forge.fedoraproject.org/commops/interns/issues/161) (deploy and run the mini-project on the GPU host). ## Type of Change - [x] Documentation update ## Checklist - [x] My branch is up to date with `main` - [x] Commits are signed off with DCO (`git commit -s`) - [x] I did not push directly to `main` - [x] I checked changed files for ambiguous Unicode characters (see CONTRIBUTING.md) - [x] I ran any relevant checks/tests locally - [x] Documentation is updated where needed ## Additional Notes The steps in this runbook were run and verified on the GPU instance; the CPU-vs-GPU timings above come from that run.

gtfrans2re added 1 commit

2026-08-12 00:32:22 +00:00

docs: add GPU deployment runbook (CDI passthrough) 75cee13d11

Document enabling GPU acceleration: install the NVIDIA Container Toolkit and
generate a CDI config so Podman can pass the GPU into the container. Without it,
ramalama silently falls back to CPU (76 min vs 6 min for the vectordb build).

Assisted-by: Claude Opus 4.8
Signed-off-by: gtfrans2re <francoisgonothitoure@gmail.com>