docs: add GPU deployment runbook (CDI passthrough)
Some checks failed
CI / checks (push) Has been cancelled

Document enabling GPU acceleration: install the NVIDIA Container Toolkit and
generate a CDI config so Podman can pass the GPU into the container. Without it,
ramalama silently falls back to CPU (76 min vs 6 min for the vectordb build).

Assisted-by: Claude Opus 4.8
Signed-off-by: gtfrans2re <francoisgonothitoure@gmail.com>
This commit is contained in:
Francois Gonothi Toure 2026-08-11 20:29:04 -04:00 • committed by Carol Chen
commit bc1d7f296b

114
docs/gpu-deployment.md Normal file
View file

@ -0,0 +1,114 @@
# Running the RAG pipeline on a GPU
This documents how to enable GPU acceleration for the RAG pipeline on the
deployment host, so builds and inference use the GPU instead of falling back to
the CPU.
It assumes the NVIDIA drivers are already installed and `nvidia-smi` works on
the host. If not, install the drivers first (see the driver setup runbook in
the packaging-guide-ramalama repo).
## The blocker: the pipeline silently runs on CPU
Installing the NVIDIA drivers is necessary but not sufficient.
`ramalama` runs its models inside a Podman container, and by default the
container cannot see the GPU. When this happens, every `ramalama` command
prints:
```
Device all does not have a CDI configuration
No CDI configuration found for all
You can use the "nvidia-ctk cdi generate" command from the
nvidia-container-toolkit to generate a CDI configuration.
```
This is only a warning, so the pipeline continues, but it runs entirely on the
CPU. The cost is large: on this host, building the vector store for the
Community Blog corpus (8888 chunks) took about 76 minutes on the CPU and about
6 minutes once the GPU was passed through, a roughly 13x difference. GPU
utilization stayed at 0% for the whole CPU run, which is how you can confirm
the fallback is happening.
The fix is to configure the NVIDIA Container Toolkit and generate a CDI
(Container Device Interface) configuration so Podman can pass the GPU into the
container.
## Steps
All commands run as root on the deployment host.
### 1. Install the NVIDIA Container Toolkit
Add the toolkit repository and install it.
```bash
curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo \
| tee /etc/yum.repos.d/nvidia-container-toolkit.repo
dnf install -y nvidia-container-toolkit
```
### 2. Generate the CDI configuration
This writes a device spec that Podman reads to inject the GPU.
```bash
nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
# confirm the devices were registered
nvidia-ctk cdi list
```
`nvidia-ctk cdi list` should show entries such as `nvidia.com/gpu=0` and
`nvidia.com/gpu=all`.
### 3. Verify GPU passthrough into a container
Before running the pipeline, confirm a container can see the GPU.
This is much faster than discovering a problem mid-build.
```bash
podman run --rm --device nvidia.com/gpu=all \
docker.io/nvidia/cuda:12.4.1-base-ubi9 nvidia-smi
```
If this prints the GPU table from inside the container, passthrough works.
### 4. Run the pipeline
With CDI configured, `ramalama` auto-detects it and the CDI warning no longer
appears. Run the build as usual:
```bash
ramalama rag --chunk-size 256 data/source/commblog quay.io/fedora/editorial-guide-ramalama:commblog
```
If `ramalama` still reports no CDI configuration, pass the device explicitly:
```bash
ramalama rag --device nvidia.com/gpu=all --chunk-size 256 \
data/source/commblog quay.io/fedora/editorial-guide-ramalama:commblog
```
## Confirming the GPU is actually used
Sample the GPU while a build runs and check that utilization goes above 0%:
```bash
nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used --format=csv -l 2 > gpu-load.csv &
# run the build, then:
awk -F, 'NR>1{gsub(/ %/,"",$2); if($2+0>max)max=$2} END{print "peak GPU util:", max"%"}' gpu-load.csv
```
A non-zero peak confirms the work is on the GPU.
A flat 0% means it is still falling back to the CPU, and the CDI setup above
needs revisiting.
## Notes
- CDI is the mechanism Podman uses for device passthrough; it is the rootless
friendly path and does not require running containers as privileged.
- The CDI spec at /etc/cdi/nvidia.yaml is tied to the current driver. If the
driver is updated, regenerate it with `nvidia-ctk cdi generate` again.
- `nvcc` (the CUDA compiler) is not required for inference; the CUDA runtime is
enough, and it is provided through the driver and container image.