docs: add GPU deployment runbook (CDI passthrough)
Some checks failed
CI / checks (push) Has been cancelled
Some checks failed
CI / checks (push) Has been cancelled
Document enabling GPU acceleration: install the NVIDIA Container Toolkit and generate a CDI config so Podman can pass the GPU into the container. Without it, ramalama silently falls back to CPU (76 min vs 6 min for the vectordb build). Assisted-by: Claude Opus 4.8 Signed-off-by: gtfrans2re <francoisgonothitoure@gmail.com>
This commit is contained in:
parent
1d328ec837
commit
bc1d7f296b
1 changed files with 114 additions and 0 deletions
114
docs/gpu-deployment.md
Normal file
114
docs/gpu-deployment.md
Normal file
|
|
@ -0,0 +1,114 @@
|
|||
# Running the RAG pipeline on a GPU
|
||||
|
||||
This documents how to enable GPU acceleration for the RAG pipeline on the
|
||||
deployment host, so builds and inference use the GPU instead of falling back to
|
||||
the CPU.
|
||||
|
||||
It assumes the NVIDIA drivers are already installed and `nvidia-smi` works on
|
||||
the host. If not, install the drivers first (see the driver setup runbook in
|
||||
the packaging-guide-ramalama repo).
|
||||
|
||||
## The blocker: the pipeline silently runs on CPU
|
||||
|
||||
Installing the NVIDIA drivers is necessary but not sufficient.
|
||||
`ramalama` runs its models inside a Podman container, and by default the
|
||||
container cannot see the GPU. When this happens, every `ramalama` command
|
||||
prints:
|
||||
|
||||
```
|
||||
Device all does not have a CDI configuration
|
||||
No CDI configuration found for all
|
||||
You can use the "nvidia-ctk cdi generate" command from the
|
||||
nvidia-container-toolkit to generate a CDI configuration.
|
||||
```
|
||||
|
||||
This is only a warning, so the pipeline continues, but it runs entirely on the
|
||||
CPU. The cost is large: on this host, building the vector store for the
|
||||
Community Blog corpus (8888 chunks) took about 76 minutes on the CPU and about
|
||||
6 minutes once the GPU was passed through, a roughly 13x difference. GPU
|
||||
utilization stayed at 0% for the whole CPU run, which is how you can confirm
|
||||
the fallback is happening.
|
||||
|
||||
The fix is to configure the NVIDIA Container Toolkit and generate a CDI
|
||||
(Container Device Interface) configuration so Podman can pass the GPU into the
|
||||
container.
|
||||
|
||||
## Steps
|
||||
|
||||
All commands run as root on the deployment host.
|
||||
|
||||
### 1. Install the NVIDIA Container Toolkit
|
||||
|
||||
Add the toolkit repository and install it.
|
||||
|
||||
```bash
|
||||
curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo \
|
||||
| tee /etc/yum.repos.d/nvidia-container-toolkit.repo
|
||||
|
||||
dnf install -y nvidia-container-toolkit
|
||||
```
|
||||
|
||||
### 2. Generate the CDI configuration
|
||||
|
||||
This writes a device spec that Podman reads to inject the GPU.
|
||||
|
||||
```bash
|
||||
nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
|
||||
|
||||
# confirm the devices were registered
|
||||
nvidia-ctk cdi list
|
||||
```
|
||||
|
||||
`nvidia-ctk cdi list` should show entries such as `nvidia.com/gpu=0` and
|
||||
`nvidia.com/gpu=all`.
|
||||
|
||||
### 3. Verify GPU passthrough into a container
|
||||
|
||||
Before running the pipeline, confirm a container can see the GPU.
|
||||
This is much faster than discovering a problem mid-build.
|
||||
|
||||
```bash
|
||||
podman run --rm --device nvidia.com/gpu=all \
|
||||
docker.io/nvidia/cuda:12.4.1-base-ubi9 nvidia-smi
|
||||
```
|
||||
|
||||
If this prints the GPU table from inside the container, passthrough works.
|
||||
|
||||
### 4. Run the pipeline
|
||||
|
||||
With CDI configured, `ramalama` auto-detects it and the CDI warning no longer
|
||||
appears. Run the build as usual:
|
||||
|
||||
```bash
|
||||
ramalama rag --chunk-size 256 data/source/commblog quay.io/fedora/editorial-guide-ramalama:commblog
|
||||
```
|
||||
|
||||
If `ramalama` still reports no CDI configuration, pass the device explicitly:
|
||||
|
||||
```bash
|
||||
ramalama rag --device nvidia.com/gpu=all --chunk-size 256 \
|
||||
data/source/commblog quay.io/fedora/editorial-guide-ramalama:commblog
|
||||
```
|
||||
|
||||
## Confirming the GPU is actually used
|
||||
|
||||
Sample the GPU while a build runs and check that utilization goes above 0%:
|
||||
|
||||
```bash
|
||||
nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used --format=csv -l 2 > gpu-load.csv &
|
||||
# run the build, then:
|
||||
awk -F, 'NR>1{gsub(/ %/,"",$2); if($2+0>max)max=$2} END{print "peak GPU util:", max"%"}' gpu-load.csv
|
||||
```
|
||||
|
||||
A non-zero peak confirms the work is on the GPU.
|
||||
A flat 0% means it is still falling back to the CPU, and the CDI setup above
|
||||
needs revisiting.
|
||||
|
||||
## Notes
|
||||
|
||||
- CDI is the mechanism Podman uses for device passthrough; it is the rootless
|
||||
friendly path and does not require running containers as privileged.
|
||||
- The CDI spec at /etc/cdi/nvidia.yaml is tied to the current driver. If the
|
||||
driver is updated, regenerate it with `nvidia-ctk cdi generate` again.
|
||||
- `nvcc` (the CUDA compiler) is not required for inference; the CUDA runtime is
|
||||
enough, and it is provided through the driver and container image.
|
||||
Loading…
Add table
Add a link
Reference in a new issue