Investigate libbpf errors when building kernels on K3 hardware.

jmontleon commented

2026-06-03 15:38:31 +00:00

Owner

Originally reported around May 11 2026 @kashyapc reported seeing the following from a kernel build running in a container on a SpacemiT K3 he had access to.
https://matrix.to/#/!GhnNOKFLyvWBmnjqRS:fedoraproject.org/$dGYwmYpjCmM9rObd09u-JVvfM8TRxKmTC8zPwWOHhjw?via=fedoraproject.org&via=matrix.org&via=fedora.im

libbpf: btf: type [23553]: referenced type [69632] is not FUNC_PROTO
Failed to parse base BTF './vmlinux': -22

June 2 on a K3 Koji Builder running 7.1-rc6 omni kernel we see similar:
https://riscv-koji.fedoraproject.org/koji/taskinfo?taskID=222282

libbpf: btf: type [75584]: referenced type [1705] is not FUNC_PROTO
Failed to parse base BTF './vmlinux': -22
make[2]: *** [scripts/Makefile.modfinal:60: drivers/hid/hid-gt683r.ko] Error 1
make[2]: *** Deleting file 'drivers/hid/hid-gt683r.ko'
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [/builddir/build/BUILD/kernel-7.1.0-build/kernel-7.1-rc6/linux-7.1.0-0.rc6.40.2.0.riscv64.omni.fc44.riscv64/Makefile:2110: modules] Error 2
make: *** [Makefile:248: __sub-make] Error 2
+ exit 1

Yaneti reported reproducing more or less the same error doing a mock build on a K3 system running the Bianbu kernel.
https://matrix.to/#/!GhnNOKFLyvWBmnjqRS:fedoraproject.org/$atuU1ZN4FcXyHW2IHSt_gVT8h7zOE1jgtyUBqG56jiY?via=fedoraproject.org&via=matrix.org&via=fedora.im

WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/ufs/host/ufs-spacemit_k3.o
libbpf: btf: type [100703]: referenced type [50173] is not FUNC_PROTO
Failed to parse base BTF './vmlinux': -22
make[2]: *** [scripts/Makefile.modfinal:60: crypto/lrw.ko] Error 1

The same build succeeded in Copr:
https://copr.fedorainfracloud.org/coprs/g/forge-riscv-members/riscv64_omni_mainline_kernel/build/10550035/

Originally reported around May 11 2026 @kashyapc reported seeing the following from a kernel build running in a container on a SpacemiT K3 he had access to. https://matrix.to/#/!GhnNOKFLyvWBmnjqRS:fedoraproject.org/$dGYwmYpjCmM9rObd09u-JVvfM8TRxKmTC8zPwWOHhjw?via=fedoraproject.org&via=matrix.org&via=fedora.im ``` libbpf: btf: type [23553]: referenced type [69632] is not FUNC_PROTO Failed to parse base BTF './vmlinux': -22 ``` June 2 on a K3 Koji Builder running 7.1-rc6 omni kernel we see similar: https://riscv-koji.fedoraproject.org/koji/taskinfo?taskID=222282 ``` libbpf: btf: type [75584]: referenced type [1705] is not FUNC_PROTO Failed to parse base BTF './vmlinux': -22 make[2]: *** [scripts/Makefile.modfinal:60: drivers/hid/hid-gt683r.ko] Error 1 make[2]: *** Deleting file 'drivers/hid/hid-gt683r.ko' make[2]: *** Waiting for unfinished jobs.... make[1]: *** [/builddir/build/BUILD/kernel-7.1.0-build/kernel-7.1-rc6/linux-7.1.0-0.rc6.40.2.0.riscv64.omni.fc44.riscv64/Makefile:2110: modules] Error 2 make: *** [Makefile:248: __sub-make] Error 2 + exit 1 ``` Yaneti reported reproducing more or less the same error doing a mock build on a K3 system running the Bianbu kernel. https://matrix.to/#/!GhnNOKFLyvWBmnjqRS:fedoraproject.org/$atuU1ZN4FcXyHW2IHSt_gVT8h7zOE1jgtyUBqG56jiY?via=fedoraproject.org&via=matrix.org&via=fedora.im ``` WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/ufs/host/ufs-spacemit_k3.o libbpf: btf: type [100703]: referenced type [50173] is not FUNC_PROTO Failed to parse base BTF './vmlinux': -22 make[2]: *** [scripts/Makefile.modfinal:60: crypto/lrw.ko] Error 1 ``` The same build succeeded in Copr: https://copr.fedorainfracloud.org/coprs/g/forge-riscv-members/riscv64_omni_mainline_kernel/build/10550035/

jmontleon changed title from ~~Investigate K3 libbpf errors when building kernels on K3 hardware.~~ to Investigate libbpf errors when building kernels on K3 hardware.

2026-06-03 15:39:14 +00:00

kashyapc commented

2026-06-03 16:13:35 +00:00

Owner

Thanks, Jason! (Copy/pasting my comment from the kernel benchmark here: #17 (comment))

Intervention on K3: disable BTF kernel module

I had to disable BTF because I was hitting a crash with this error:

...
libbpf: btf: type [23553]: referenced type [69632] is not FUNC_PROTO
Failed to parse base BTF './vmlinux': -22
...

I got the Fedora omni kernel build from here:

$ rpm -ivh https://download.copr.fedorainfracloud.org/results/@forge-riscv-members/riscv64_omni_kernel/fedora-43-riscv64/10358297-kernel/kernel-6.19.14-200.0.riscv64.omni.fc43.src.rpm

Disabled the BTF module:

$ cd ./rpmbuild/BUILD/kernel-6.19.14-build/kernel-6.19.14/linux-6.19.14-200.0.riscv64.omni.fc43.riscv64/
$ ./scripts/config --disable DEBUG_INFO_BTF --disable DEBUG_INFO_BTF_MODULES

$ grep INFO_BTF .config
# CONFIG_DEBUG_INFO_BTF is not set
# CONFIG_DEBUG_INFO_BTF_MODULES is not set

Recompile. The build succeeds in ~3h24 minutes:

$ make olddefconfig
$ time rpmbuild -ba --noprep /root/rpmbuild/SPECS/kernel.spec |& tee take3_kernel_build_no_BTF.txt
...
Wrote: /root/rpmbuild/RPMS/riscv64/kernel-debuginfo-6.19.14-200.0.riscv64.omni.fc43.riscv64.rpm
[...] 
real    204m44.593s
user    1031m6.288s
sys     127m10.717s

Thanks, Jason! (Copy/pasting my comment from the kernel benchmark here: https://forge.fedoraproject.org/riscv/planning/issues/17#issuecomment-708191) ------ #### Intervention on K3: disable BTF kernel module I had to disable BTF because I was hitting a crash with this error: ``` ... libbpf: btf: type [23553]: referenced type [69632] is not FUNC_PROTO Failed to parse base BTF './vmlinux': -22 ... ``` I got the Fedora omni kernel build from here: ``` $ rpm -ivh https://download.copr.fedorainfracloud.org/results/@forge-riscv-members/riscv64_omni_kernel/fedora-43-riscv64/10358297-kernel/kernel-6.19.14-200.0.riscv64.omni.fc43.src.rpm ``` Disabled the BTF module: ``` $ cd ./rpmbuild/BUILD/kernel-6.19.14-build/kernel-6.19.14/linux-6.19.14-200.0.riscv64.omni.fc43.riscv64/ $ ./scripts/config --disable DEBUG_INFO_BTF --disable DEBUG_INFO_BTF_MODULES $ grep INFO_BTF .config # CONFIG_DEBUG_INFO_BTF is not set # CONFIG_DEBUG_INFO_BTF_MODULES is not set ``` Recompile. The build succeeds in **~3h24 minutes**: ``` $ make olddefconfig $ time rpmbuild -ba --noprep /root/rpmbuild/SPECS/kernel.spec |& tee take3_kernel_build_no_BTF.txt ... Wrote: /root/rpmbuild/RPMS/riscv64/kernel-debuginfo-6.19.14-200.0.riscv64.omni.fc43.riscv64.rpm [...] real 204m44.593s user 1031m6.288s sys 127m10.717s ```

jmontleon commented

2026-06-03 16:27:30 +00:00

Author

Owner

Some things I want to try

I am using the same kernel the K3 is running on a bpi-f3 to try and build the same package to help elliminate anything related with either. I don't think the host kernel is responsible since we observe this on hosts running Bianbu vendor kernels as well.
Build a vanilla Fedora kernel on K3 and make sure it's not related to any patches in the omni kernel. Every failed build attempt is of an omni kernel so far.
In the past with eswin vendor kernels they were copying header files around during module build which triggered a vmlinux rebuild after modules built. In the eswin case the failures were 100% reproducible every build attempt, on any hardware, so I really don't think this is what is going on here. But it would be good to closely watch a mock build, check the vmlinux modification time after the initial build and make sure the build isn't rerunning and we don't end up with a newer vmlinux to try and rule this scenario out.

Other suggests are welcome.
@davidlt suggested Gemini 3.5 FLash basically suggests make V=2, then maybe make V=1 -d modules > build_debug.log 2>&1, or just run -j1 1st and 2nd option might tell what's causing vmlinux to be rebuilt.

We can rpmbuild -bp rpmbuild/SPECS/kernel.spec, copy kernel-riscv64-fedora.config from rpmbuild/SOURCES to rpmbuild/BUILD/.../.config and run make commands manually to try and diagnose as well.

Some things I want to try - I am using the same kernel the K3 is running on a bpi-f3 to try and build the same package to help elliminate anything related with either. I don't think the host kernel is responsible since we observe this on hosts running Bianbu vendor kernels as well. - Build a vanilla Fedora kernel on K3 and make sure it's not related to any patches in the omni kernel. Every failed build attempt is of an omni kernel so far. - In the past with eswin vendor kernels they were copying header files around during module build which triggered a vmlinux rebuild after modules built. In the eswin case the failures were 100% reproducible every build attempt, on any hardware, so I really don't think this is what is going on here. But it would be good to closely watch a mock build, check the vmlinux modification time after the initial build and make sure the build isn't rerunning and we don't end up with a newer vmlinux to try and rule this scenario out. Other suggests are welcome. @davidlt suggested `Gemini 3.5 FLash basically suggests make V=2, then maybe make V=1 -d modules > build_debug.log 2>&1, or just run -j1 1st and 2nd option might tell what's causing vmlinux to be rebuilt.` We can rpmbuild -bp rpmbuild/SPECS/kernel.spec, copy kernel-riscv64-fedora.config from rpmbuild/SOURCES to rpmbuild/BUILD/.../.config and run make commands manually to try and diagnose as well.

jmontleon commented

2026-06-04 02:46:33 +00:00

Author

Owner

Another failure on the K3, albeit a different error message:

/usr/bin/ld.bfd: BFD version 2.46-3.fc44 internal error, aborting at ../../bfd/elfcode.h:224 in bfd_elf64_swap_symbol_out
Please report this bug.
make[2]: *** [scripts/Makefile.modfinal:60: drivers/hwmon/pmbus/mp2993.ko] Error 1
make[2]: *** Deleting file 'drivers/hwmon/pmbus/mp2993.ko'
make[2]: *** Waiting for unfinished jobs....

Host kernel was:

jason@k3-01:~$ uname -a
Linux k3-01.montleon.net 7.1.0-0.rc6.40.2.0.riscv64.omni.fc43.riscv64 #1 SMP PREEMPT_DYNAMIC Tue Jun  2 20:30:32 UTC 2026 riscv64 GNU/Linux

Using the same kernel on a bpi-f3 the build succeeded.

[jason@bpi02 ~]$ uname -a
Linux bpi02.montleon.net 7.1.0-0.rc6.40.2.0.riscv64.omni.fc43.riscv64 #1 SMP PREEMPT_DYNAMIC Tue Jun  2 23:25:18 UTC 2026 riscv64 GNU/Linux

In both case the mock build command was /mock -r fedora-44-riscv64 --rebuild ./kernel-7.1.0-0.rc6.40.2.0.riscv64.omni.fc43.src.rpm

Another failure on the K3, albeit a different error message: ``` /usr/bin/ld.bfd: BFD version 2.46-3.fc44 internal error, aborting at ../../bfd/elfcode.h:224 in bfd_elf64_swap_symbol_out Please report this bug. make[2]: *** [scripts/Makefile.modfinal:60: drivers/hwmon/pmbus/mp2993.ko] Error 1 make[2]: *** Deleting file 'drivers/hwmon/pmbus/mp2993.ko' make[2]: *** Waiting for unfinished jobs.... ``` Host kernel was: ``` jason@k3-01:~$ uname -a Linux k3-01.montleon.net 7.1.0-0.rc6.40.2.0.riscv64.omni.fc43.riscv64 #1 SMP PREEMPT_DYNAMIC Tue Jun 2 20:30:32 UTC 2026 riscv64 GNU/Linux ``` Using the same kernel on a bpi-f3 the build succeeded. ``` [jason@bpi02 ~]$ uname -a Linux bpi02.montleon.net 7.1.0-0.rc6.40.2.0.riscv64.omni.fc43.riscv64 #1 SMP PREEMPT_DYNAMIC Tue Jun 2 23:25:18 UTC 2026 riscv64 GNU/Linux ``` In both case the mock build command was `/mock -r fedora-44-riscv64 --rebuild ./kernel-7.1.0-0.rc6.40.2.0.riscv64.omni.fc43.src.rpm`

jmontleon commented

2026-06-04 03:10:30 +00:00

Author

Owner

After the initial vmlinuz.efi build:

1.2G -rwxr-xr-x. 1 jason mock 1.2G Jun  3 20:26 /var/lib/mock/fedora-44-riscv64/root/builddir/build/BUILD/kernel-7.1.0-build/kernel-7.1-rc6/linux-7.1.0-0.rc6.40.2.0.riscv64.omni.fc44.riscv64/vmlinux
jason@k3-01:~$ date
Wed Jun  3 08:51:21 PM UTC 2026

After the failure the file was unchanged so I do not believe it is being recompiled

$ stat /var/lib/mock/fedora-44-riscv64/root/builddir/build/BUILD/kernel-7.1.0-build/kernel-7.1-rc6/linux-7.1.0-0.rc6.40.2.0.riscv64.omni.fc44.riscv64/vmlinux
  File: /var/lib/mock/fedora-44-riscv64/root/builddir/build/BUILD/kernel-7.1.0-build/kernel-7.1-rc6/linux-7.1.0-0.rc6.40.2.0.riscv64.omni.fc44.riscv64/vmlinux
  Size: 1253408200      Blocks: 2448064    IO Block: 4096   regular file
Device: 0,44    Inode: 2038481     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: ( 1000/   jason)   Gid: (  135/    mock)
Context: unconfined_u:object_r:mock_var_lib_t:s0
Access: 2026-06-03 20:26:32.518009687 +0000
Modify: 2026-06-03 20:26:28.618212453 +0000
Change: 2026-06-03 20:26:28.618212453 +0000
 Birth: 2026-06-03 20:20:57.680419208 +0000

After the initial vmlinuz.efi build: ``` 1.2G -rwxr-xr-x. 1 jason mock 1.2G Jun 3 20:26 /var/lib/mock/fedora-44-riscv64/root/builddir/build/BUILD/kernel-7.1.0-build/kernel-7.1-rc6/linux-7.1.0-0.rc6.40.2.0.riscv64.omni.fc44.riscv64/vmlinux jason@k3-01:~$ date Wed Jun 3 08:51:21 PM UTC 2026 ``` After the failure the file was unchanged so I do not believe it is being recompiled ``` $ stat /var/lib/mock/fedora-44-riscv64/root/builddir/build/BUILD/kernel-7.1.0-build/kernel-7.1-rc6/linux-7.1.0-0.rc6.40.2.0.riscv64.omni.fc44.riscv64/vmlinux File: /var/lib/mock/fedora-44-riscv64/root/builddir/build/BUILD/kernel-7.1.0-build/kernel-7.1-rc6/linux-7.1.0-0.rc6.40.2.0.riscv64.omni.fc44.riscv64/vmlinux Size: 1253408200 Blocks: 2448064 IO Block: 4096 regular file Device: 0,44 Inode: 2038481 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 1000/ jason) Gid: ( 135/ mock) Context: unconfined_u:object_r:mock_var_lib_t:s0 Access: 2026-06-03 20:26:32.518009687 +0000 Modify: 2026-06-03 20:26:28.618212453 +0000 Change: 2026-06-03 20:26:28.618212453 +0000 Birth: 2026-06-03 20:20:57.680419208 +0000 ```

jmontleon commented

2026-06-05 17:38:44 +00:00

Author

Owner

Using upstream source (I used 7.0.11, but so far version seems irrelevant 7.1-rc6 and 6.19.4 have failed too)
The matching fedora config src.fedoraproject.org/rpms/kernel@bfa612b934/f/kernel-riscv64-fedora.config

The build succeeds with:
make -s -j8 vmlinuz.efi; make -s -j8 modules

The build fails with:
make -s 'HOSTCFLAGS=-O2 -fPIE' 'HOSTLDFLAGS=-pie' -j8 vmlinuz.efi; make -s 'HOSTCFLAGS=-O2 -fPIE' 'HOSTLDFLAGS=-pie' -j8 modules

- Using upstream source (I used 7.0.11, but so far version seems irrelevant 7.1-rc6 and 6.19.4 have failed too) - The matching fedora config https://src.fedoraproject.org/rpms/kernel/blob/bfa612b934b5251a6e8076382d16611472244433/f/kernel-riscv64-fedora.config The build succeeds with: `make -s -j8 vmlinuz.efi; make -s -j8 modules` The build fails with: `make -s 'HOSTCFLAGS=-O2 -fPIE' 'HOSTLDFLAGS=-pie' -j8 vmlinuz.efi; make -s 'HOSTCFLAGS=-O2 -fPIE' 'HOSTLDFLAGS=-pie' -j8 modules`

jmontleon commented

2026-06-05 18:35:34 +00:00

Author

Owner

Doing the same and compiling on bianbu didn't error for me, so it specific to Fedora.

davidlt commented

2026-06-06 13:24:18 +00:00

Owner

So is it Fedora defconfig or -s 'HOSTCFLAGS=-O2 -fPIE' 'HOSTLDFLAGS=-pie' flags that we pass?

The binutils error is strange. yaneti ran things in a loop + stress-ng at the same time. We got some crash reports. I fed it into Gemini 3.1 Flash + max thinking effort. I haven't looked too deep, but in all cases it was an invalid pointer IIRC. Surprisingly it was 56 (mostly) and 57 bits flipping, otherwise the address seemed somewhat okay (not verified). Gemini started speculating if this could be related to memory tagging extension (RVA23). Is it a memory corruption somewhere, or a tag which is somehow not ignored. Anyways, unknown.

So is it Fedora `defconfig` or `-s 'HOSTCFLAGS=-O2 -fPIE' 'HOSTLDFLAGS=-pie'` flags that we pass? The binutils error is strange. yaneti ran things in a loop + stress-ng at the same time. We got some crash reports. I fed it into Gemini 3.1 Flash + max thinking effort. I haven't looked too deep, but in all cases it was an invalid pointer IIRC. Surprisingly it was 56 (mostly) and 57 bits flipping, otherwise the address seemed somewhat okay (not verified). Gemini started speculating if this could be related to memory tagging extension (RVA23). Is it a memory corruption somewhere, or a tag which is somehow not ignored. Anyways, unknown.

jmontleon commented

2026-06-08 03:15:25 +00:00

Author

Owner

At this point I don't really feel I have a grasp of what's actually causing the error. All I can provide are my observations.

To date I have never managed to reproduce it running just make -j8 vmlinuz.efi; make -j8 modules with the defconfig or kernel-riscv64-fedora.config from Fedora.
I have never produced it with defconfig+btf no matter what I pass to make, but defconfig is around 100 modules and Fedora is something like 4100, so I suppose it may take several tries with defconfig to reproduce the error if it's a game of numbers and I have not tried much.
I have never reproduced it trying to build with Bianbu userspace, but I haven't tried this much either. I want to try some more builds to see if I can cause the error to occur in Bianbu. If we can't make it occur in Bianbu I feel it must be a bug in a version of something in Fedora rather than memory corruption or something else along those lines.
kernel seems irrelevant. We've boot into Bianbu with 6.18.3 and seen it in a podman containers, I've seen it in a Fedora chroot booted in Bianbu and I've seen it booted into Fedora with the 7.1-rc6 omni kernel.
I had some successful builds just omitting -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-hardened-ld-errors (-specs=/usr/lib/rpm/redhat/redhat-hardened-ld basically sets HOSTLDFLAGS=-pie IIUC) and was again able to reproduce failure several times simplifying the make flags to make -s 'HOSTCFLAGS=-O2 -fPIE' 'HOSTLDFLAGS=-pie' -j8 vmlinuz.efi; make -s 'HOSTCFLAGS=-O2 -fPIE' 'HOSTLDFLAGS=-pie' -j8 modules.
Perhaps most frustratingly, after several builds reproducing it I had suddenly had several successes.
I am suspicious of memory corruption or maybe reading from storage. Today I ran rpmbuild -bb to start a kernel build and a patch failed to apply on the first run:

+ git --work-tree=. apply
error: corrupt patch at line 22694
error: Bad exit status from /var/tmp/rpm-tmp.R6UyFf (%prep)

RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.R6UyFf (%prep)

The patch was generated via the kernel-ark scripts so there wasn't really room for manual error in generating it and anyway simply rerunning rpmbuild -bb a moment later it applied successfully.

At this point I don't really feel I have a grasp of what's actually causing the error. All I can provide are my observations. - To date I have never managed to reproduce it running just `make -j8 vmlinuz.efi; make -j8 modules` with the defconfig or kernel-riscv64-fedora.config from Fedora. - I have never produced it with defconfig+btf no matter what I pass to make, but defconfig is around 100 modules and Fedora is something like 4100, so I suppose it may take several tries with defconfig to reproduce the error if it's a game of numbers and I have not tried much. - I have never reproduced it trying to build with Bianbu userspace, but I haven't tried this much either. I want to try some more builds to see if I can cause the error to occur in Bianbu. If we can't make it occur in Bianbu I feel it must be a bug in a version of something in Fedora rather than memory corruption or something else along those lines. - kernel seems irrelevant. We've boot into Bianbu with 6.18.3 and seen it in a podman containers, I've seen it in a Fedora chroot booted in Bianbu and I've seen it booted into Fedora with the 7.1-rc6 omni kernel. - I had some successful builds just omitting `-specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-hardened-ld-errors` (-specs=/usr/lib/rpm/redhat/redhat-hardened-ld basically sets HOSTLDFLAGS=-pie IIUC) and was again able to reproduce failure several times simplifying the make flags to `make -s 'HOSTCFLAGS=-O2 -fPIE' 'HOSTLDFLAGS=-pie' -j8 vmlinuz.efi; make -s 'HOSTCFLAGS=-O2 -fPIE' 'HOSTLDFLAGS=-pie' -j8 modules`. - Perhaps most frustratingly, after several builds reproducing it I had suddenly had several successes. - I am suspicious of memory corruption or maybe reading from storage. Today I ran rpmbuild -bb to start a kernel build and a patch failed to apply on the first run: ``` + git --work-tree=. apply error: corrupt patch at line 22694 error: Bad exit status from /var/tmp/rpm-tmp.R6UyFf (%prep) RPM build errors: Bad exit status from /var/tmp/rpm-tmp.R6UyFf (%prep) ``` The patch was generated via the kernel-ark scripts so there wasn't really room for manual error in generating it and anyway simply rerunning rpmbuild -bb a moment later it applied successfully.

Rows
Columns

Investigate libbpf errors when building kernels on K3 hardware. #20

Intervention on K3: disable BTF kernel module