reinstall some buildhw-x86's #13556

New issue

Closed

opened 2026-09-11 17:26:25 +00:00 by kevin · 7 comments

kevin commented

2026-09-11 17:26:25 +00:00

Owner

Description of request

We need to reinstall some of the buildhw-x86 boxes.

There's an issue with inodes. btrfs allocates inodes starting from 1 and increasing for each new node.
Eventually, it gets over 4G and then when a i686 (32bit) build tries to run on them, it fails in odd ways due to not understanding the inode numbers.

The following should probably be reinstalled:

I disable buildhw-x86-04 the other day because it's over. 08 is close, the others less so...

buildhw-x86-04.rdu3.fedoraproject.org | CHANGED | rc=0 | (stdout) 4317851321 /var/tmp/inodes
buildhw-x86-08.rdu3.fedoraproject.org | CHANGED | rc=0 | (stdout) 4175997817 /var/tmp/inodes
buildhw-x86-13.rdu3.fedoraproject.org | CHANGED | rc=0 | (stdout) 4061276540 /var/tmp/inodes
buildhw-x86-09.rdu3.fedoraproject.org | CHANGED | rc=0 | (stdout) 3998894349 /var/tmp/inodes
buildhw-x86-10.rdu3.fedoraproject.org | CHANGED | rc=0 | (stdout) 3928385235 /var/tmp/inodes
buildhw-x86-12.rdu3.fedoraproject.org | CHANGED | rc=0 | (stdout) 3693624310 /var/tmp/inodes
buildhw-x86-02.rdu3.fedoraproject.org | CHANGED | rc=0 | (stdout) 3074760210 /var/tmp/inodes

### Description of request We need to reinstall some of the buildhw-x86 boxes. There's an issue with inodes. btrfs allocates inodes starting from 1 and increasing for each new node. Eventually, it gets over 4G and then when a i686 (32bit) build tries to run on them, it fails in odd ways due to not understanding the inode numbers. The following should probably be reinstalled: I disable buildhw-x86-04 the other day because it's over. 08 is close, the others less so... ``` buildhw-x86-04.rdu3.fedoraproject.org | CHANGED | rc=0 | (stdout) 4317851321 /var/tmp/inodes buildhw-x86-08.rdu3.fedoraproject.org | CHANGED | rc=0 | (stdout) 4175997817 /var/tmp/inodes buildhw-x86-13.rdu3.fedoraproject.org | CHANGED | rc=0 | (stdout) 4061276540 /var/tmp/inodes buildhw-x86-09.rdu3.fedoraproject.org | CHANGED | rc=0 | (stdout) 3998894349 /var/tmp/inodes buildhw-x86-10.rdu3.fedoraproject.org | CHANGED | rc=0 | (stdout) 3928385235 /var/tmp/inodes buildhw-x86-12.rdu3.fedoraproject.org | CHANGED | rc=0 | (stdout) 3693624310 /var/tmp/inodes buildhw-x86-02.rdu3.fedoraproject.org | CHANGED | rc=0 | (stdout) 3074760210 /var/tmp/inodes ```

gwmngilfen commented

2026-09-14 08:52:36 +00:00

Member

I guess we should add a Zabbix check for this too?

kevin commented

2026-09-14 17:47:45 +00:00

Author

Owner

Yes, that would be lovely. It's only the x86 builders... the check could touch a file, check it's inode number and then delete it.

predictive failure would be super great for this too! (buildvm-x86-12 will pass 4GB of inodes in 16 days or whatever)

Yes, that would be lovely. It's only the x86 builders... the check could touch a file, check it's inode number and then delete it. predictive failure would be super great for this too! (buildvm-x86-12 will pass 4GB of inodes in 16 days or whatever)

james added the

labels

2026-09-14 19:05:29 +00:00

james commented

2026-09-15 02:49:25 +00:00

Member

I did buildhw-x86-04 ... had a little bit of "fun" due to dhcp config. thinking it was autosign02.

Also had to manually install python3-libdnf5 or package requests blew up in ansible.

I did `buildhw-x86-04` ... had a little bit of "fun" due to dhcp config. thinking it was `autosign02`. Also had to manually install `python3-libdnf5` or package requests blew up in ansible.

kevin commented

2026-09-15 20:46:14 +00:00

Author

Owner

We should adjust our kickstarts to always install python3-libdnf5