infra/tickets

Fork 2

Low disk space on ns* hosts #13548

New issue

Closed

opened 2026-09-07 11:07:03 +00:00 by gwmngilfen · 8 comments

gwmngilfen commented

2026-09-07 11:07:03 +00:00

Member

Description of request

We got an 80% warning for ns01 today, but looking into it, they're all slowly running out of space. It seems like it was stable until around the start of August, and it's been steadily rising ever since:

They won't run out imminently, but we should investigate and fix the issue before they do.

### Description of request We got an 80% warning for ns01 today, but looking into it, they're all slowly running out of space. It seems like it was stable until around the start of August, and it's been steadily rising ever since: ![image](/attachments/05e01d84-2bb4-4e07-8d27-a371b5ea3288) They won't run out imminently, but we should investigate and fix the issue before they do.

image.png

42 KiB

gwmngilfen added the

labels

2026-09-07 11:07:03 +00:00

james commented

2026-09-07 16:29:03 +00:00

Member

I can't see anything obvious that changed, although the /etc/logrotate.d dir. being timestamped Aug 1st and /var/log/messages rotations being uncompressed and 1-2GB each is a possible hint at a solution if nothing else.

I can't see anything obvious that changed, although the `/etc/logrotate.d` dir. being timestamped Aug 1st and `/var/log/messages` rotations being uncompressed and 1-2GB each is a possible hint at a solution if nothing else.

gwmngilfen commented

2026-09-08 08:15:12 +00:00

Author

Member

yeah, I saw the logs looked a bit large - I wonder if it's nothing more than increased DNS traffic that accounts for the space use.

kevin commented

2026-09-08 16:47:58 +00:00

Owner

Yeah, its definitely logs...

There's a ton of firewall messages:

Sep 6 00:28:39 ns01 kernel: IN=eth0 OUT= MAC=52:54:00:e4:7e:47:02:00:5e:00:01:01:08:00 SRC= DST=10.16.163.33 LEN=74 TOS=0x00 PREC=0x00 TTL=44 ID=58669 PROTO=UDP SPT=51586 DPT=53 LEN=54

What is doing those? I don't see anything in nftables... but those are a) useless and b) very noisy. ;)

Yeah, its definitely logs... There's a ton of firewall messages: Sep 6 00:28:39 ns01 kernel: IN=eth0 OUT= MAC=52:54:00:e4:7e:47:02:00:5e:00:01:01:08:00 SRC=<redact> DST=10.16.163.33 LEN=74 TOS=0x00 PREC=0x00 TTL=44 ID=58669 PROTO=UDP SPT=51586 DPT=53 LEN=54 What is doing those? I don't see anything in nftables... but those are a) useless and b) very noisy. ;)

james commented

2026-09-08 18:20:12 +00:00

Member

I tried to work out where the rsyslog file for logrotate come from. Traced through a bunch of ansible, and eventually realized the obvious it's the default file from the rsyslog pkg.

Yeh, I saw those ... they are from the ip_country filters which all have log in their rules. The counts on the rules don't seem that big, so I didn't think it was a bunch more than normal (but not sure), but obviously the text logs expand the size a lot (also the fact we don't compress them). I tried to work out where the rsyslog file for logrotate come from. Traced through a bunch of ansible, and eventually realized the obvious **it's the default file from the rsyslog pkg**.

james added a new dependency

2026-09-08 20:29:55 +00:00

infra/ansible#3666 - ns0*: Use proxy rsyslog logrotate file, to compress.

james commented

2026-09-08 20:31:20 +00:00

Member

I created a PR+FBR for the log file compression. Stopping the firewall logging requires changes to the script in the private repo. though, so maybe leave until post freeze ... or do the FBR for it here?

kevin commented

2026-09-08 21:23:05 +00:00

Owner

Well, the compress pr has enough votes... so I guess we can try that first and if it's still a problem look at removing the log messages?

and we can remove them after freeze in any case.

Well, the compress pr has enough votes... so I guess we can try that first and if it's still a problem look at removing the log messages? and we can remove them after freeze in any case.

gwmngilfen commented

2026-09-10 09:05:19 +00:00

Author

Member

This is looking good, I think:

-rw-------. 1 root   root             57M Sep 10 09:03 messages
-rw-------. 1 root   root            1.4G Aug 16 00:25 messages-20260816
-rw-------. 1 root   root            1.6G Aug 23 00:17 messages-20260823
-rw-------. 1 root   root            1.9G Aug 30 00:27 messages-20260830
-rw-------. 1 root   root            2.3G Sep  6 00:28 messages-20260906
-rw-------. 1 root   root             41M Sep  9 00:31 messages-20260909.xz
-rw-------. 1 root   root             13M Sep 10 00:42 messages-20260910.xz

41M vs 2.3G is much better, as expected 😁 - and disk use is falling:

I think we should keep this open for another week or so to check, and then close if it's still good.

This is looking good, I think: ``` -rw-------. 1 root root 57M Sep 10 09:03 messages -rw-------. 1 root root 1.4G Aug 16 00:25 messages-20260816 -rw-------. 1 root root 1.6G Aug 23 00:17 messages-20260823 -rw-------. 1 root root 1.9G Aug 30 00:27 messages-20260830 -rw-------. 1 root root 2.3G Sep 6 00:28 messages-20260906 -rw-------. 1 root root 41M Sep 9 00:31 messages-20260909.xz -rw-------. 1 root root 13M Sep 10 00:42 messages-20260910.xz ``` 41M vs 2.3G is much better, as expected 😁 - and disk use is falling: ![image](/attachments/0b4b65a7-5a42-4b1a-bb4c-612ba11aa9ed) I think we should keep this open for another week or so to check, and then close if it's still good.

image.png

71 KiB

👍 1

gwmngilfen commented

2026-09-17 12:12:22 +00:00

Author

Member

Seems to have stabilised, here's ns01 over 30 days. Closing this as complete.