However, this is per-vhost, there is no global status equivalent to /apache-status?auto, so we'll have to add this to the VHost definitions for balancers. Annoying, but not too bad. I tested on forge.stg.fpo using this (indented lines were already present)

  <Proxy "balancer://apps-ocp">
  BalancerMember "https://worker01.ocp.stg.rdu3.fedoraproject.org"
  BalancerMember "https://worker02.ocp.stg.rdu3.fedoraproject.org"
  BalancerMember "https://worker03.ocp.stg.rdu3.fedoraproject.org"
  </Proxy>
ProxyPass /balancer-manager !
ProxyPass /apache-status !
  ProxyPass / "balancer://apps-ocp/"
  ProxyPassReverse / "balancer://apps-ocp/"

<Location "/balancer-manager">
SetHandler balancer-manager
Require local
</Location>

This allowed me to curl the balancer-manager page locally using a host header: curl -k -H "Host: forge.stg.fedoraproject.org" 'https://127.0.0.1/balancer-manager' which works, but outputs raw HTML (ugh). I copied to my machine and rendered it locally, see attached screenshot.

So, it is possible to do this, via:

Updating the various Apache configs to include a balancer-manager location
Figuring out how to tell Zabbix what services to inspect, likely via an Ansible-managed discovery file
Figuring out how to parse the output so Zabbix can get useful metrics from it

Not fun, but doable

The alternative is to rewrite our configs. I've not tested this, but clanker discussions and research suggests that moving the balancer definitions out of the vhosts should mean we can have a single global /balancer-manager page. It would look something like:

# Defined globally, outside of any VirtualHost block
<Proxy "balancer://apps-ocp">
BalancerMember "https://worker01.ocp.stg.rdu3.fedoraproject.org"
BalancerMember "https://worker02.ocp.stg.rdu3.fedoraproject.org"
BalancerMember "https://worker03.ocp.stg.rdu3.fedoraproject.org"
</Proxy>

# Virtual Hosts simply reference the global name:
<VirtualHost *:443>
    ServerName mysite.foo
    ProxyPass / balancer://apps-ocp/
</VirtualHost>

Given we generally use the same nodes for all the balanced apps, this might be feasible. Should I test this path before we make a decision? If it works I prefer the latter...

OK, I started digging into this.... Firstly, the BalancerManager is the place to get data from - see https://httpd.apache.org/docs/current/mod/mod_proxy_balancer.html#balancer_manager However, this is **per-vhost**, there is no global status equivalent to `/apache-status?auto`, so we'll have to add this to the VHost definitions for balancers. Annoying, but not too bad. I tested on forge.stg.fpo using this (indented lines were already present) ``` <Proxy "balancer://apps-ocp"> BalancerMember "https://worker01.ocp.stg.rdu3.fedoraproject.org" BalancerMember "https://worker02.ocp.stg.rdu3.fedoraproject.org" BalancerMember "https://worker03.ocp.stg.rdu3.fedoraproject.org" </Proxy> ProxyPass /balancer-manager ! ProxyPass /apache-status ! ProxyPass / "balancer://apps-ocp/" ProxyPassReverse / "balancer://apps-ocp/" <Location "/balancer-manager"> SetHandler balancer-manager Require local </Location> ``` This allowed me to curl the balancer-manager page locally using a host header: `curl -k -H "Host: forge.stg.fedoraproject.org" 'https://127.0.0.1/balancer-manager'` which works, but outputs raw HTML (ugh). I copied to my machine and rendered it locally, see attached screenshot. So, it is *possible* to do this, via: - Updating the various Apache configs to include a balancer-manager location - Figuring out how to tell Zabbix what services to inspect, likely via an Ansible-managed discovery file - Figuring out how to parse the output so Zabbix can get useful metrics from it Not fun, but doable --- The alternative is to rewrite our configs. I've not tested this, but clanker discussions and research suggests that moving the balancer definitions *out* of the vhosts should mean we can have a single global /balancer-manager page. It would look something like: ``` # Defined globally, outside of any VirtualHost block <Proxy "balancer://apps-ocp"> BalancerMember "https://worker01.ocp.stg.rdu3.fedoraproject.org" BalancerMember "https://worker02.ocp.stg.rdu3.fedoraproject.org" BalancerMember "https://worker03.ocp.stg.rdu3.fedoraproject.org" </Proxy> # Virtual Hosts simply reference the global name: <VirtualHost *:443> ServerName mysite.foo ProxyPass / balancer://apps-ocp/ </VirtualHost> ``` Given we generally use the same nodes for all the balanced apps, this might be feasible. Should I test this path before we make a decision? If it works I prefer the latter...

gwmngilfen added

points

and removed

points

labels

2026-08-26 10:00:53 +00:00

kevin commented

2026-08-26 21:46:09 +00:00

Owner

Hum... yeah, if we do 'per balancer' tho all the openshift apps will be lumped into one bucket right?

So, for me it would be better if they were per virthost so we could split them out and see if say bodhi was getting a lot of traffic, but koschei wasn't, etc.

Or does the global one also report per virthost, just on the same page?

Hum... yeah, if we do 'per balancer' tho all the openshift apps will be lumped into one bucket right? So, for me it would be better if they were per virthost so we could split them out and see if say bodhi was getting a lot of traffic, but koschei wasn't, etc. Or does the global one also report per virthost, just on the same page?

gwmngilfen commented

2026-08-27 08:21:40 +00:00

Author

Member

I think we can do per-virthost, so long as the actual definitions are outside the virthost. I 'll test it on STG, I can't see any harm if the changes are in managed files, we can always re-run the playbook

I *think* we can do per-virthost, so long as the actual definitions are outside the virthost. I 'll test it on STG, I can't see any harm if the changes are in managed files, we can always re-run the playbook

gwmngilfen commented

2026-08-27 14:27:31 +00:00

Author

Member

Alright, I did some experiments on staging, and this seems fairly achievable. Basically, it's:

Add a balancer-manager location to apachestatus.conf
Move the Proxy block (balancer name & members) to the top of the main .conf (so it's outside the VirtualHost)
Rename the balancer inside the virthost

This means we can keep our existing loops/tasks over the list of ocp apps, and just change the hardcoded "ocp-apps" balancer to "$virthost-balancer" or something. Here's the actial change I made to both proxy01.stg and proxy02.stg

+++ apachestatus.conf
(add to end)
<Location "/balancer-status">
    SetHandler balancer-manager
    Require ip 127.0.0.1 ::1
    ProxyPass !
</Location>

+++ forge.stg.fedoraproject.org.conf
(add at start)
<Proxy "balancer://forge-stg">
BalancerMember "https://worker01.ocp.stg.rdu3.fedoraproject.org"
BalancerMember "https://worker02.ocp.stg.rdu3.fedoraproject.org"
BalancerMember "https://worker03.ocp.stg.rdu3.fedoraproject.org"
</Proxy>

+++ forge.stg.fedoraproject.org/forge.conf
- <Proxy "balancer://apps-ocp">
- BalancerMember "https://worker01.ocp.stg.rdu3.fedoraproject.org"
- BalancerMember "https://worker02.ocp.stg.rdu3.fedoraproject.org"
- BalancerMember "https://worker03.ocp.stg.rdu3.fedoraproject.org"
- </Proxy>
- ProxyPass / "balancer://apps-ocp/"
- ProxyPassReverse / "balancer://apps-ocp/"
+ ProxyPass / "balancer://forge-stg/"
+ ProxyPassReverse / "balancer://forge-stg/"

Here's a screenshot of the rendered HTML for curl -kL 'http://127.0.0.1/balancer-status'

That still doesn't make it super parseable by Zabbix, but I guess we can something with the <td> entries ...

Alright, I did some experiments on staging, and this seems *fairly* achievable. Basically, it's: - Add a balancer-manager location to apachestatus.conf - Move the Proxy block (balancer name & members) to the top of the main <vhostname>.conf (so it's outside the VirtualHost) - Rename the balancer inside the virthost This means we can keep our existing loops/tasks over the list of ocp apps, and just change the hardcoded "ocp-apps" balancer to "$virthost-balancer" or something. Here's the actial change I made to both proxy01.stg and proxy02.stg ``` +++ apachestatus.conf (add to end) <Location "/balancer-status"> SetHandler balancer-manager Require ip 127.0.0.1 ::1 ProxyPass ! </Location> +++ forge.stg.fedoraproject.org.conf (add at start) <Proxy "balancer://forge-stg"> BalancerMember "https://worker01.ocp.stg.rdu3.fedoraproject.org" BalancerMember "https://worker02.ocp.stg.rdu3.fedoraproject.org" BalancerMember "https://worker03.ocp.stg.rdu3.fedoraproject.org" </Proxy> +++ forge.stg.fedoraproject.org/forge.conf - <Proxy "balancer://apps-ocp"> - BalancerMember "https://worker01.ocp.stg.rdu3.fedoraproject.org" - BalancerMember "https://worker02.ocp.stg.rdu3.fedoraproject.org" - BalancerMember "https://worker03.ocp.stg.rdu3.fedoraproject.org" - </Proxy> - ProxyPass / "balancer://apps-ocp/" - ProxyPassReverse / "balancer://apps-ocp/" + ProxyPass / "balancer://forge-stg/" + ProxyPassReverse / "balancer://forge-stg/" ``` Here's a screenshot of the rendered HTML for `curl -kL 'http://127.0.0.1/balancer-status'` ![image](/attachments/cb3c9f3a-6af0-494b-95fe-49f76ce5da14) --- That still doesn't make it *super* parseable by Zabbix, but I guess we can something with the `<td>` entries ...

image.png

131 KiB

kevin commented

2026-08-31 17:24:12 +00:00

Owner

Yeah, the info there doesn't look super great either... just 'to' and 'from'. I guess thats requests?

gwmngilfen commented

2026-09-01 09:24:01 +00:00

Author

Member

Yeah, it's not great, but I think we can work with it, if we decide to.

The interesting info appears to be:

Status: obvious, we'd want to know if a worker dropped out, right? This has about 8 states a node can be in, such as Drain or HotSpare
Elected: this shows how many times the worker has been picked, so we can see the round-robin is working here
Busy: the number of active connections to the worker
Load: is somewhat arbitrary, it just tracks which worker is next (it always sums to 0, so at next pick you select the highest, add 100 to everything and subtract 300 from the chosen one)
To/From is bytes - this was surprisingly hard to find, I had to go read the source code to confirm it. I assume the counters reset when Apache is restarted, but it seems useful to check things are actually balanced
HC is HealthCheck, which we're not doing here but should probably consider.

That seems enough to track things like failed backends (assuming we define a HC), problem workers, and balancing info. The only wrinkle is parsing the data in Zabbix, but I think I can do that.

Shall I work up a PR we can try in stg?

Yeah, it's not great, but I think we can work with it, if we decide to. The interesting info appears to be: - Status: obvious, we'd want to know if a worker dropped out, right? This has about 8 states a node can be in, such as Drain or HotSpare - Elected: this shows how many times the worker has been picked, so we can see the round-robin is working here - Busy: the number of active connections to the worker - Load: is somewhat arbitrary, it just tracks which worker is next (it always sums to 0, so at next pick you select the highest, add 100 to everything and subtract 300 from the chosen one) - To/From is bytes - this was surprisingly hard to find, I had to go read the [source code](https://github.com/omnigroup/Apache/blob/master/httpd/modules/proxy/mod_proxy_balancer.c#L1503-L1528) to confirm it. I assume the counters reset when Apache is restarted, but it seems useful to check things are actually balanced - HC is HealthCheck, which we're not doing here but should probably consider. That seems enough to track things like failed backends (assuming we define a HC), problem workers, and balancing info. The only wrinkle is parsing the data in Zabbix, but I think I can do that. Shall I work up a PR we can try in stg?

kevin commented

2026-09-01 22:09:24 +00:00

Owner

yeah, thats nearly not the interesting info we get from haproxy. ;(

Lets ponder on it, and perhaps discuss in the meeting thursday.

We may want to see how hard it would be to just convert all the apache balancers over to haproxy and reuse the haproxy monitoring on them. The hard part is the seperate port for each service.

yeah, thats nearly not the interesting info we get from haproxy. ;( Lets ponder on it, and perhaps discuss in the meeting thursday. We may want to see how hard it would be to just convert all the apache balancers over to haproxy and reuse the haproxy monitoring on them. The hard part is the seperate port for each service.

👍 1

gwmngilfen modified the project from Sprint ( 2026-08-24 to 2026-09-06) to Current Sprint ( 2026-09-07 to 2026-09-20)

2026-09-09 14:13:12 +00:00

gwmngilfen commented

2026-09-09 14:15:50 +00:00

Author

Member

Looks like we're not going to go this way - the change is large and the return on it is not great.

I'll close this, but I'll open a new spike for looking at migrating things to HAProxy, and refer back here.

Looks like we're not going to go this way - the change is large and the return on it is not great. I'll close this, but I'll open a new spike for looking at migrating things to HAProxy, and refer back here.

gwmngilfen closed this issue

2026-09-09 14:15:50 +00:00

gwmngilfen referenced this issue

2026-09-09 14:21:11 +00:00

Spike: look into how we might migrate things from Apache LoadBalancer to HAProxy #13553