infra/tickets

Fork 1

Create Ansible/Git-based CMDB system #12817

New issue

Open

opened 2025-09-25 13:01:30 +00:00 by gwmngilfen · 9 comments

gwmngilfen commented

2025-09-25 13:01:30 +00:00

Member

Problem: The Fedora Infrastructure CMDB doesn't really exist in a single place - we have some information like MAC addresses, IPs etc in Ansible, others in Red Hat internal spreadsheets, and more in other places. This causes issues:

it's evolved over time, leading to confusion over what is tracked where or is even necessary
it's hard to understand / update
it's internal only for somethings
even for RH team members, it's hard to discover
there's no changelog / change tracking / accountability
there's no easy way to make reports on it (eg for warranty info)

In short, this isn't really fit for purpose. We should do something better.

Analysis of current data

What's in the internal spreadsheet? Reordering by "type of data", you might group it like this:

hardware stuff
- manufacturer & model
- serial
- purchase date
- under warranty?
- warranty type
- warranty renew / end / eol (dates)
- in-use: yes/no
- datacenter / rack / pdu - (possibly out of date?)
software stuff
- hostname
- arch / cores / mem / disk
- mgmt mac / vlan / ip
- other nic mac/vlan
- usage - this is effectively the ansible group anyway

How could we track / use this better?

Proposal

Let's use Ansible as the CMDB-ish thing that stores all this. What might that look like?

we already have some of this data (notably NIC MACs, BMC data, and IP addresses) in the Ansible host_vars, and other stuf is visible in the facts cache (eg cpus, etc). We could extend this (and possibly consolidate it under one key) which would place all the required info in one file (per host) and under Git control (for audit/historical servers)

Consderations

Security is question one, but in conversation with managers, the current thinking is "no, it's not". The only things I think might be worth thinking about are serial & purchase info / warranty info, and I can't see how you make a scam out of that.

Second, part of the goal here is to make the data more actionable, especially warranties. I can see several routes here

Having the data in Ansible means we can populate the Zabbix inventory with the data
- That makes it possible to run search queries in the UI (eg, Dell R650 hosts)
- It also means we can perhaps build triggers to send alerts when warranties are due to expire, and other similar things
Reporting is also possible, either from the Zabbix UI again (it can build reports), or by running some script over the Ansible inventory directly
This would be compatible with CentOS too, since that is also Ansible and Zabbix based infra.

Other options

Do we have any alternatives?

We could use a new sheet, but that largely risks repeating the mistakes of the past, is internal, etc
We could build something out of the facts_cache, but that seems more brittle than using the inventory, especially as we'd still have to deploy some custom facts for non-discoverable things like purchase dates
We could put it all in the Ansible-private repo, but that means it can't go in Zabbix (since thats open to all)

Implementation

As it stands now, the two infras (Fedora & CentOS) have different structures in their inventory for how to track this stuff. Rather than "pick a winner", I feel it's fairer to change both 😀

More seriously, I do think it would make sense to have this under a single key, eg:

hardware:
  serial: ABCDEF123
  purchase_date: 2025-09-01
  warranty_duration: 1095 # 3 years in days
  warranty_start: 2025-09-05
  warranty_type: next-day

Having it all under {{ hardware.* }} makes it easy to know where to look for things, and stops it getting distributed all over the host_vars file (which is definitely an issue for some of the hosts with lots of vars). We could start with things not currently tracked, such as warranty data, and can then look at migrating existing vars (eg, mac0-3, bmc, etc) to this key later on.

For the Zabbix side, we can definitely do this via Ansible, as https://www.zabbix.com/documentation/current/en/manual/api/reference/host/object#host-inventory is the possible list of inventory keys, and https://docs.ansible.com/ansible/latest/collections/community/zabbix/zabbix_host_module.html#id4 has examples of sending those keys

Next steps

This is very much the high-level epic. I think we'll need to break this down into action items, including, but not limited to:

initial addition of the "hardware" key to the existing host_vars
migration of existing keys to the hardware key (and resulting code changes elsewhere
setting up population of the Zabbix inventory
designing triggers in Zabbix to warn on harware items (eg expiring warranty)
desgining reporting systems on the hadrware info (via Zabbix or CLI scripts)

Problem: The Fedora Infrastructure CMDB doesn't really exist in a single place - we have *some* information like MAC addresses, IPs etc in Ansible, others in Red Hat internal spreadsheets, and more in other places. This causes issues: - it's evolved over time, leading to confusion over what is tracked where or is even necessary - it's hard to understand / update - it's internal only for somethings - even for RH team members, it's hard to discover - there's no changelog / change tracking / accountability - there's no easy way to make reports on it (eg for warranty info) In short, this isn't really fit for purpose. We should do something better. ### Analysis of current data What's in the internal spreadsheet? Reordering by "type of data", you might group it like this: - hardware stuff - manufacturer & model - serial - purchase date - under warranty? - warranty type - warranty renew / end / eol (dates) - in-use: yes/no - datacenter / rack / pdu - (possibly out of date?) - software stuff - hostname - arch / cores / mem / disk - mgmt mac / vlan / ip - other nic mac/vlan - usage - this is effectively the ansible group anyway How could we track / use this better? ### Proposal Let's use Ansible as the CMDB-ish thing that stores all this. What might that look like? - we already have some of this data (notably NIC MACs, BMC data, and IP addresses) in the Ansible host_vars, and other stuf is visible in the facts cache (eg cpus, etc). We could extend this (and possibly consolidate it under one key) which would place all the required info in one file (per host) and under Git control (for audit/historical servers) ### Consderations Security is question one, but in conversation with managers, the current thinking is "no, it's not". The only things I think might be worth thinking about are serial & purchase info / warranty info, and I can't see how you make a scam out of that. Second, part of the goal here is to make the data more actionable, especially warranties. I can see several routes here - Having the data in Ansible means we can populate the Zabbix inventory with the data - That makes it possible to run search queries in the UI (eg, Dell R650 hosts) - It also means we can perhaps build triggers to send alerts when warranties are due to expire, and other similar things - Reporting is also possible, either from the Zabbix UI again (it can build reports), or by running some script over the Ansible inventory directly - This would be compatible with CentOS too, since that is also Ansible and Zabbix based infra. ### Other options Do we have any alternatives? - We could use a new sheet, but that largely risks repeating the mistakes of the past, is internal, etc - We could build something out of the facts_cache, but that seems more brittle than using the inventory, especially as we'd still have to deploy some custom facts for non-discoverable things like purchase dates - We could put it all in the Ansible-private repo, but that means it can't go in Zabbix (since thats open to all) ### Implementation As it stands now, the two infras (Fedora & CentOS) have different structures in their inventory for how to track this stuff. Rather than "pick a winner", I feel it's fairer to change both :grinning: More seriously, I do think it would make sense to have this under a single key, eg: ```yaml hardware: serial: ABCDEF123 purchase_date: 2025-09-01 warranty_duration: 1095 # 3 years in days warranty_start: 2025-09-05 warranty_type: next-day ``` Having it all under `{{ hardware.* }}` makes it easy to know where to look for things, and stops it getting distributed all over the host_vars file (which is definitely an issue for some of the hosts with lots of vars). We could start with things not currently tracked, such as warranty data, and can then look at migrating existing vars (eg, mac0-3, bmc, etc) to this key later on. For the Zabbix side, we can definitely do this via Ansible, as https://www.zabbix.com/documentation/current/en/manual/api/reference/host/object#host-inventory is the possible list of inventory keys, and https://docs.ansible.com/ansible/latest/collections/community/zabbix/zabbix_host_module.html#id4 has examples of sending those keys # Next steps This is very much the high-level epic. I think we'll need to break this down into action items, including, but not limited to: - initial addition of the "hardware" key to the existing host_vars - migration of existing keys to the hardware key (and resulting code changes elsewhere - setting up population of the Zabbix inventory - designing triggers in Zabbix to warn on harware items (eg expiring warranty) - desgining reporting systems on the hadrware info (via Zabbix or CLI scripts)

kevin commented

2025-09-25 18:43:06 +00:00

Owner

This looks great to me. I'm happy to help work on it too.

I think additionally we could nuke the inventory/hardware file because we can determine it via the existance of a hardware: key or not.

The serial is already in facts, but I guess duplicating it makes it easier to work with as a variable.

This looks great to me. I'm happy to help work on it too. I think additionally we could nuke the inventory/hardware file because we can determine it via the existance of a hardware: key or not. The serial is already in facts, but I guess duplicating it makes it easier to work with as a variable.

phsmoura commented

2025-09-25 19:05:14 +00:00

Member

Metadata Update from @phsmoura:

Issue priority set to: Waiting on Assignee (was: Needs Review)
Issue tagged with: medium-gain, medium-trouble, ops

**Metadata Update from @phsmoura**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble, ops

gwmngilfen commented

2025-10-03 12:38:38 +00:00

Author

Member

Metadata Update from @gwmngilfen:

Issue tagged with: backlog

**Metadata Update from @gwmngilfen**: - Issue tagged with: backlog

kevin commented

2026-03-05 17:28:52 +00:00

Owner

So, there has been progress here. A initial workflow has been figured out and @jbley is going to see what information can be migrated from the old spreadsheet here. After we do a few of these and get the data needed well known, we can spread this out to other interested folks to contibute to.

patrikp added

and removed

labels

2026-03-05 17:33:30 +00:00

jbley commented

2026-03-06 07:22:50 +00:00

I'll work with @gwmngilfen on getting info on the missing information (where applicable).

gwmngilfen commented

2026-03-06 10:06:24 +00:00

Author

Member

@jbley has opened infra/ansible#3197 which I'll review next week.

@jbley has opened https://forge.fedoraproject.org/infra/ansible/pulls/3197 which I'll review next week.

gwmngilfen commented

2026-05-28 15:47:31 +00:00

Author

Member

still in progress, I've been trying to finish up the Nagios migration first, then this and a bunch of other cleanup can get done

james commented

2026-05-28 17:40:17 +00:00

Member

Is there a reason to do "warranty_start + warranty_days" instead of just "warranty_end"?

james commented

2026-05-29 21:16:03 +00:00

Member

FWIW I updated my script which queries the ansible fact cache data, to get just the relevant HW things out of it. Eg. (run on batcave in ~root)

./af2cmdb.sh .ansible_facts_cache/vmhost-x86-01.rdu3.fedoraproject.org

FWIW I updated my script which queries the ansible fact cache data, to get just the relevant HW things out of it. Eg. (run on batcave in ~root) `./af2cmdb.sh .ansible_facts_cache/vmhost-x86-01.rdu3.fedoraproject.org`