Port the bodhi puppet module to ansible

lmacken commented

2014-01-24 02:27:51 +00:00

The bodhi puppet module needs to be ported to ansible. We should probably do it as-is, and then make the necessary changes for bodhi 2.0 after.

janeznemanic commented

2014-02-02 03:03:44 +00:00

Here is what I have done by now. I tried to port bodhi from puppet to ansible in a way that would resemble current puppet classes. I added a new role named bodhi with few nested roles within it. So at the moment it looks like this:

roles/bodhi/

roles/bodhi/app/

roles/bodhi/masher/

roles/bodhi/masher/jobrunner/

roles/bodhi/masher/epelmasher/

With such a roles structure I tried to imitate the puppet's class inheritance. So for example if you would like to include role epelmasher in a playbook you would include the following roles:

roles/bodhi

roles/bodhi/masher

roles/bodhi/masher/epelmasher.

Basically you just include all roles on the way to epelmasher and you just need to know the path. So that is how things are at the moment.

The second alternative I had in mind was to have the following roles:

roles/bodhi/base

roles/bodhi/masher

with a few extra variables in playbooks to control tasks.
What do you think guys? Give me as much feedback as possible. To me it doesn't matter if we keep my current solution or if we use something else, just tell what you think, what would you prefer and what you want me to do.

Here is what I have done by now. I tried to port bodhi from puppet to ansible in a way that would resemble current puppet classes. I added a new role named bodhi with few nested roles within it. So at the moment it looks like this: roles/bodhi/ roles/bodhi/app/ roles/bodhi/masher/ roles/bodhi/masher/jobrunner/ roles/bodhi/masher/epelmasher/ With such a roles structure I tried to imitate the puppet's class inheritance. So for example if you would like to include role epelmasher in a playbook you would include the following roles: roles/bodhi roles/bodhi/masher roles/bodhi/masher/epelmasher. Basically you just include all roles on the way to epelmasher and you just need to know the path. So that is how things are at the moment. The second alternative I had in mind was to have the following roles: roles/bodhi/base roles/bodhi/masher with a few extra variables in playbooks to control tasks. What do you think guys? Give me as much feedback as possible. To me it doesn't matter if we keep my current solution or if we use something else, just tell what you think, what would you prefer and what you want me to do.

kevin commented

2014-02-03 01:11:40 +00:00

Owner

So, a bit of background and my thoughts: :)

Currently we have 6 machines bodhi is setup on in puppet
(Luke: please correct me if I am wrong with any of this)

app01, app02, app03, app04: These machines are the ones maintainers hit to manage updates, the public/anon web pages people can see updates on, provide karma, etc.
releng04: This is the machine that does 2 things: Fedora updates pushes (so it needs masher config) and periodic jobs (like checking for old buildroot overrides or telling maintainers that a update has spent enough time in testing, etc), so it needs jobrunner.
relepel01: EPEL updates, so this just needs epel masher config.

In moving to ansible, we want to drop the app servers and make 2 (at least) new machines... bodhi01 and bodhi02 that handle what the app servers did. The other two I guess can stay the same, or we could rename them 'fedoraupdates01' 'epelupdates01' or something.

To my mind it would be nice if we had just 3 roles... the app (goes on bodhi01/02), the masher (goes on the two releng machines, possibly just with variables to decide if it has epel or fedora mash configs, or possibly just installs them all to both), the jobrunner (goes on releng04). This allows us flexability if we want to move the jobrunner somewhere, and such.

Does that make sense or help?
Luke: any other thoughts? Should we make any changes for bodhi 2.0 too? if so, what?

So, a bit of background and my thoughts: :) Currently we have 6 machines bodhi is setup on in puppet (Luke: please correct me if I am wrong with any of this) * app01, app02, app03, app04: These machines are the ones maintainers hit to manage updates, the public/anon web pages people can see updates on, provide karma, etc. * releng04: This is the machine that does 2 things: Fedora updates pushes (so it needs masher config) and periodic jobs (like checking for old buildroot overrides or telling maintainers that a update has spent enough time in testing, etc), so it needs jobrunner. * relepel01: EPEL updates, so this just needs epel masher config. In moving to ansible, we want to drop the app servers and make 2 (at least) new machines... bodhi01 and bodhi02 that handle what the app servers did. The other two I guess can stay the same, or we could rename them 'fedoraupdates01' 'epelupdates01' or something. To my mind it would be nice if we had just 3 roles... the app (goes on bodhi01/02), the masher (goes on the two releng machines, possibly just with variables to decide if it has epel or fedora mash configs, or possibly just installs them all to both), the jobrunner (goes on releng04). This allows us flexability if we want to move the jobrunner somewhere, and such. Does that make sense or help? Luke: any other thoughts? Should we make any changes for bodhi 2.0 too? if so, what?

lmacken commented

2014-02-04 03:09:20 +00:00

Author

Replying to [comment:4 kevin]:

So, a bit of background and my thoughts: :)

Currently we have 6 machines bodhi is setup on in puppet
(Luke: please correct me if I am wrong with any of this)

app01, app02, app03, app04: These machines are the ones maintainers hit to
manage updates, the public/anon web pages people can see updates on, provide
karma, etc.

releng04: This is the machine that does 2 things: Fedora updates pushes (so
it needs masher config) and periodic jobs (like checking for old buildroot
overrides or telling maintainers that a update has spent enough time in
testing, etc), so it needs jobrunner.

Yup, sounds right. Even though it does 2 things, it only inherits from the
single bodhi::masher::jobrunner subclass.

relepel01: EPEL updates, so this just needs epel masher config.

In moving to ansible, we want to drop the app servers and make 2 (at least)
new machines... bodhi01 and bodhi02 that handle what the app servers did. The
other two I guess can stay the same, or we could rename them
'fedoraupdates01' 'epelupdates01' or something.

To my mind it would be nice if we had just 3 roles... the app (goes on
bodhi01/02), the masher (goes on the two releng machines, possibly just with
variables to decide if it has epel or fedora mash configs, or possibly just
installs them all to both), the jobrunner (goes on releng04). This allows us
flexability if we want to move the jobrunner somewhere, and such.

Does that make sense or help?
Luke: any other thoughts? Should we make any changes for bodhi 2.0 too? if so, what?

This makes sense to me. In the future, splitting the 'jobrunner' out into simple
cron jobs is probably the way to go, so having the flexibility to run it
elsewhere is fine. Right now it's simply a 1-line config file change to enable
these jobs (even though we have seperate config templates with lots of duplicate options), so it could also potentially be narrowed down to just app &
masher roles, having a 'jobrunner' boolean option to enable them.

Replying to [comment:4 kevin]: > So, a bit of background and my thoughts: :) > > Currently we have 6 machines bodhi is setup on in puppet > (Luke: please correct me if I am wrong with any of this) > > * app01, app02, app03, app04: These machines are the ones maintainers hit to > manage updates, the public/anon web pages people can see updates on, provide > karma, etc. > > * releng04: This is the machine that does 2 things: Fedora updates pushes (so > it needs masher config) and periodic jobs (like checking for old buildroot > overrides or telling maintainers that a update has spent enough time in > testing, etc), so it needs jobrunner. Yup, sounds right. Even though it does 2 things, it only inherits from the single bodhi::masher::jobrunner subclass. > * relepel01: EPEL updates, so this just needs epel masher config. > > In moving to ansible, we want to drop the app servers and make 2 (at least) > new machines... bodhi01 and bodhi02 that handle what the app servers did. The > other two I guess can stay the same, or we could rename them > 'fedoraupdates01' 'epelupdates01' or something. > > To my mind it would be nice if we had just 3 roles... the app (goes on > bodhi01/02), the masher (goes on the two releng machines, possibly just with > variables to decide if it has epel or fedora mash configs, or possibly just > installs them all to both), the jobrunner (goes on releng04). This allows us > flexability if we want to move the jobrunner somewhere, and such. > > Does that make sense or help? > Luke: any other thoughts? Should we make any changes for bodhi 2.0 too? if so, what? This makes sense to me. In the future, splitting the 'jobrunner' out into simple cron jobs is probably the way to go, so having the flexibility to run it elsewhere is fine. Right now it's simply a 1-line config file change to enable these jobs (even though we have seperate config templates with lots of duplicate options), so it could also potentially be narrowed down to just app & masher roles, having a 'jobrunner' boolean option to enable them.

janeznemanic commented

2014-02-06 04:26:32 +00:00

I decided to change everything to just two roles and to control certain tasks in roles through variables.
Now there are:

roles/bodhi/base

roles/bodhi/masher

Extra boolean variables are jobrunner, epelmasher and bodhiapp. These are by default set to false. You can change their values, when you include a role in playbook. I included dependency in masher role which automatically includes base role.

Have a look and comment it.

I decided to change everything to just two roles and to control certain tasks in roles through variables. Now there are: roles/bodhi/base roles/bodhi/masher Extra boolean variables are jobrunner, epelmasher and bodhiapp. These are by default set to false. You can change their values, when you include a role in playbook. I included dependency in masher role which automatically includes base role. Have a look and comment it.

janeznemanic commented

2014-02-06 04:26:56 +00:00

attachment
bodhi_ansible_port

attachment [bodhi_ansible_port](/fedora-infrastructure/issue/raw/files/01f2ec7348a3e56c0898bd1905f368c8ca28bf9e84ebbf33607ece9966c3718a-bodhi_ansible_port)

lmacken commented

2014-02-06 05:44:26 +00:00

Author

Looks good, but I'm confused about the 'bodhiapp' boolean. It looks like all it does is apply the hotfix from ticket [ticket:3755], but that's really only needed for the mashers.

lmacken commented

2014-02-06 05:55:05 +00:00

Author

I just applied that bodhi hotfix upstream, so you can remove it from the playbook

janeznemanic commented

2014-02-06 23:21:22 +00:00

OK I will do that. Is there anything else I can improve?

kevin commented

2014-02-07 01:50:09 +00:00

Owner

A few more things:

We could probibly drop all the old mash files from the old eol releases now i would think?

We need a playbooks/group/bodhi.yml playbook... this should create the instances as the first play, then configure them with the role(s) in the second. Look at any of the other existing group playbooks.

Along with that, we need inventory/group_vars/bodhi and host_vars/releng04, relepel01, bodhi01, bodhi02. I can get you information for ip address, etc on those when needed.

The instances also need nfs mounts. Look at tasks/koji/releng_config.yml for how to do that. Perhaps we should split that out to a nfs_client role since we use it there and here and in builders too.

Take a look at the existing manifests/nodes/releng04* and relepel01* in puppet, there's a number of misc other scripts we need to port over, but I guess those all shouldn't be in the bodhi role. Perhaps releng04/relepel01 should be new releng role and just include the bodhi masher stuff. Thats likely the way to go. So, perhaps we concentrate on getting the bodhi role done and bodhi01/02 and then look at releng04/relepel01 next after those are done.

A few more things: We could probibly drop all the old mash files from the old eol releases now i would think? We need a playbooks/group/bodhi.yml playbook... this should create the instances as the first play, then configure them with the role(s) in the second. Look at any of the other existing group playbooks. Along with that, we need inventory/group_vars/bodhi and host_vars/releng04, relepel01, bodhi01, bodhi02. I can get you information for ip address, etc on those when needed. The instances also need nfs mounts. Look at tasks/koji/releng_config.yml for how to do that. Perhaps we should split that out to a nfs_client role since we use it there and here and in builders too. Take a look at the existing manifests/nodes/releng04* and relepel01* in puppet, there's a number of misc other scripts we need to port over, but I guess those all shouldn't be in the bodhi role. Perhaps releng04/relepel01 should be new releng role and just include the bodhi masher stuff. Thats likely the way to go. So, perhaps we concentrate on getting the bodhi role done and bodhi01/02 and then look at releng04/relepel01 next after those are done.

janeznemanic commented

2014-02-07 23:43:30 +00:00

OK, I will try to come up with something though I will probably need help. I will keep you informed.

janeznemanic commented

2014-02-08 02:53:05 +00:00

Here is playbooks/groups/bodhi.yml. NFS mount is missing at the moment. I will do that tomorrow. Guys could you have a look and tell me which roles and tasks in plays are OK and which are not. And what do I add.

janeznemanic commented

2014-02-08 02:56:09 +00:00

attachment
bodhi.yml

attachment [bodhi.yml](/fedora-infrastructure/issue/raw/files/cdefec3b2a30ac0b28aaa0f0248f493219b228164436eaac57ae101fe0c8bb5f-bodhi.yml)

kevin commented

2014-02-09 03:21:18 +00:00

Owner

Minor nits:

Hosts is seperated with : and not ,
Instead of seperate plays for each of the types, you can instead use 'when:' that just checks for either hostname or a variable and only runs that role when it's true. Then, all those plays can collapse into 2 (one to make the machines, the second to configure them, with conditional when's).

Minor nits: * Hosts is seperated with : and not , * Instead of seperate plays for each of the types, you can instead use 'when:' that just checks for either hostname or a variable and only runs that role when it's true. Then, all those plays can collapse into 2 (one to make the machines, the second to configure them, with conditional when's).

janeznemanic commented

2014-02-10 03:05:40 +00:00

Here are few new things. Comments, suggestions? Variables' names in new nfs role?

janeznemanic commented

2014-02-10 03:07:02 +00:00

attachment
bodhi

attachment [bodhi](/fedora-infrastructure/issue/raw/files/3e1daf32107283c2a00457b198c1f0fad05367d518eeb4195ebeac89a9377be8-bodhi)

janeznemanic commented

2014-02-10 03:07:20 +00:00

attachment
bodhi.2.yml

attachment [bodhi.2.yml](/fedora-infrastructure/issue/raw/files/758c23f6b4fde3673f37f4fa8c9967bb502319e9012fa3a307bf9b0ddd6329bf-bodhi.2.yml)

janeznemanic commented

2014-02-10 03:07:53 +00:00

attachment
nfs_role.yml

attachment [nfs_role.yml](/fedora-infrastructure/issue/raw/files/d31e0733286a1609efc209f68a035ac30683842cfc41e3e8c1e5cc4aa20b3982-nfs_role.yml)

kevin commented

2014-02-11 00:42:54 +00:00

Owner

service: fedocal

should be service: bodhi ?

I think we need mod_wsgi on all of them... even the application ones.

For the nfs mount, we also need a weird route to the netapp...
look in roles/koji_builder/tasks/main.yml starting at "name: route config for netapp network"

Can you do a diff against current ansible HEAD so we can see the full changes? I think we are getting close here. ;)

service: fedocal should be service: bodhi ? I think we need mod_wsgi on all of them... even the application ones. For the nfs mount, we also need a weird route to the netapp... look in roles/koji_builder/tasks/main.yml starting at "name: route config for netapp network" Can you do a diff against current ansible HEAD so we can see the full changes? I think we are getting close here. ;)

janeznemanic commented

2014-02-11 23:09:23 +00:00

Sorry for fedocal, I didn't notice. Mod_wsgi will be added and nfs mount changed.

What is missing at the moment is host variables, domain names in inventory and perhaps some changes in dns repo. That's how I see it. But besides that basic bodhi porting is probably done.

After we are done with basic porting I suggest that we do releng04 and relepel01.

Sorry for fedocal, I didn't notice. Mod_wsgi will be added and nfs mount changed. What is missing at the moment is host variables, domain names in inventory and perhaps some changes in dns repo. That's how I see it. But besides that basic bodhi porting is probably done. After we are done with basic porting I suggest that we do releng04 and relepel01.

janeznemanic commented

2014-02-12 00:41:12 +00:00

Had troubles with diff, so here is patch.

janeznemanic commented

2014-02-12 00:41:41 +00:00

attachment
0001-Porting-bodhi-to-ansible.patch

attachment [0001-Porting-bodhi-to-ansible.patch](/fedora-infrastructure/issue/raw/files/f9939758c64cbbfc45ac689fae98918e9e3721427a34e67b7e3c89dd2b36214b-0001-Porting-bodhi-to-ansible.patch)

janeznemanic commented

2014-02-12 00:49:07 +00:00

I added that task from koji to nfs_client role. I also added two more tasks that seem relevant to that "route config" tasks. If they are not needed I will remove them.

kevin commented

2014-02-15 02:02:03 +00:00

Owner

Quite the patch now at this point. ;)

So, from a lookover it looks ok to me, but we may run into various little issues.

I'd like to commit this next week and then see if we can bring up a bodhi01.stg first with it. If we get that all going, then bring up a bodhi01/02 in production and switch bodhi web traffic from the app servers over to those. Once thats all stable, we can look at moving on to relepel01 and releng01. I think those will be a seperate playbook (or in with the existing releng one) and just include the needed bodhi roles.

Thanks a bunch for working on this. If you have time next week to work on it on irc we can push things forward and I can have you tweak any issues we run into...

Quite the patch now at this point. ;) So, from a lookover it looks ok to me, but we may run into various little issues. I'd like to commit this next week and then see if we can bring up a bodhi01.stg first with it. If we get that all going, then bring up a bodhi01/02 in production and switch bodhi web traffic from the app servers over to those. Once thats all stable, we can look at moving on to relepel01 and releng01. I think those will be a seperate playbook (or in with the existing releng one) and just include the needed bodhi roles. Thanks a bunch for working on this. If you have time next week to work on it on irc we can push things forward and I can have you tweak any issues we run into...

lmacken commented

2014-02-15 03:15:20 +00:00

Author

Replying to [comment:20 kevin]:

Quite the patch now at this point. ;)

So, from a lookover it looks ok to me, but we may run into various little issues.

I'd like to commit this next week and then see if we can bring up a bodhi01.stg first with it. If we get that all going, then bring up a bodhi01/02 in production and switch bodhi web traffic from the app servers over to those. Once thats all stable, we can look at moving on to relepel01 and releng01. I think those will be a seperate playbook (or in with the existing releng one) and just include the needed bodhi roles.

Sounds good

Thanks a bunch for working on this. If you have time next week to work on it on irc we can push things forward and I can have you tweak any issues we run into...

Yeah, +1. Thanks for doing this janeznemanic, I look forward to testing it out next week.

Replying to [comment:20 kevin]: > Quite the patch now at this point. ;) > > So, from a lookover it looks ok to me, but we may run into various little issues. > > I'd like to commit this next week and then see if we can bring up a bodhi01.stg first with it. If we get that all going, then bring up a bodhi01/02 in production and switch bodhi web traffic from the app servers over to those. Once thats all stable, we can look at moving on to relepel01 and releng01. I think those will be a seperate playbook (or in with the existing releng one) and just include the needed bodhi roles. Sounds good > Thanks a bunch for working on this. If you have time next week to work on it on irc we can push things forward and I can have you tweak any issues we run into... Yeah, +1. Thanks for doing this janeznemanic, I look forward to testing it out next week.

janeznemanic commented

2014-02-18 23:51:47 +00:00

Kevin, shall we try to apply things tomorrow? What about host variables? Have you taken care of that?
How are we going to handle time zone difference? I'm in CET. I have time after 17.00 h UTC, is that fine with you?

Kevin, shall we try to apply things tomorrow? What about host variables? Have you taken care of that? How are we going to handle time zone difference? I'm in CET. I have time after 17.00 h UTC, is that fine with you?

kevin commented

2014-02-19 03:13:57 +00:00

Owner

Tomorrow might work, but I have a meeting at 18UTC... so we could work on it from 17-18 and see where we are at at the end of that?

I still need to allocate ips for the new bodhi01/02's... I can do that tomorrow morning. ;)

Tomorrow might work, but I have a meeting at 18UTC... so we could work on it from 17-18 and see where we are at at the end of that? I still need to allocate ips for the new bodhi01/02's... I can do that tomorrow morning. ;)

kevin commented

2014-02-20 07:04:06 +00:00

Owner

ok. I got sidetracked, but finally got to pushing this in. ;)

We now have a bodhi01.stg instance up and running.

Thanks a bunch for all the work on this.

ok. I got sidetracked, but finally got to pushing this in. ;) We now have a bodhi01.stg instance up and running. Thanks a bunch for all the work on this.

Rows
Columns

Port the bodhi puppet module to ansible #4193