POC: Forgejo actions with testing-farm/tmt #757
No reviewers
Labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
atomic-desktops/config!757
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "ci/_atomic-desktops-config:testing-farm"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
As per forge/forge#364 (comment), here is a proof of concept for the zuul migration to tmt. There are of course missing components and failing tests as things like test duration, hardware setup, run environment, etc. need to be figured out. The example CI state can be seen in https://forge.fedoraproject.org/ci/_atomic-desktops-config/actions
7cedb1e637to309e2d8fe0Thanks! Can we run things as root in testing-farm jobs?
@ -0,0 +17,4 @@steps:- uses: https://forge.fedoraproject.org/ci/testing-farm@mainwith:api_key: ${{ secrets.TESTING_FARM_API_TOKEN }}Do I need to setup something here?
Yes indeed so, besides the configuring
runs-onfor the specifics of the repo, I think this is the only other thing needed to get started and experimenting.The token you can get from https://testing-farm.io/tokens/
That document is good, but just get a token should be all the context needed. Interfacing with testing-farm would be in either https://forge.fedoraproject.org/ci/testing-farm or the
plans.fmfI've added the token to this repo.
@ -0,0 +13,4 @@jobs:testing-farm:runs-on: dockerHum, OK, we need a privileged container here. Can we do that?
That is fine. This is just the runner that is calling testing-farm cli. It will need to be updated to the runner that you will get enabled on
atomic-desktopsgroup. This runner can be super minimal because of that since its only job here is to do a curl post and wait for response, all the rest is done in testing-farm and configured in theci/plans.fmfAh yes, makes sense.
Is the doc at https://docs.testing-farm.io/Testing%20Farm/0.1/onboarding.html what I should read/follow?
Yes, on testing-farm you are in either a container/vm/bare-metal machine as root user by default. I am pretty sure it is a root user, and not a privileged user.
@ -0,0 +5,4 @@on:push:pull_request_target:The triggers here are important to consider, let me break down the state that they describe:
pull_request_target: runs on a pull request with the contents of.forgejo/workflows/tests.yamlthat is inmain. This is the only pull request method supported because of how forgejo handles secrets differently from Github (secrets are not available inpull_requestmode on forgejo)push: run it on any commit pushes, and it duplicates the runs when run in a PR.branchesshould be set here to limit the execution, but I am not sure how it should be setup here. I left it without anybranchesalso to debug on my side, but it should definitely be changedOne thing I want to highlight as a consequence of this is that if you even need to update this, you will not see the live results of the changes in a PR. Best workflow to mitigate that is to also run on a well-defined branch name and use that as run through
on.pushflow.For now I think we'll only need to run CI on PRs.
So something is off as we get permissions denied even though we are root:
So the experience is not great (the URL are not clickable in the action logs) so I'm doing basic checks using a runner in #759.
Let's focus on the more advanced compose checks here. Can you give me edit access to this PR so that I can iterate? Thanks
Indeed I did not invest much time into that forgejo action mostly because I do not hope for it to be a long-term solution, and instead packit should take over that kind of interaction. I just want to offer it as a quick stop-gap solution to get other's CI back on their feet as fast as possible.
Oh I thought I already made it available. Try again, it should be available now.
A few tips on how to configure, you can add a
provisionstep and specify more explicitly what configuration you need, e.g. to use a VM, add something like this to theplans.fmf:(available options I think are only
virtual,containerandartemis. See alsohardwarefor some configuration likedisksize)Currently the default is to use a container (tmt-log gives you some basic data under it)
but maybe that's the part that's getting in the way because it does not have systemd?
I did a quick check on what I can do to improve the experience, and it does not look good. It would require significant rewriting, making it a JS Github/Forgejo action or a compatibility layer at the cli. I would like to put off this work until there is clear long-term users of this tool over the alternatives that are in the works.
309e2d8fe0to12bcfadd93Reading https://forgejo.org/docs/next/user/actions/reference/#onpull_request_target,
pull_request_targetis dangerous as it gives write access to the repo. It also uses the action from the main branch for good reasons, so we can not easily checks things in PRs here.Ah, OK we need the push trigger
@siosm wrote in #757 (comment):
Sorry, I don't quite see that part. The write permissions should not be there, but you can also just put
permissions: {}to nullify them all. The only thing needed here are the secrets which is why it ispull_request_targetnotpull_request, and the metadata.Yes, that's the main difference between those 2, and also why tinkering with workflow file is difficult. The workflow file itself is quite minimal, so you can also just get that in early and try it out. Even with push trigger it would be tricky here because it will appear on my fork instead. Feel free to close the PR and reorganize it as needed to be able to test it.
Hum, I added the config to run under a VM but this is still running in a container:
I don't see a reference to
permissionsin https://forgejo.org/docs/next/user/actions/reference/. I know this exists on GitHub but is it implemented in Forgejo? But anyway, I think I understand the flow now and essentially this workflow should only run the code from the testing farm action so this should be reasonable.I added the push trigger for now so that we can use this one to try things out.
That is odd, I will check with testing-farm, why that is the case. In the meantime can try
artemis(basically AWS/VM)6cf61b28c1ca45830a16I think I know what is going wrong here. This needs to use
composekey. Trying to figure out how to expose that on the action@siosm Huzzah we are getting somewhere https://artifacts.dev.testing-farm.io/56f59982-c346-4245-9a7f-f3e5d2b906ed/
This looks good!
1e17127c0f47c72eb650😅 We'll need more than that 🙂
This worked! https://forge.fedoraproject.org/ci/_atomic-desktops-config/actions/runs/10/jobs/0/attempt/1
Looks like it took ~18 min: https://artifacts.dev.testing-farm.io/65d63f3b-a0a4-43ce-bd64-dc1a21d93b1b/
Can we run multiple jobs in parallel? I think we would need multiple workflows? (so we would really need: forge/forge#568)
Looking at https://forge.fedoraproject.org/ci/testing-farm, I see:
plan .* Regex pattern of plans to be run@siosm wrote in #757 (comment):
Multiple plans, yes, that will be consolidated into one testing-farm job and result, but the individual plan result would be seen on the testing-farm results.
Multiple testing-farm queries, that is blocked part by forge/forge#568 I believe. I tried that on https://forge.fedoraproject.org/ci/_fedora-kiwi-descriptions/actions/runs/1/jobs/0/attempt/1 and they were stuck waiting on a forgejo runner to be available.
But if you can live without the separation of results in forgejo for a while, and waiting on the forgejo runner to be available in general, then you should not really be blocked right now
Oops I made the testing-farm token too short-lived. I've re-triggered it https://artifacts.dev.testing-farm.io/a1a3805e-3509-4f61-b354-19c831d9f9dc/
Thanks, it looks like this is working.
Saw your intent :). You need different plans (each with a single test) in order to have these run in parallel, hopefully the example here clarifies how things need to be organized to run in a specific way
Ah, thanks!
docker-slimtype of forgejo runner #568@ -0,0 +24,4 @@with:api_key: ${{ secrets.TESTING_FARM_API_TOKEN }}# compose: "Fedora-Rawhide"compose: "Fedora-44"List of composes is here. Can take
Fedora-latestThanks for helping me figure this out. I think we'll move forward with that. I've cleaned things up and made: #767
@siosm wrote in #757 (comment):
Awesome. I guess I can clean up the fork also?
Yes, we should be able to cleanup the fork now.
Pull request closed