mvadkert commented

Please add this IAM role for Testing Farm, we need it to autoscale our EKS clusters.

name: fedora-ci-eks-autoscaler
description: IAM role for k8s cluster autoscaler - https://docs.aws.amazon.com/eks/latest/best-practices/cas.html

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/k8s.io/cluster-autoscaler/enabled": "true"
                },
                "StringLike": {
                    "aws:ResourceTag/k8s.io/cluster-autoscaler/testing-farm-*": "owned"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeScalingActivities",
                "autoscaling:DescribeTags",
                "ec2:DescribeImages",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeLaunchTemplateVersions",
                "ec2:GetInstanceTypesFromInstanceRequirements",
                "eks:DescribeNodegroup"
            ],
            "Resource": "*"
        }
    ]
}

Please add this IAM role for Testing Farm, we need it to autoscale our EKS clusters. name: `fedora-ci-eks-autoscaler` description: IAM role for k8s cluster autoscaler - https://docs.aws.amazon.com/eks/latest/best-practices/cas.html ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "autoscaling:SetDesiredCapacity", "autoscaling:TerminateInstanceInAutoScalingGroup" ], "Resource": "*", "Condition": { "StringEquals": { "aws:ResourceTag/k8s.io/cluster-autoscaler/enabled": "true" }, "StringLike": { "aws:ResourceTag/k8s.io/cluster-autoscaler/testing-farm-*": "owned" } } }, { "Effect": "Allow", "Action": [ "autoscaling:DescribeAutoScalingGroups", "autoscaling:DescribeAutoScalingInstances", "autoscaling:DescribeLaunchConfigurations", "autoscaling:DescribeScalingActivities", "autoscaling:DescribeTags", "ec2:DescribeImages", "ec2:DescribeInstanceTypes", "ec2:DescribeLaunchTemplateVersions", "ec2:GetInstanceTypesFromInstanceRequirements", "eks:DescribeNodegroup" ], "Resource": "*" } ] } ```

mvadkert commented

2025-09-02 15:45:02 +00:00

Author

Documentation: https://docs.aws.amazon.com/eks/latest/best-practices/cas.html

smoliicek commented

2025-09-02 19:06:05 +00:00

Member

Metadata Update from @smoliicek:

Issue priority set to: Waiting on Assignee (was: Needs Review)
Issue tagged with: medium-gain, medium-trouble

**Metadata Update from @smoliicek**: - Issue priority set to: Waiting on Assignee (was: Needs Review) - Issue tagged with: medium-gain, medium-trouble

kevin commented

2025-09-02 22:38:24 +00:00

Owner

Metadata Update from @kevin:

Issue assigned to kevin

**Metadata Update from @kevin**: - Issue assigned to kevin

kevin commented

2025-09-02 22:38:28 +00:00

Owner

Done. Can you please test?

mvadkert commented

2025-09-03 20:30:50 +00:00

Author

@kevin hmm, I cannot see it?

https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-2#/roles/details/fedora-ci-eks-autoscaler?section=permissions

@kevin hmm, I cannot see it? https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-2#/roles/details/fedora-ci-eks-autoscaler?section=permissions

kevin commented

2025-09-04 19:55:27 +00:00

Owner

Oh, I set it up as a policy instead of a role.

https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-2#/policies/details/arn%3Aaws%3Aiam%3A%3A125523088429%3Apolicy%2Ffedora-ci-testing-farm-autoscaler?section=permissions

I'm not sure how to add a role for this. I guess I'll read over the docs more.

Oh, I set it up as a policy instead of a role. https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-2#/policies/details/arn%3Aaws%3Aiam%3A%3A125523088429%3Apolicy%2Ffedora-ci-testing-farm-autoscaler?section=permissions I'm not sure how to add a role for this. I guess I'll read over the docs more.

mvadkert commented

2025-09-04 21:10:32 +00:00

Author

it should be simple:

https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/roles/create
select Custom trust policy
Paste in the json I gave you, next, next, give it the name I provided

Should be done

it should be simple: 1. https://us-east-1.console.aws.amazon.com/iam/home?region=us-east-1#/roles/create 2. select `Custom trust policy` 3. Paste in the json I gave you, next, next, give it the name I provided Should be done

kevin commented

2025-09-04 22:15:30 +00:00

Owner

ah, I see...

well, it doesn't like it. ;)

Ln 4, Col 8
	
Missing Principal: Add a Principal element to the policy statement. Learn more 
Ln 10, Col 24
	
Role Trust Policy Syntax Error Resource: Role trust policies apply to the role that they are attached to. You cannot specify a resource. Remove the Resource or NotResource element. Learn more 
Ln 18, Col 8
	
Missing Principal: Add a Principal element to the policy statement. Learn more 
Ln 32, Col 24
	
Role Trust Policy Syntax Error Resource: Role trust policies apply to the role that they are attached to. You cannot specify a resource. Remove the Resource or NotResource element. Learn more

ah, I see... well, it doesn't like it. ;) ``` Ln 4, Col 8 Missing Principal: Add a Principal element to the policy statement. Learn more Ln 10, Col 24 Role Trust Policy Syntax Error Resource: Role trust policies apply to the role that they are attached to. You cannot specify a resource. Remove the Resource or NotResource element. Learn more Ln 18, Col 8 Missing Principal: Add a Principal element to the policy statement. Learn more Ln 32, Col 24 Role Trust Policy Syntax Error Resource: Role trust policies apply to the role that they are attached to. You cannot specify a resource. Remove the Resource or NotResource element. Learn more ```

kevin commented

2025-09-15 20:46:50 +00:00

Owner

Any ideas? I can't get past the syntaxt errors with the proposed one (even from their docs)... so I am surely doing something wrong. ;(

kevin commented

2025-10-22 21:54:57 +00:00

Owner

Any ideas what to try here?

mvadkert commented

2025-12-03 00:29:02 +00:00

Author

Asked @mwinters if he can try to help here.

mwinters commented

2025-12-04 00:03:32 +00:00

Contributor

Thanks for ringing me in. Happy to help! I've included the above details in this post so you should be able to just follow it top to bottom.

Preface: Use Karpenter

First, I highly recommend using Karpenter to autoscale your clusters instead of the vanilla Cluster Autoscaler. It is slightly more complex and another tool to learn, but it simplifies many other admin tasks including cluster upgrades, cost optimization, and security.

But, I'll assume here that we want to use Cluster Autoscaler anyway for simplicity.

Pod IAM Credentials with PIA

There are several ways to get AWS credentials into a pod. I'm operating entirely blind here -- I have no access to Fedora AWS and no idea what our preferences / policies / norms are. But the modern recommended approach is to use the "Pod Identity Agent", aka PIA, which is both a Kubernetes add-on and a set of AWS IAM features. This allows you to explicitly and easily create a chain of IAM Policy -> IAM Role -> k8s Service Account -> My Pod.

My assumptions

Based on what I've gathered from random convos, I'll assume:

Most setup is happening via the AWS Console, not IAC or CLI.
We do not need cross-cluster or cross-account (AWS account) permissions.
We want to use IAM best practices for isolation.

Step 1: Setup PIA in your cluster

With EKS Auto Mode

If you're using EKS Auto Mode then you are probably already done, unless your cluster is very old. PIA is installed by default into Auto Mode clusters. You can check this with kubectl get pods -n kube-system | grep 'eks-pod-identity-agent'.

Without Auto Mode

Step 1.1: Grant the worker node's IAM role permissions

If the worker nodes are using an IAM role which includes the AWS-managed policy AmazonEKSWorkerNodePolicy then you are already done.
If not, add this policy to the IAM role:

{
    "Version":"2012-10-17",		 	 	 
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "eks-auth:AssumeRoleForPodIdentity"
            ],
            "Resource": "*"
        }
    ]
}

Step 1.2: Enable PIA in the EKS console

1. AWS Console -> EKS -> Clusters -> My Cluster
2. Add-ons -> Get more add-ons -> EKS Pod Identity Agent -> Next
3. If necessary, fill out any additional config that a k8s Service Account in your cluster would need, e.g. ImagePullSecrets.
4. Next. Wait a few minutes and you should see PIA running with kubectl get pods -n kube-system | grep 'eks-pod-identity-agent'.

Step 2: Create an IAM Policy with the permissions needed by your pod

1. AWS Console -> IAM -> Policies -> Create Policy -> JSON mode
2. Paste the following and click Next:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:ResourceTag/k8s.io/cluster-autoscaler/enabled": "true"
                },
                "StringLike": {
                    "aws:ResourceTag/k8s.io/cluster-autoscaler/testing-farm-*": "owned"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeScalingActivities",
                "autoscaling:DescribeTags",
                "ec2:DescribeImages",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeLaunchTemplateVersions",
                "ec2:GetInstanceTypesFromInstanceRequirements",
                "eks:DescribeNodegroup"
            ],
            "Resource": "*"
        }
    ]
}

3. Name the policy something like foocluster_autoscaling -> set Tags (because you do this on everything, right? right??!!!) -> Create Policy.

Step 3: Attach the Policy to a new IAM Role

1. AWS Console -> IAM -> Roles -> Create Role -> Custom Trust Policy
2. Paste the following trust policy and click Next:

{
    "Version":"2012-10-17",		 	 	 
    "Statement": [
        {
            "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}

3. Search for the foocluster_autoscaling policy we just created -> click the checkbox next to it -> Next
4. Name the role something like foocluster_autoscaling (yes, the role name is the same as the policy name) -> set tags -> Create role.

Step 4. Create a Service Account for this app in your Kubernetes cluster

Note: You don't have to do this just yet. But if you do, then it will show up in the GUI in the next step instead of requiring you to type it.

kubectl -n kube-system create serviceaccount cluster-autoscaler

Step 5. Associate the k8s Service Account with the AWS IAM Role

1. AWS Console -> EKS -> Clusters -> My Cluster
2. Access tab -> Pod identity associations -> Create
3. Select the foocluster_autoscaling role, the Kubernetes namespace (kube-system in this case, I think), and the cluster-autoscaler Service Account from that namespace.
- If step 4 above isn't done yet, you can just hardcode these values and they'll apply whenever someone does that step.

Step 6. Make sure the pod isn't using the default Service Account

spec:
  serviceAccountName: cluster-autoscaler
  automountServiceAccountToken: false

#HMU

You know where to find me if you have questions! And what FAS groups to add me to if you don't any more! ;)

Thanks for ringing me in. Happy to help! I've included the above details in this post so you should be able to just follow it top to bottom. # Preface: Use Karpenter First, I _highly_ recommend using [Karpenter](https://docs.aws.amazon.com/eks/latest/best-practices/karpenter.html) to autoscale your clusters instead of the vanilla Cluster Autoscaler. It is slightly more complex and another tool to learn, but it simplifies many other admin tasks including cluster upgrades, cost optimization, and security. But, I'll assume here that we want to use Cluster Autoscaler anyway for simplicity. # Pod IAM Credentials with PIA There are several ways to get AWS credentials into a pod. I'm operating entirely blind here -- I have no access to Fedora AWS and no idea what our preferences / policies / norms are. But the modern recommended approach is to use the "Pod Identity Agent", aka PIA, which is both a Kubernetes add-on and a set of AWS IAM features. This allows you to explicitly and easily create a chain of IAM Policy -> IAM Role -> k8s Service Account -> My Pod. ## My assumptions Based on what I've gathered from random convos, I'll assume: - Most setup is happening via the AWS Console, not IAC or CLI. - We do not need cross-cluster or cross-account (AWS account) permissions. - We want to use IAM best practices for isolation. ## Step 1: Setup PIA in your cluster ### With EKS Auto Mode If you're using EKS Auto Mode then you are probably already done, unless your cluster is very old. PIA is installed by default into Auto Mode clusters. You can check this with `kubectl get pods -n kube-system | grep 'eks-pod-identity-agent'`. ### Without Auto Mode #### Step 1.1: Grant the worker node's IAM role permissions - If the worker nodes are using an IAM role which includes the AWS-managed policy `AmazonEKSWorkerNodePolicy` then you are already done. - If not, add this policy to the IAM role: ```json { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "eks-auth:AssumeRoleForPodIdentity" ], "Resource": "*" } ] } ``` #### Step 1.2: Enable PIA in the EKS console **1.** AWS Console -> EKS -> Clusters -> My Cluster **2.** Add-ons -> Get more add-ons -> EKS Pod Identity Agent -> Next **3.** If necessary, fill out any additional config that a k8s Service Account in your cluster would need, e.g. `ImagePullSecrets`. **4.** Next. Wait a few minutes and you should see PIA running with `kubectl get pods -n kube-system | grep 'eks-pod-identity-agent'`. ## Step 2: Create an IAM Policy with the permissions needed by your pod **1.** AWS Console -> IAM -> Policies -> Create Policy -> JSON mode **2.** Paste the following and click Next: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "autoscaling:SetDesiredCapacity", "autoscaling:TerminateInstanceInAutoScalingGroup" ], "Resource": "*", "Condition": { "StringEquals": { "aws:ResourceTag/k8s.io/cluster-autoscaler/enabled": "true" }, "StringLike": { "aws:ResourceTag/k8s.io/cluster-autoscaler/testing-farm-*": "owned" } } }, { "Effect": "Allow", "Action": [ "autoscaling:DescribeAutoScalingGroups", "autoscaling:DescribeAutoScalingInstances", "autoscaling:DescribeLaunchConfigurations", "autoscaling:DescribeScalingActivities", "autoscaling:DescribeTags", "ec2:DescribeImages", "ec2:DescribeInstanceTypes", "ec2:DescribeLaunchTemplateVersions", "ec2:GetInstanceTypesFromInstanceRequirements", "eks:DescribeNodegroup" ], "Resource": "*" } ] } ``` **3.** Name the policy something like `foocluster_autoscaling` -> set Tags (because you do this on everything, right? right??!!!) -> Create Policy. ## Step 3: Attach the Policy to a new IAM Role **1.** AWS Console -> IAM -> Roles -> Create Role -> Custom Trust Policy **2.** Paste the following trust policy and click Next: ```json { "Version":"2012-10-17", "Statement": [ { "Sid": "AllowEksAuthToAssumeRoleForPodIdentity", "Effect": "Allow", "Principal": { "Service": "pods.eks.amazonaws.com" }, "Action": [ "sts:AssumeRole", "sts:TagSession" ] } ] } ``` **3.** Search for the `foocluster_autoscaling` policy we just created -> click the checkbox next to it -> Next **4.** Name the role something like `foocluster_autoscaling` (yes, the role name is the same as the policy name) -> set tags -> Create role. ## Step 4. Create a Service Account for this app in your Kubernetes cluster **Note:** You don't _have_ to do this just yet. But if you do, then it will show up in the GUI in the next step instead of requiring you to type it. `kubectl -n kube-system create serviceaccount cluster-autoscaler` ## Step 5. Associate the k8s Service Account with the AWS IAM Role **1.** AWS Console -> EKS -> Clusters -> My Cluster **2.** Access tab -> Pod identity associations -> Create **3.** Select the `foocluster_autoscaling` role, the Kubernetes namespace (`kube-system` in this case, I think), and the `cluster-autoscaler` Service Account from that namespace. - If step 4 above isn't done yet, you can just hardcode these values and they'll apply whenever someone does that step. ## Step 6. Make sure the pod isn't using the default Service Account ```yaml spec: serviceAccountName: cluster-autoscaler automountServiceAccountToken: false ``` # #HMU You know where to find me if you have questions! And what FAS groups to add me to if you don't any more! ;)

kevin commented

2025-12-05 00:04:16 +00:00

Owner

I can do the IAM parts of this... but I don't have access to the eks cluster at all, so I can't see those parts of it.

@mvadkert Thoughts? Do you want to look at Karpenter? it looks interesting... but of course up to you.

I can do the IAM parts of this... but I don't have access to the eks cluster at all, so I can't see those parts of it. @mvadkert Thoughts? Do you want to look at Karpenter? it looks interesting... but of course up to you.

mwinters commented

2025-12-05 00:22:54 +00:00

Contributor

I can do the IAM parts of this... but I don't have access to the eks cluster at all, so I can't see those parts of it.

@mvadkert Thoughts? Do you want to look at Karpenter? it looks interesting... but of course up to you.

My $0.02: if you have time to apply the change, this shouldn't break anything since Cluster Autoscaler already can't work. So I'd say: go for it. You'll just have to type the namespace and serviceaccount in Step 5 instead of selecting them in the GUI.

> I can do the IAM parts of this... but I don't have access to the eks cluster at all, so I can't see those parts of it. > > @mvadkert Thoughts? Do you want to look at Karpenter? it looks interesting... but of course up to you. > My $0.02: if you have time to apply the change, this shouldn't break anything since Cluster Autoscaler already can't work. So I'd say: go for it. You'll just have to type the namespace and serviceaccount in Step 5 instead of selecting them in the GUI.

kevin commented

2026-04-02 16:44:03 +00:00

Owner

@mvadkert is this still needed? can you answer my queries from #12761 (comment) ?

@mvadkert is this still needed? can you answer my queries from https://forge.fedoraproject.org/infra/tickets/issues/12761#issuecomment-266673 ?

mvadkert commented

2026-04-07 20:15:28 +00:00

Author

@kevin we definitely are deploying karpenter this year, will recheck again the requirements, sorry for the delay.