Upgarde issue from 35 to 1.0

[sree42111]

Description

We have updated Karpenter from version 0.35 to 0.36, and then to 0.37, v1.0. We have applied the patches according to the documentation. However, we are currently encountering the following error, despite having the correct EC2 create tag permission.

UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::5482:assumed-role/KarpenterNodeRole-DR-R4E-N/17272406301 is not authorized to perform: ec2:CreateTags on resource: arn:aws:ec2:us-east-1:5480:spot-instances-request/* because no identity-based policy allows the ec2:CreateTags action

aws sts decode-authorization-message
{
"DecodedMessage": "{"allowed":false,"explicitDeny":false,"matchedStatements":{"items":[]},"failures":{"items":[]},"context":{"principal":{"id":"AROAX7JP:1727241","arn":"arn:aws:sts::548:assumed-role/KarpenterNodeRole-DR-P-N/1727223001"},"action":"RunInstances","resource":"arn:aws:ec2:us-east-1:54820:spot-instances-request/","conditions":{"items":[{"key":"aws:Region","values":{"items":[{"value":"us-east-1"}]}},{"key":"aws:ID","values":{"items":[{"value":""}]}},{"key":"aws:Service","values":{"items":[{"value":"ec2"}]}},{"key":"aws:Resource","values":{"items":[{"value":"spot-instances-request/"}]}},{"key":"aws:Type","values":{"items":[{"value":"spot-instances-request"}]}},{"key":"aws:Account","values":{"items":[{"value":"54"}]}},{"key":"ec2:Region","values":{"items":[{"value":"us-east-1"}]}},{"key":"aws:ARN","values":{"items":[{"value":"arn:aws:ec2:us-east-1:548:spot-instances-request/"}]}}]}}}"
}
Karpneter policy :

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowScopedEC2InstanceActions",
"Effect": "Allow",
"Resource": [
"arn:aws:ec2:us-east-1::image/",
"arn:aws:ec2:us-east-1::snapshot/",
"arn:aws:ec2:us-east-1::spot-instances-request/",
"arn:aws:ec2:us-east-1::security-group/",
"arn:aws:ec2:us-east-1::subnet/",
"arn:aws:ec2:us-east-1::launch-template/"
],
"Action": [
"ec2:RunInstances",
"ec2:CreateFleet"
]
},
{
"Sid": "AllowScopedEC2LaunchTemplateActions",
"Effect": "Allow",
"Resource": "arn:aws:ec2:us-east-1::launch-template/",
"Action": "ec2:CreateLaunchTemplate",
"Condition": {
"StringEquals": {
"aws:RequestTag/kubernetes.io/cluster/DR-P-N": "owned"
},
"StringLike": {
"aws:RequestTag/karpenter.sh/provisioner-name": ""
}
}
},
{
"Sid": "AllowScopedEC2InstanceActionsWithTags",
"Effect": "Allow",
"Resource": [
"arn:aws:ec2:us-east-1::fleet/",
"arn:aws:ec2:us-east-1::instance/",
"arn:aws:ec2:us-east-1::volume/",
"arn:aws:ec2:us-east-1::network-interface/"
],
"Action": [
"ec2:RunInstances",
"ec2:CreateFleet"
],
"Condition": {
"StringEquals": {
"aws:RequestTag/kubernetes.io/cluster/DR-P-N": "owned"
},
"StringLike": {
"aws:RequestTag/karpenter.sh/provisioner-name": ""
}
}
},
{
"Sid": "AllowScopedResourceCreationTagging",
"Effect": "Allow",
"Resource": [
"arn:aws:ec2:us-east-1::fleet/",
"arn:aws:ec2:us-east-1::instance/",
"arn:aws:ec2:us-east-1::volume/",
"arn:aws:ec2:us-east-1::network-interface/",
"arn:aws:ec2:us-east-1::launch-template/"
],
"Action": "ec2:CreateTags",
"Condition": {
"StringEquals": {
"aws:RequestTag/kubernetes.io/cluster/DR-P-R4E-N": "owned",
"ec2:CreateAction": [
"RunInstances",
"CreateFleet",
"CreateLaunchTemplate"
]
},
"StringLike": {
"aws:RequestTag/karpenter.sh/provisioner-name": "*"

[k8s-ci-robot]

This issue is currently awaiting triage.

If Karpenter contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[jigisha620]

I think the permissions that it is looking for is on the KarpenterNodeRole which is covered by this section of our documentation. The permission that is missing is part of the AmazonEKS_CNI_Policy which is a managed policy. The policy that you have shared is part of https://karpenter.sh/v1.0/reference/cloudformation/#controller-authorization and not KarpenterNodeRole.

[sree42111]

@jigisha620 karpenterNode Role already have AmazonEKS_CNI_Policy attached to it

[jigisha620]

Is this an ec2 instance arn - arn:aws:ec2:us-east-1:5480:spot-instances-request/*? It doesn't look right to me.

[jigisha620]

/assign @jigisha620