Using IAM Roles for Kubernetes service accounts in AWS EKS using Terraform Nov 19 2022

Part of the design principles of the security pillar in the AWS Well architected framework is “Implement a strong identity foundation”, that is:

“Implement the principle of least privilege and enforce separation of duties with appropriate authorization for each interaction with your AWS resources. Centralize identity management, and aim to eliminate reliance on long-term static credentials.”

Well Architected Framework - Security Pillar - Design principles

When we start using Kubernetes in AWS EKS, we might take some shortcuts during the learning phase and add all the policies to a role that we directly assign to our nodes. The issue is that we give much more privileges than we require just for practicality. And it can also become a habit. The way we learn something sometimes becomes our default pattern. So let’s break that habit and explore how to set IAM Roles to Kubernetes service accounts in AWS EKS.

Note: If you are interested in learning more about how to set up the directory structure for your Terraform project, you might find my guide, Meditations on Directory Structure for Terraform Projects, useful.

Bash Beyond Basics Increase your efficiency and understanding of the shell

If you are interested in this topic you might enjoy my course Bash Byond Basics. This course helps you level up your bash skills. This is not a course on shell-scripting, is a course on improving your efficiency by showing you the features of bash that are seldom discussed and often ignored.

Every day you spend many hours working in the shell, every little improvement in your worklflows will pay dividends many fold!

Learn more

What are IAM Roles for Kubernetes service accounts?

The IAM Roles for Kubernetes service accounts allow us to associate an IAM role with a Kubernetes service account. This feature is available through the Amazon EKS Pod Identity Webhook. The IAM role for a service account is then used to provide AWS credentials to the pod or resource using the service account. These credentials are automatically rotated before they expire.

If you have used WebIdentity federation with AWS Cognito, you will be familiar with the concept of IAM roles for service accounts.

Ok, so what do we need?

OIDC Issuer - Already set up if you are running in EKS
OIDC provider
The IAM role that we want to associate with the service account
The Kubernetes service account that we want to associate with the IAM role

Set up OIDC provider

The first step is to set up the OIDC provider. OIDC, if you are unfamiliar, is an authentication protocol similar to SAML, but it is based on JSON Web Tokens (JWT). AWS also uses it to authenticate users in AWS Cognito.

There are two parts to using OIDC an Issuer and a Provider. The issuer is the entity that issues the token, and the provider is the entity that validates the token. In our case, the issuer is handled by AWS, and Kubernetes handle the provider. When we create the EKS cluster, we already have an Issuer endpoint. If you want to use your OIDC issuer, you can provide that to EKS. But in this example, we’ll stick to the issuer already in the EKS cluster.

To get the URL of the issuer, run the following command:

1
aws eks describe-cluster --name temp-cluster --query "cluster.identity.oidc.issuer" --output text

You should see something like the following:

https://oidc.eks.us-east-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Let’s check first if an OIDC provider is already created for our cluster. Run the following command:

1
aws iam list-open-id-connect-providers

If you get an empty list, then you need to create the OIDC provider. Let’s define it in terraform:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

locals {
  oidc_issuer = "https://oidc.eks.us-east-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  # We'll use the cluster_namespace and the service_account_name later
  cluster_namespace = "my-namespace"
  service_account_name = "my-service-account"
}

# Set open IDC provider
data "tls_certificate" "cluster" {
  # If we created the EKS cluster in terraform, we would do something like the following
  # url = aws_eks_cluster.main.identity[0].oidc[0].issuer
  url = local.oidc_issuer
}

resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.cluster.certificates.0.sha1_fingerprint]
  # If we created the EKS cluster in terraform, we would do something like the following
  # url             = aws_eks_cluster.main.identity[0].oidc[0].issuer
  url = local.oidc_issuer
}

Now, if we run the command to list the OIDC providers, we should see the one we just created:

1
aws iam list-open-id-connect-providers

With the OIDC provider created, we can now create the IAM role and the Kubernetes service account.

Creating the IAM role

We will create a new IAM role, and we will call it eks-test-role. We want the pod to read a parameter from the SSM parameter store, so we will also create a policy for that and attach it to the same role.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
data "aws_region" "current" {}
data "aws_caller_identity" "current" {}

# SSM Parameter Store Service Role

resource "aws_iam_role" "pod_sa" {
  name = "eks-test-role"

  assume_role_policy = data.aws_iam_policy_document.pod_sa_assume_role_policy.json
}

data "aws_iam_policy_document" "pod_sa_assume_role_policy" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(local.oidc_issuer, "https://", "")}"]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(local.oidc_issuer, "https://", "")}:aud"

      values = ["sts.amazonaws.com"]
    }

    condition {
      test     = "StringEquals"
      variable = "${replace(local.oidc_issuer, "https://", "")}:sub"

      values = ["system:serviceaccount:${local.cluster_namespace}:${local.service_account_name}"]
    }
  }
}

We will create an additional policy allowing the pod to read any parameter in the parameter store. We are going to call it cluster-ssm-parameter-policy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
resource "aws_iam_role_policy_attachment" "sa_role_ssm_parameter_policy" {
  policy_arn = aws_iam_policy.ssm_parameter_policy.arn
  role       = aws_iam_role.pod_sa.name
}

resource "aws_iam_policy" "ssm_parameter_policy" {
  name = "cluster-ssm-parameter-policy"

  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Action" : [
          "ssm:DescribeParameters",
          "ssm:GetParameter",
          "ssm:GetParameters",
          "ssm:GetParametersByPath"
        ],
        "Effect" : "Allow",
        "Resource" : [
          "arn:aws:ssm:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:parameter/*"
        ]
      },
    ]
  })
}

We will create a go program that will serve as a basic HTTP server. The server’s sole function is to echo the parameter value. For Kubernetes to be able to pull the image from ECR, we need to create an ECR repository. Let’s do that. We are going to call it test-ecr:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# ECR
resource "aws_ecr_repository" "main" {
  name = "test-ecr"
}

resource "aws_ssm_parameter" "ecr" {
  name  = "/ecr/ecr-url"
  type  = "String"
  value = aws_ecr_repository.main.repository_url
}

resource "aws_ecr_lifecycle_policy" "main" {
  repository = aws_ecr_repository.main.name

  policy = <<EOF
{
    "rules": [
        {
            "rulePriority": 1,
            "description": "Keep last 10 images",
            "selection": {
                "tagStatus": "tagged",
                "tagPrefixList": ["v"],
                "countType": "imageCountMoreThan",
                "countNumber": 10
            },
            "action": {
                "type": "expire"
            }
        }
    ]
}
EOF
}

The following is a quick and dirty go program that will read the parameter from the parameter store and echo it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
package main

import (
  "fmt"
  "log"
  "net/http"

  "github.com/aws/aws-sdk-go/aws"
  "github.com/aws/aws-sdk-go/aws/session"
  "github.com/aws/aws-sdk-go/service/ssm"
)

func main() {
  http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    // Create an SSM client
    sess, err := session.NewSession(&aws.Config{
      Region: aws.String("us-east-1"),
    })
    if err != nil {
      panic(err)
    }
    svc := ssm.New(sess)

    // Get the parameter
    param, err := svc.GetParameter(&ssm.GetParameterInput{
      Name: aws.String("/my/parameter"),
    })
    if err != nil {
      fmt.Fprintf(w, "Error:")
      fmt.Fprintf(w, err.Error())
    } else {
      // Print the value
      fmt.Fprintf(w, "Greeting: ")
      fmt.Fprintf(w, *param.Parameter.Value)
    }
  })

  fmt.Println("Starting server in port 8080")

  if err := http.ListenAndServe(":8080", nil); err != nil {
    log.Fatal(err)
  }
}

We will also need a Dockerfile to build the image:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
FROM golang:1.19-alpine

WORKDIR /usr/src/app

COPY go.mod go.sum ./

RUN go mod download && go mod verify

COPY main.go .
RUN go build -v -o /usr/local/bin/app ./...

EXPOSE 8080

CMD [ "app" ]

We can now build the image and push it to ECR. Let’s create a file called deploy.sh and add the following content:

1
2
3
4
5
#!/bin/bash
export AWS_ECR_URL=$(aws ecr describe-repositories | jq -r '.repositories[0].repositoryUri')
docker build -f Dockerfile . -t ${AWS_ECR_URL}:latest
aws ecr get-login-password | docker login --username AWS --password-stdin $(echo $AWS_ECR_URL | sed -r "s/\/.*//g")
docker push ${AWS_ECR_URL}:latest

We can now run the script:

1
2
chmod +x deploy.sh
./deploy.sh

And we should have our Image ready to deploy in Kubernetes.

Create the SSM parameter

We are going to create an SSM parameter that the pod will read. We are going to call it /my/parameter:

1
2
3
4
5
resource "aws_ssm_parameter" "my_parameter" {
  name  = "/my/parameter"
  type  = "String"
  value = "Hola Mundo"
}

Create the Kubernetes configuration

We will create the Kubernetes namespace.yml, deployment.yml, service.yml, and service-account.yml.

The namespace.yml file will create the namespace that we are going to use:

1
2
3
4
apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace

The service-account.yml file will create the service account that we are going to use:

1
2
3
4
5
6
7
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account
  namespace: my-namespace
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ROLE_NAME>"

Replace <AWS_ACCOUNT_ID> and <ROLE_NAME> with the values you created in the previous steps.

The service.yml file will create the service that we are going to use:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
apiVersion: v1
kind: Service
metadata:
  name: sserver
  namespace: my-namespace
spec:
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: NodePort
  selector:
    app: sserver

The deployment.yml file will create the deployment that we are going to use:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sserver
  namespace: my-namespace
spec:
  selector:
    matchLabels:
      app: sserver
  replicas: 1
  template:
    metadata:
      labels:
        app: sserver
    spec:
      serviceAccountName: my-service-account
      containers:
        - name: sserver
          image: <ERC_URL>:latest
          ports:
            - containerPort: 8080

Replace <ERC_URL> with the value of the ECR we created in the previous steps.

Ok, now let’s deploy everything:

1
2
3
4
kubectl apply -f namespace.yml
kubectl apply -f service-account.yml  
kubectl apply -f service.yml
kubectl apply -f deployment.yml

And we should be ready to test.

Test that everything is working

We can now test if our pod uses the service account we assigned.

We can port forward to our local host and hit our endpoint to see if we get the parameter’s value. Assuming we only have one pod running in my namespace, we can do the following:

1
kubectl port-forward $(kubectl get pods -n my-namespace --no-headers | awk '{print $1}') 8080:8080 -n my-namespace

In a different terminal, we can do a curl to the endpoint:

1
curl localhost:8080

And we should see:

Greeting: Hola Mundo

Final thoughts

Setting up the OIDC might be the most confusing part if we haven’t worked with it before. But remember you need an Issuer, which is already provided if you use AWS EKS. You can link to an external issuer in the same way. We also need a provider accessible to our cluster, as we did.

After the OIDC part is completed, we only need to create our IAM roles as we usually do and build the association by creating a service account and assigning the role to it.

I hope this helps you to understand how to use IAM roles for service accounts in Kubernetes.

References

AWS Documentation - Kubernetes service accounts AWS Documentation - Well architected framework Illustrated guide to OAuth and OpenID Connect

** If you want to check what else I'm currently doing, be sure to follow me on twitter @rderik or subscribe to the newsletter. If you want to send me a direct message, you can send it to derik@rderik.com.