Setting up access to a private repository in ArgoCD with SSM Parameter Store and External Secrets Operator Nov 26 2022

ArgoCD's documentation is quite good. I just feel there is one key question that is often left unanswered. How do I get my private SSH key into ArgoCD in a declarative way that doesn't require hard coding the key into a secret YAML file?

In this post, we are going to use the External Secrets Operator (ESO) to get the private SSH key from AWS SSM Parameter Store and inject it into ArgoCD using a Kubernetes Secret.

If you already have ArgoCD setup, skip directly to the section Setting up ArgoCD for private repositories. If not, follow along.

Ok, let's get started.

Bash Beyond Basics Increase your efficiency and understanding of the shell

If you are interested in this topic you might enjoy my course Bash Byond Basics. This course helps you level up your bash skills. This is not a course on shell-scripting, is a course on improving your efficiency by showing you the features of bash that are seldom discussed and often ignored.

Every day you spend many hours working in the shell, every little improvement in your worklflows will pay dividends many fold!

Learn more

Installing ArgoCD

First, we need to install ArgoCD in our Kubernetes cluster. We will use Helm to do this, but we will do it in a declarative way so we can use our code as the source of truth. Let's start by creating the directory and the required files for our Helm chart.

1
2

mkdir argo-cd
touch argo-cd/{Chart,requirements}.yaml

The content for the Chart.yaml file is:

1
2

name: argo-cd
version: 0.1.0

The content for requirements.yaml is:

1
2
3
4

dependencies:
  - name: argo-cd
    version: 5.13.6
    repository: https://argoproj.github.io/argo-helm

Now we can install the chart:

1
2
3

cd argo-cd/
helm dependency update
helm install argo-cd . -n argocd  --create-namespace

That should get ArgoCD set up in our Kubernetes Cluster. We need a repository with some Kubernetes templates. If you don't have one, you can use the one I created for this post. It is a public repository. You can find it here. Fork it, and work with it. Later you can make it private or create a different private repository.

Create the ArgoCD Project and application

We will test that everything works correctly with ArgoCD using a public repository. Once ArgoCD runs correctly, we can focus on using a private repo. You can skip this section if you don't want to run these tests. Let's now create a new ArgoCD project and application.

The project:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: server-proj
  namespace: argocd
  # Finaliser that ensures that the project is not deleted until any application does not reference it
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  # Project description
  description: Server Project
  destinations:
    # Update your namespace
    - namespace: my-namespace
      server: https://kubernetes.default.svc
  sourceRepos:
    # Make sure to add your repository here
    - https://github.com/rderik/slartybartfast.git
  clusterResourceWhitelist:
  - group: ''
    kind: '*'

The application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: server
  namespace: argocd
spec:
  destination:
    namespace: my-namespace
    server: https://kubernetes.default.svc
  project: server-proj
  source:
    # Change the repository to your repository
    repoURL: https://github.com/rderik/slartybartfast.git
    # point the path to the path where your Kubernetes templates are
    path: server
  syncPolicy:
    automated: # automated sync by default retries failed attempts 5 times with following delays between attempts ( 5s, 10s, 20s, 40s, 80s ); retry controlled using `retry` field.
      prune: false # Specifies if resources should be pruned during auto-syncing ( false by default ).
      selfHeal: false # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).
      allowEmpty: true # Allows deleting all application resources during automatic syncing ( false by default ).

Ok, we can create the project and the application:

1
2

kubectl apply -f project.yaml
kubectl apply -f application.yaml

Finally, we can view our application in the ArgoCD dashboard by doing a port forwarding to the argocd-server service:

1

kubectl port-forward svc/argocd-server -n argocd 8080:443

I run most of my tests in a VM, so when I do port forwarding, I need the port to be bound to all interfaces:

1

kubectl port-forward svc/argo-cd-argocd-server -n argocd 8080:443 --address=0.0.0.0

We can now visit the ArgoCD dashboard at https://localhost:8080 and log in with the default username and password. The default username is admin. To obtain the default password, you need to read a secret created by default during installation.

1

kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d

Log in, and you should see your application being created!

If we do a port forward to the newly created pod, we can see the Nginx welcome page:

1

kubectl port-forward $(kubectl get pods -n my-namespace --no-headers | awk '{print $1}') 8081:80 -n my-namespace

In another terminal:

1

curl localhost:8081

Ok, now we have a working ArgoCD installation. We can focus on setting up the private repository access.

Setting up the private repository access

We need to create an SSH key pair. We will use the key pair to access the private repository.

1

$ ssh-keygen -t rsa -b 4096 -C "key for private repository" -f id_rsa_private_repo

Now we need to add the public key to the repository. In my case, I'm using GitHub, so I need to add the public key to the repository. If you also use GitHub, go to the repository Settings > Deploy Keys, and add the PUBLIC key. We only need read-only, so there is no need to add write permissions.

Let's create a secure string parameter in AWS SSM parameter store that contains the PEM-encoded private key. We will use this as a secret to add the private key to ArgoCD.

1
2
3
4

aws ssm put-parameter \
  --name "/argo-cd/github-repo/ssh-key" \
  --type SecureString \
  --value "$(cat id_rsa_private_repo)"

Finally, now we can look at setting up ArgoCD for our private repository.

Setting up ArgoCD for private repositories

We need to create a secret that will contain the private key. We will use the private key to access the private repository.

ArgoCD's documentation explains that we need to create a secret with the private key so ArgoCD can access the repository. The secret looks like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

apiVersion: v1
kind: Secret
metadata:
  name: private-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: git@github.com:argoproj/my-private-repository
  sshPrivateKey: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
    -----END OPENSSH PRIVATE KEY-----

This approach has the issue that we would have to write our private key in plain text in the secret template. We don't want to do that. So we have a few options to avoid that. We can create a template and documentation that explains how to fill up this file and then manually apply it. That works, but it is a little bit cumbersome. We'll use External Secrets Operator and avoid doing that.

Setting up External Secrets Operator

As we did before for the other dependencies, we will set up the External Secrets Operator in a declarative way using Helm.

1
2

mkdir external-secrets-operator/
touch external-secrets-operator/{Chart,requirements}.yaml

The content for the Chart.yaml file is:

1
2

name: external-secrets-operator
version: 0.1.0

The content for requirements.yaml is:

1
2
3
4

dependencies:
  - name: external-secrets
    version: 0.6.1
    repository: https://charts.external-secrets.io

Now we can install the chart:

1
2
3

cd external-secrets-operator
helm dependency update
helm install external-secrets-operator . -n external-secrets-operator  --create-namespace

Perfect, we have everything set up. Now we can create the secret that will contain the private key.

Creating the secret

Make sure that your IAM role has access to the SSM parameter store. The role will need a policy similar to the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

resource "aws_iam_policy" "ssm_parameter_policy" {
  name = "cluster-ssm-parameter-policy"

  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Action" : [
          "ssm:DescribeParameters",
          "ssm:GetParameter",
          "ssm:GetParameters",
          "ssm:GetParametersByPath"
        ],
        "Effect" : "Allow",
        "Resource" : [
          "arn:aws:ssm:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:/argo-cd/*"
        ]
      },
    ]
  })
}

With that out of the way, let's define the Secret Store:

1
2
3
4
5
6
7
8
9
10
11
12
13

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: my-aws-secret-store
  # we need the secret to exist in the same namespace as ArgoCD
  namespace: argocd
spec:
  provider:
    aws:
      service: ParameterStore
      # define a specific role to limit access
      # to certain secrets
      region: us-east-1

We are going to get the AWS credentials from the current environment. If you wish to use a secret and key stored in a Kubernetes secret, you could add the section:

1
2
3
4
5
6
7
8
9
10

      # Auth defines the information necessary to authenticate against AWS by
      # getting the accessKeyID and secretAccessKey from an already created Kubernetes Secret
      auth:
        secretRef:
          accessKeyID:
            name: awssm-secret
            key: access-key
          secretAccessKey:
            name: awssm-secret
            key: secret-access-key

Reference: https://external-secrets.io/v0.6.1/api/secretstore/

Ok, now we can create the external secret. If we take a look at the documentation, the secret should look like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

apiVersion: v1
kind: Secret
metadata:
  name: private-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: git@github.com:argoproj/my-private-repository
  sshPrivateKey: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
    -----END OPENSSH PRIVATE KEY-----

To generate a secret with that structure, we will use the template feature of the External Secrets Operator to create the secret. The external secret will look like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: private-repo-ssh-key
  namespace: argocd
spec:
  # SecretStoreRef defines which SecretStore to use when fetching the secret data
  secretStoreRef:
    name: my-aws-secret-store
    kind: SecretStore  # or ClusterSecretStore
    # Specify a blueprint for the resulting Kind=Secret
  target:
    name: my-aws-secret-repo-ssh-key
    template:
      metadata:
        labels:
          argocd.argoproj.io/secret-type: repository
      # Use inline templates to construct your desired config file that contains your secret
      data:
        type: git
        url: git@github.com:rderik/slartypriv.git
        sshPrivateKey: |
          {{ .sshPrivateKey | toString }}
  data:
  - secretKey: sshPrivateKey
    remoteRef:
      key: /argo-cd/github-repo/ssh-key

Remember to update the url and your SSM parameter key to match your repositories. Remember you are going to be using ssh and not https.

With that out of the way, we can apply the two templates:

1
2

kubectl apply -f secret-store.yaml
kubectl apply -f external-secret.yaml

If we get any errors and need to review what is going on, we can use the following command:

1
2

kubectl describe SecretStore my-aws-secret-store -n argocd
kubectl describe ExternalSecret private-repo-ssh-key -n argocd

And if we want to check if the secret created in Kubernetes matches our SSH key:

1
2

kubectl get secret my-aws-secret-repo-ssh-key \
  -n argocd -o jsonpath="{.data.sshPrivateKey}" | base64 -d

That's it. Now we can use our private repository in ArgoCD.

I created a private repository with the following url:

1

git@github.com:rderik/slartypriv.git

And the project in ArgoCD looks like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: server-proj-private
  namespace: argocd
  # Finaliser that ensures that the project is not deleted until any application does not reference it
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  # Project description
  description: Server Project
  destinations:
    # Update your namespace
    - namespace: my-namespace-private
      server: https://kubernetes.default.svc
  sourceRepos:
    # Make sure to add your repository here
    - git@github.com:rderik/slartypriv.git
  clusterResourceWhitelist:
  - group: ''
    kind: '*'

The application looks like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: server-private
  namespace: argocd
spec:
  destination:
    namespace: my-namespace-private
    server: https://kubernetes.default.svc
  project: server-proj-private
  source:
    # Change the repository to your repository
    repoURL: git@github.com:rderik/slartypriv.git
    # point the path to the path where your Kubernetes templates are
    path: server
  syncPolicy:
    automated:
      prune: false
      selfHeal: false
      allowEmpty: true

Notice that the sourceRepos and the repoURL have changed from https to ssh, so update your repository URLs correctly to use ssh. If we apply the project and the application, we should see it working without a problem in the ArgoCD dashboard.

1
2

kubectl apply -f project-private.yaml
kubectl apply -f application-private.yaml

And That's it! We now have a private repository in ArgoCD.

Final thoughts

There are many moving parts when we are working with Kubernetes and ArgoCD, and we need to pay a lot of attention to all the small details. I got stuck in the past debugging for a long time until I realised that the repoURL was still using https instead of ssh. And I got stuck another time until I realised I had a typo on the Secret template.

Anyways, I hope this is useful.

References


** If you want to check what else I'm currently doing, be sure to follow me on twitter @rderik or subscribe to the newsletter. If you want to send me a direct message, you can send it to derik@rderik.com.