Using GitOps with Linkerd with Argo CD

GitOps is an approach to automate the management and delivery of your Kubernetes infrastructure and applications using Git as a single source of truth. It usually utilizes some software agents to detect and reconcile any divergence between version-controlled artifacts in Git with what’s running in a cluster.

This guide will show you how to set up Argo CD to manage the installation and upgrade of Linkerd using a GitOps workflow.

Specifically, this guide provides instructions on how to securely generate and manage Linkerd’s mTLS private keys and certificates using Sealed Secrets and cert-manager. It will also show you how to integrate the auto proxy injection feature into your workflow. Finally, this guide conclude with steps to upgrade Linkerd to a newer version following a GitOps workflow.

Linkerd GitOps workflow

Linkerd GitOps workflow

The software and tools used in this guide are selected for demonstration purposes only. Feel free to choose others that are most suited for your requirements.

You will need to clone this example repository to your local machine and replicate it in your Kubernetes cluster following the steps defined in the next section.

Set up the repositories

Clone the example repository to your local machine:

  1. git clone https://github.com/linkerd/linkerd-examples.git

This repository will be used to demonstrate Git operations like add, commit and push later in this guide.

Add a new remote endpoint to the repository to point to the in-cluster Git server, which will be set up in the next section:

  1. cd linkerd-examples
  3. git remote add git-server git://localhost/linkerd-examples.git

Note

To simplify the steps in this guide, we will be interacting with the in-cluster Git server via port-forwarding. Hence, the remote endpoint that we just created targets your localhost.

Deploy the Git server to the scm namespace in your cluster:

  1. kubectl apply -f gitops/resources/git-server.yaml

Later in this guide, Argo CD will be configured to watch the repositories hosted by this Git server.

Note

This Git server is configured to run as a daemon over the git protocol, with unauthenticated access to the Git data. This setup is not recommended for production use.

Confirm that the Git server is healthy:

  1. kubectl -n scm rollout status deploy/git-server

Clone the example repository to your in-cluster Git server:

  1. git_server=`kubectl -n scm get po -l app=git-server -oname | awk -F/ '{ print $2 }'`
  3. kubectl -n scm exec "${git_server}" -- \
  4.   git clone --bare https://github.com/linkerd/linkerd-examples.git

Confirm that the remote repository is successfully cloned:

  1. kubectl -n scm exec "${git_server}" -- ls -al /git/linkerd-examples.git

Confirm that you can push from the local repository to the remote repository via port-forwarding:

  1. kubectl -n scm port-forward "${git_server}" 9418  &
  3. git push git-server master

Deploy Argo CD

Install Argo CD:

  1. kubectl create ns argocd
  3. kubectl -n argocd apply -f \
  4.   https://raw.githubusercontent.com/argoproj/argo-cd/v1.6.1/manifests/install.yaml

Confirm that all the pods are ready:

  1. for deploy in "application-controller" "dex-server" "redis" "repo-server" "server"; \
  2.   do kubectl -n argocd rollout status deploy/argocd-${deploy}; \
  3. done

Use port-forward to access the Argo CD dashboard:

  1. kubectl -n argocd port-forward svc/argocd-server 8080:443  \
  2.   > /dev/null 2>&1 &

The Argo CD dashboard is now accessible at https://localhost:8080, using the default admin username and password.

Note

The default admin password is the auto-generated name of the Argo CD API server pod. You can use the argocd account update-password command to change it.

Authenticate the Argo CD CLI:

  1. argocd_server=`kubectl -n argocd get pods -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2`
  3. argocd login 127.0.0.1:8080 \
  4.   --username=admin \
  5.   --password="${argocd_server}" \
  6.   --insecure

Configure project access and permissions

Set up the demo project to group our applications:

  1. kubectl apply -f gitops/project.yaml

This project defines the list of permitted resource kinds and target clusters that our applications can work with.

Confirm that the project is deployed correctly:

  1. argocd proj get demo

On the dashboard:

New project in Argo CD dashboard

New project in Argo CD dashboard

Deploy the applications

Deploy the main application which serves as the “parent” application that of all the other applications:

  1. kubectl apply -f gitops/main.yaml

Note

The “app of apps” pattern is commonly used in Argo CD workflows to bootstrap applications. See the Argo CD documentation for more information.

Confirm that the main application is deployed successfully:

  1. argocd app get main

Sync the main application:

  1. argocd app sync main

Synchronize the main application

Synchronize the main application

Notice that only the main application is synchronized.

Next, we will synchronize the remaining applications individually.

Deploy cert-manager

Synchronize the cert-manager application:

  1. argocd app sync cert-manager

Note

This guide uses cert-manager 0.15.0 due to an issue with cert-manager 0.16.0 and kubectl <1.19 and Helm 3.2, which Argo CD uses. See the upgrade notes here.

Confirm that cert-manager is running:

  1. for deploy in "cert-manager" "cert-manager-cainjector" "cert-manager-webhook"; \
  2.   do kubectl -n cert-manager rollout status deploy/${deploy}; \
  3. done

Synchronize the cert-manager application

Synchronize the cert-manager application

Deploy Sealed Secrets

Synchronize the sealed-secrets application:

  1. argocd app sync sealed-secrets

Confirm that sealed-secrets is running:

  1. kubectl -n kube-system rollout status deploy/sealed-secrets

Synchronize the sealed-secrets application

Synchronize the sealed-secrets application

Create mTLS trust anchor

Before proceeding with deploying Linkerd, we will need to create the mTLS trust anchor. Then we will also set up the linkerd-bootstrap application to manage the trust anchor certificate.

Create a new mTLS trust anchor private key and certificate:

  1. step certificate create root.linkerd.cluster.local sample-trust.crt sample-trust.key \
  2.   --profile root-ca \
  3.   --no-password \
  4.   --not-after 43800h \
  5.   --insecure

Confirm the details (encryption algorithm, expiry date, SAN etc.) of the new trust anchor:

  1. step certificate inspect sample-trust.crt

Create a SealedSecret resource to store the encrypted trust anchor:

  1. kubectl -n linkerd create secret tls linkerd-trust-anchor \
  2.   --cert sample-trust.crt \
  3.   --key sample-trust.key \
  4.   --dry-run=client -oyaml | \
  5. kubeseal --controller-name=sealed-secrets -oyaml - | \
  6. kubectl patch -f - \
  7.   -p '{"spec": {"template": {"type":"kubernetes.io/tls", "metadata": {"labels": {"linkerd.io/control-plane-component":"identity", "linkerd.io/control-plane-ns":"linkerd"}, "annotations": {"linkerd.io/created-by":"linkerd/cli stable-2.8.1", "linkerd.io/identity-issuer-expiry":"2021-07-19T20:51:01Z"}}}}}' \
  8.   --dry-run=client \
  9.   --type=merge \
  10.   --local -oyaml > gitops/resources/linkerd/trust-anchor.yaml

This will overwrite the existing SealedSecret resource in your local gitops/resources/linkerd/trust-anchor.yaml file. We will push this change to the in-cluster Git server.

Confirm that only the spec.encryptedData is changed:

  1. git diff gitops/resources/linkerd/trust-anchor.yaml

Commit and push the new trust anchor secret to your in-cluster Git server:

  1. git add gitops/resources/linkerd/trust-anchor.yaml
  3. git commit -m "update encrypted trust anchor"
  5. git push git-server master

Confirm the commit is successfully pushed:

  1. kubectl -n scm exec "${git_server}" -- git --git-dir linkerd-examples.git log -1

Deploy linkerd-bootstrap

Synchronize the linkerd-bootstrap application:

  1. argocd app sync linkerd-bootstrap

Note

If the issuer and certificate resources appear in a degraded state, it’s likely that the SealedSecrets controller failed to decrypt the sealed linkerd-trust-anchor secret. Check the SealedSecrets controller for error logs.

For debugging purposes, the sealed resource can be retrieved using the kubectl -n linkerd get sealedsecrets linkerd-trust-anchor -oyaml command. Ensure that this resource matches the gitops/resources/linkerd/trust-anchor.yaml file you pushed to the in-cluster Git server earlier.

Synchronize the linkerd-bootstrap application

Synchronize the linkerd-bootstrap application

SealedSecrets should have created a secret containing the decrypted trust anchor. Retrieve the decrypted trust anchor from the secret:

  1. trust_anchor=`kubectl -n linkerd get secret linkerd-trust-anchor -ojsonpath="{.data['tls\.crt']}" | base64 -d -w 0 -`

Confirm that it matches the decrypted trust anchor certificate you created earlier in your local sample-trust.crt file:

  1. diff -b \
  2.   <(echo "${trust_anchor}" | step certificate inspect -) \
  3.   <(step certificate inspect sample-trust.crt)

Deploy Linkerd

Now we are ready to install Linkerd. The decrypted trust anchor we just retrieved will be passed to the installation process using the identityTrustAnchorsPEM parameter.

Prior to installing Linkerd, note that the global.identityTrustAnchorsPEM parameter is set to an “empty” certificate string:

  1. argocd app get linkerd -ojson | \
  2.   jq -r '.spec.source.helm.parameters[] | select(.name == "identityTrustAnchorsPEM") | .value'

Empty default trust anchor

Empty default trust anchor

We will override this parameter in the linkerd application with the value of ${trust_anchor}.

Locate the identityTrustAnchorsPEM variable in your local gitops/argo-apps/linkerd.yaml file, and set its value to that of ${trust_anchor}.

Ensure that the multi-line string is indented correctly. E.g.,

  1.   source:
  2.     chart: linkerd2
  3.     repoURL: https://helm.linkerd.io/stable
  4.     targetRevision: 2.8.0
  5.     helm:
  6.       parameters:
  7.       - name: identityTrustAnchorsPEM
  8.         value: |
  9.           -----BEGIN CERTIFICATE-----
  10.           MIIBlTCCATygAwIBAgIRAKQr9ASqULvXDeyWpY1LJUQwCgYIKoZIzj0EAwIwKTEn
  11.           MCUGA1UEAxMeaWRlbnRpdHkubGlua2VyZC5jbHVzdGVyLmxvY2FsMB4XDTIwMDkx
  12.           ODIwMTAxMFoXDTI1MDkxNzIwMTAxMFowKTEnMCUGA1UEAxMeaWRlbnRpdHkubGlu
  13.           a2VyZC5jbHVzdGVyLmxvY2FsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE+PUp
  14.           IR74PsU+geheoyseycyquYyes5eeksIb5FDm8ptOXQ2xPcBpvesZkj6uIyS3k4qV
  15.           E0S9VtMmHNeycL7446NFMEMwDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYB
  16.           Af8CAQEwHQYDVR0OBBYEFHypCh7hiSLNxsKhMylQgqD9t7NNMAoGCCqGSM49BAMC
  17.           A0cAMEQCIEWhI86bXWEd4wKTnG07hBfBuVCT0bxopaYnn3wRFx7UAiAwXyh5uaVg
  18.           MwCC5xL+PM+bm3PRqtrmI6TocWH07GbMxg==
  19.           -----END CERTIFICATE-----

Confirm that only one spec.source.helm.parameters.value field is changed:

  1. git diff gitops/argo-apps/linkerd.yaml

Commit and push the changes to the Git server:

  1. git add gitops/argo-apps/linkerd.yaml
  3. git commit -m "set identityTrustAnchorsPEM parameter"
  5. git push git-server master

Synchronize the main application:

  1. argocd app sync main

Confirm that the new trust anchor is picked up by the linkerd application:

  1. argocd app get linkerd -ojson | \
  2.   jq -r '.spec.source.helm.parameters[] | select(.name == "identityTrustAnchorsPEM") | .value'

Override mTLS trust anchor

Override mTLS trust anchor

Synchronize the linkerd application:

  1. argocd app sync linkerd

Check that Linkerd is ready:

  1. linkerd check

Synchronize Linkerd

Synchronize Linkerd

Test with emojivoto

Deploy emojivoto to test auto proxy injection:

  1. argocd app sync emojivoto

Check that the applications are healthy:

  1. for deploy in "emoji" "vote-bot" "voting" "web" ; \
  2.   do kubectl -n emojivoto rollout status deploy/${deploy}; \
  3. done

Synchronize emojivoto

Synchronize emojivoto

Upgrade Linkerd to 2.8.1

Use your editor to change the spec.source.targetRevision field to 2.8.1 in the gitops/argo-apps/linkerd.yaml file:

Confirm that only the targetRevision field is changed:

  1. git diff gitops/argo-apps/linkerd.yaml

Commit and push this change to the Git server:

  1. git add gitops/argo-apps/linkerd.yaml
  3. git commit -m "upgrade Linkerd to 2.8.1"
  5. git push git-server master

Synchronize the main application:

  1. argocd app sync main

Synchronize the linkerd application:

  1. argocd app sync linkerd

Confirm that the upgrade completed successfully:

  1. linkerd check

Confirm the new version of the control plane:

  1. linkerd version

Clean up

All the applications can be removed by removing the main application:

  1. argocd app delete main --cascade=true