v2.3 to 2.4

Known Issues

Broken project filter before 2.4.27

Argo CD 2.4.0 introduced a breaking API change, renaming the project filter to projects.

Impact to API clients

A similar issue applies to other API clients which communicate with the Argo CD API server via its REST API. If the client uses the project field to filter projects, the filter will not be applied. The failing project filter could have detrimental consequences if, for example, you rely on it to list Applications to be deleted.

Impact to CLI clients

CLI clients older that v2.4.0 rely on client-side filtering and are not impacted by this bug.

How to fix the problem

Upgrade to Argo CD >=2.4.27, >=2.5.15, or >=2.6.6. This version of Argo CD will accept both project and projects as valid filters.

KSonnet support is removed

Ksonnet was deprecated in 2019 and is no longer maintained. The time has come to remove it from the Argo CD.

Helm 2 support is removed

Helm 2 has not been officially supported since Nov 2020. In order to ensure a smooth transition, Helm 2 support was preserved in the Argo CD. We feel that Helm 3 is stable, and it is time to drop Helm 2 support.

Support for private repo SSH keys using the SHA-1 signature hash algorithm is removed

Note: this change was back-ported to 2.3.7 and 2.2.12.

Argo CD 2.4 upgraded its base image from Ubuntu 20.04 to Ubuntu 22.04, which upgraded OpenSSH to 8.9. OpenSSH starting with 8.8 dropped support for the ssh-rsa SHA-1 key signature algorithm.

The signature algorithm is not the same as the algorithm used when generating the key. There is no need to update keys.

The signature algorithm is negotiated with the SSH server when the connection is being set up. The client offers its list of accepted signature algorithms, and if the server has a match, the connection proceeds. For most SSH servers on up-to-date git providers, acceptable algorithms other than ssh-rsa should be available.

Before upgrading to Argo CD 2.4, check whether your git provider(s) using SSH authentication support algorithms newer than rsa-ssh.

1. Make sure your version of SSH >= 8.9 (the version used by Argo CD). If not, upgrade it before proceeding.

shell ssh -V

Example output: OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022

1. Once you have a recent version of OpenSSH, follow the directions from the OpenSSH 8.8 release notes:

To check whether a server is using the weak ssh-rsa public key algorithm, for host authentication, try to connect to it after removing the ssh-rsa algorithm from ssh(1)’s allowed list:

shell ssh -oHostKeyAlgorithms=-ssh-rsa user@host

If the host key verification fails and no other supported host key types are available, the server software on that host should be upgraded.

If the server does not support an acceptable version, you will get an error similar to this;

$ ssh -oHostKeyAlgorithms=-ssh-rsa vs-ssh.visualstudio.com Unable to negotiate with 20.42.134.1 port 22: no matching host key type found. Their offer: ssh-rsa

This indicates that the server needs to update its supported key signature algorithms, and Argo CD will not connect to it.

Workaround

The OpenSSH 8.8 release notes describe a workaround if you cannot change the server’s key signature algorithms configuration.

Incompatibility is more likely when connecting to older SSH implementations that have not been upgraded or have not closely tracked improvements in the SSH protocol. For these cases, it may be necessary to selectively re-enable RSA/SHA1 to allow connection and/or user authentication via the HostkeyAlgorithms and PubkeyAcceptedAlgorithms options. For example, the following stanza in ~/.ssh/config will enable RSA/SHA1 for host and user authentication for a single destination host:

Host old-host HostkeyAlgorithms +ssh-rsa PubkeyAcceptedAlgorithms +ssh-rsa

We recommend enabling RSA/SHA1 only as a stopgap measure until legacy implementations can be upgraded or reconfigured with another key type (such as ECDSA or Ed25519).

To apply this to Argo CD, you could create a ConfigMap with the desired ssh config file and then mount it at /home/argocd/.ssh/config.

Configure RBAC to account for new exec resource

2.4 introduces a new exec RBAC resource.

When you upgrade to 2.4, RBAC policies with * in the resource field and create or * in the action field will automatically grant the exec privilege.

To avoid granting the new privilege, replace the existing policy with a list of new policies explicitly listing the old resources.

The exec feature is disabled by default, but it is still a good idea to double-check your RBAC configuration to enforce least necessary privileges.

Example

Old:

p, role:org-admin, *, create, my-proj/*, allow

New:

p, role:org-admin, clusters, create, my-proj/*, allow
p, role:org-admin, projects, create, my-proj/*, allow
p, role:org-admin, applications, create, my-proj/*, allow
p, role:org-admin, repositories, create, my-proj/*, allow
p, role:org-admin, certificates, create, my-proj/*, allow
p, role:org-admin, accounts, create, my-proj/*, allow
p, role:org-admin, gpgkeys, create, my-proj/*, allow

Enable logs RBAC enforcement

2.4 introduced logs as a new RBAC resource. In 2.3, users with applications, get access automatically get logs access. In 2.5, you will have to explicitly grant logs, get access. Logs RBAC enforcement can be enabled with a flag in 2.4. We recommend enabling the flag now for an easier upgrade experience in 2.5.

Important

Logs RBAC enforcement will not be enabled by default in 2.5. This decision was made to avoid breaking logs access under Project Roles, which do not provide a mechanism to grant logs resource access.

To enabled logs RBAC enforcement, add this to your argocd-cm ConfigMap:

server.rbac.log.enforce.enable: "true"

If you want to allow the same users to continue to have logs access, just find every line that grants applications, get access and also grant logs, get.

Example

Old:

p, role:staging-db-admins, applications, get, staging-db-admins/*, allow
p, role:test-db-admins, applications, *, staging-db-admins/*, allow

New:

p, role:staging-db-admins, applications, get, staging-db-admins/*, allow
p, role:staging-db-admins, logs, get, staging-db-admins/*, allow
p, role:test-db-admins, applications, *, staging-db-admins/*, allow
p, role:test-db-admins, logs, get, staging-db-admins/*, allow

Pod Logs UI

Since 2.4.9, the LOGS tab in pod view is visible in the UI only for users with explicit allow get logs policy.

Known pod logs UI issue prior to 2.4.9

Upon pressing the “LOGS” tab in pod view by users who don’t have an explicit allow get logs policy, the red “unable to load data: Internal error” is received in the bottom of the screen, and “Failed to load data, please try again” is displayed.

Test repo-server with its new dedicated Service Account

As a security enhancement, the argocd-repo-server Deployment uses its own Service Account instead of default.

If you have a custom environment that might depend on repo-server using the default Service Account (such as a plugin that uses the Service Account for auth), be sure to test before deploying the 2.4 upgrade to production.

Plugins

Remove the shared volume from any sidecar plugins

As a security enhancement, sidecar plugins no longer share the /tmp directory with the repo-server.

If you have one or more sidecar plugins enabled, replace the /tmp volume mount for each sidecar to use a volume specific to each plugin.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
      - name: your-plugin-name
        volumeMounts:
        - mountPath: /tmp
          name: your-plugin-name-tmp
      volumes:
        # Add this volume.
        - name: your-plugin-name-tmp
          emptyDir: {}

Update plugins to use newly-prefixed environment variables

If you use plugins that depend on user-supplied environment variables, then they must be updated to be compatible with Argo CD 2.4. Here is an example of user-supplied environment variables in the plugin section of an Application spec:

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
  source:
    plugin:
      env:
        - name: FOO
          value: bar

Going forward, all user-supplied environment variables will be prefixed with ARGOCD_ENV_ before being sent to the plugin’s init, generate, or discover commands. This prevents users from setting potentially-sensitive environment variables.

If you have written a custom plugin which handles user-provided environment variables, update it to handle the new prefix.

If you use a third-party plugin which does not explicitly advertise Argo CD 2.4 support, it might not handle the prefixed environment variables. Open an issue with the plugin’s authors and confirm support before upgrading to Argo CD 2.4.

Confirm sidecar plugins have all necessary environment variables

A bug in < 2.4 caused init and generate commands to receive environment variables from the main repo-server container, taking precedence over environment variables from the plugin’s sidecar.

Starting in 2.4, sidecar plugins will not receive environment variables from the main repo-server container. Make sure that any environment variables necessary for the sidecar plugin to function are set on the sidecar plugin.

argocd-cm plugins will continue to receive environment variables from the main repo-server container.