Diffing Customization

It is possible for an application to be OutOfSync even immediately after a successful Sync operation. Some reasons for this might be:

  • There is a bug in the manifest, where it contains extra/unknown fields from the actual K8s spec. These extra fields would get dropped when querying Kubernetes for the live state, resulting in an OutOfSync status indicating a missing field was detected.
  • The sync was performed (with pruning disabled), and there are resources which need to be deleted.
  • A controller or mutating webhook is altering the object after it was submitted to Kubernetes in a manner which contradicts Git.
  • A Helm chart is using a template function such as randAlphaNum, which generates different data every time helm template is invoked.
  • For Horizontal Pod Autoscaling (HPA) objects, the HPA controller is known to reorder spec.metrics in a specific order. See kubernetes issue #74099. To work around this, you can order spec.metrics in Git in the same order that the controller prefers.

In case it is impossible to fix the upstream issue, Argo CD allows you to optionally ignore differences of problematic resources. The diffing customization can be configured for single or multiple application resources or at a system level.

Application Level Configuration

Argo CD allows ignoring differences at a specific JSON path, using RFC6902 JSON patches. The following sample application is configured to ignore differences in spec.replicas for all deployments:

  1. spec:
  2.   ignoreDifferences:
  3.   - group: apps
  4.     kind: Deployment
  5.     jsonPointers:
  6.     - /spec/replicas

The above customization could be narrowed to a resource with the specified name and optional namespace:

  1. spec:
  2.   ignoreDifferences:
  3.   - group: apps
  4.     kind: Deployment
  5.     name: guestbook
  6.     namespace: default
  7.     jsonPointers:
  8.     - /spec/replicas

System-Level Configuration

The comparison of resources with well-known issues can be customized at a system level. Ignored differences can be configured for a specified group and kind in resource.customizations key of argocd-cm ConfigMap. Following is an example of a customization which ignores the caBundle field of a MutatingWebhookConfiguration webhooks:

  1. data:
  2.   resource.customizations: |
  3.     admissionregistration.k8s.io/MutatingWebhookConfiguration:
  4.       ignoreDifferences: |
  5.         jsonPointers:
  6.         - /webhooks/0/clientConfig/caBundle

The status field of CustomResourceDefinitions is often stored in Git/Helm manifest and should be ignored during diffing. The ignoreResourceStatusField setting simplifies handling that edge case:

  1. data:
  2.   resource.compareoptions: |
  3.     # disables status field diffing in specified resource types
  4.     # 'crd' - CustomResourceDefinition-s (default)
  5.     # 'all' - all resources
  6.     # 'none' - disabled
  7.     ignoreResourceStatusField: crd

By default status field is ignored during diffing for CustomResourceDefinition resource. The behavior can be extended to all resources using all value or disabled using none.

Known Kubernetes types in CRDs (Resource limits, Volume mounts etc)

Some CRDs are re-using data structures defined in the Kubernetes source base and therefore inheriting custom JSON/YAML marshaling. Custom marshalers might serialize CRDs in a slightly different format that causes false positives during drift detection.

A typical example is the argoproj.io/Rollout CRD that re-using core/v1/PodSpec data structure. Pod resource requests might be reformatted by the custom marshaller of IntOrString data type:

from:

  1. resources:
  2.   requests:
  3.     cpu: 100m

to:

  1. resources:
  2.   requests:
  3.     cpu: 0.1

The solution is to specify which CRDs fields are using built-in Kubernetes types in the resource.customizations section of argocd-cm ConfigMap:

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: argocd-cm
  5.   namespace: argocd
  6.   labels:
  7.     app.kubernetes.io/name: argocd-cm
  8.     app.kubernetes.io/part-of: argocd
  9. data:
  10.   resource.customizations: |
  11.     argoproj.io/Rollout:
  12.       knownTypeFields:
  13.       - field: spec.template.spec
  14.         type: core/v1/PodSpec

The list of supported Kubernetes types is available in diffing_known_types.txt