k8wiz
Back to Articles

Ready to test your Argo CD knowledge? Try our Argo CD quiz!

Fix an Argo CD application that stays OutOfSync

A step-by-step guide to finding why an Argo CD application is OutOfSync right after a successful sync, and fixing it without hiding real drift.

10 min read
By k8wiz Team
  • argocd
  • troubleshooting
  • gitops
  • diffing

You sync an application, the sync succeeds, and a few seconds later Argo CD marks it OutOfSync again. If auto-sync is on, you may also see the same sync run over and over. The Argo CD docs list the usual reasons: a field in Git that the API server drops, resources left behind because pruning is off, a controller or mutating webhook that changes the object after it is applied, and Helm templates that render something new every time.

The fix is almost never "sync again". You need to find the exact field that differs, work out who changed it, and then decide whether Git is wrong or Argo CD should stop comparing that field. This guide walks through that in order, on Argo CD 3.6.

Step 1: find the resources that are out of sync

An application is OutOfSync when at least one of its resources is. Start by finding which ones, because the application view in the UI can hide a single bad resource among hundreds of healthy ones.

The Application object keeps a sync status for every managed resource under status.resources. This command prints the ones that differ:

kubectl get application my-app -n argocd -o json \
  | jq -r '.status.resources[] | select(.status == "OutOfSync") | [.kind, .namespace, .name] | @tsv'

You can get the same information from the CLI with argocd app get my-app, which lists each resource with its sync status. Write the list down. Most of the time it is one or two resources of the same kind, and that already points to the cause: a Deployment hints at replicas, a webhook configuration hints at a CA bundle, a custom resource hints at a formatting problem.

Step 2: read the diff

Now look at what Argo CD thinks is different. Before you trust the result, refresh it. A normal refresh compares again, and a hard refresh also throws away the cached manifests from the repo-server, which matters when you suspect the manifest generation itself.

argocd app get my-app --hard-refresh
argocd app diff my-app

argocd app diff exits with 0 when there is no diff, 1 when it finds one and 2 on errors, so you can also use it in scripts and CI. It skips Secrets on purpose, so if the only OutOfSync resource is a Secret, use the diff view in the UI instead.

Read every changed line and ask one question: who wrote the live value? If you or a teammate ran kubectl edit, the live state is wrong and a sync with self-heal is the right answer. If a controller, a webhook or the API server wrote it, the live value is correct and the problem is in what Argo CD compares. The rest of this guide covers that second case.

Step 3: check for resources that need pruning

Not every OutOfSync resource has a field diff. If a resource exists in the cluster with the application's tracking annotation but is no longer in Git, Argo CD marks it as needing pruning. A sync with pruning disabled leaves it in place, so the application stays OutOfSync forever.

These resources carry requiresPruning: true in the application status:

kubectl get application my-app -n argocd -o json \
  | jq -r '.status.resources[] | select(.requiresPruning == true) | [.kind, .namespace, .name] | @tsv'

If they really should go, sync with pruning:

argocd app sync my-app --prune

Some resources are generated by a tool and should neither be pruned nor count against the sync status. Mark those with the IgnoreExtraneous compare option. It only affects the sync status. A Degraded resource still makes the application Degraded.

apiVersion: v1
kind: ConfigMap
metadata:
  name: generated-settings
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous

Pair it with the Prune=false sync option on the same resource if you also want Argo CD to leave it alone during a pruning sync.

Step 4: stop fighting a controller over a field

The classic case is a Deployment with a HorizontalPodAutoscaler. Git says replicas: 2, the HPA scales it to 5, and Argo CD reports the difference. With self-heal on, Argo CD sets it back to 2, the HPA scales it up again, and the loop never ends.

The cleanest fix is to delete replicas from the manifest in Git, so there is nothing to compare. When you can't, because a chart you don't own sets it, tell Argo CD to ignore the field with ignoreDifferences in the Application spec:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: web
      jsonPointers:
        - /spec/replicas
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true

The group is the API group without the version, and name narrows the rule to one object. Leave it out to match every Deployment in the application.

The sync option on the last line matters more than it looks. By default, ignoreDifferences only changes how Argo CD computes the sync status. During the sync itself, Argo CD still applies the full desired state, so the next sync that runs for any reason writes replicas: 2 again. RespectIgnoreDifferences=true makes Argo CD patch the ignored fields out of the desired state before it applies it. It only works on resources that already exist. On the first creation, the manifest goes in as written.

Argo CD 3.6 adds a second way to do this when you can't edit the Application, for example because an ApplicationSet generates it. Put the rule on the resource itself with the argocd.argoproj.io/ignore-differences annotation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  annotations:
    argocd.argoproj.io/ignore-differences: |-
      jsonPointers:
      - /spec/replicas

Argo CD merges these rules with any matching entries in spec.ignoreDifferences. The annotation only accepts JSON pointers, so the options in the next step still need the Application spec.

Step 5: handle webhooks and injected fields

Mutating webhooks change objects on their way into the cluster. A service mesh injects a sidecar or an init container, a policy engine adds labels, cert-manager's CA injector writes a caBundle into webhook configurations. None of that is in Git, so the diff shows it.

For list items, a JSON pointer is too blunt, because it addresses an index and the index can move. A jq path expression selects items by content instead:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jqPathExpressions:
        - .spec.template.spec.initContainers[] | select(.name == "istio-init")

When one controller owns several fields, you can ignore everything it manages, based on metadata.managedFields in the live object. Check the live object first to see the exact manager name:

kubectl get deployment web -n my-namespace --show-managed-fields -o yaml \
  | grep 'manager:'

Then list that manager in the rule:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      managedFieldsManagers:
        - kube-controller-manager

Be careful with this one. It hides every change that manager makes, including ones you might care about. Prefer a narrow JSON pointer when you know the field.

If the same rule applies to every application, move it to the argocd-cm ConfigMap so each team doesn't have to repeat it. This is the documented example for webhook CA bundles:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.ignoreDifferences.admissionregistration.k8s.io_MutatingWebhookConfiguration: |
    jqPathExpressions:
    - '.webhooks[]?.clientConfig.caBundle'

Since 3.0, Argo CD also uses these system-level rules to ignore resource updates. A change to an ignored field no longer triggers a new reconciliation of the application, which saves controller work on busy clusters.

Step 6: fix values the API server rewrites

Some diffs come from the API server, not from a controller. Kubernetes normalizes values when it stores them: 1000m of CPU becomes 1, 3072Mi becomes 3Gi, and 8760h becomes 8760h0m0s. This shows up most often in custom resources that embed Kubernetes types, such as an Argo Rollout with a pod template. The custom resource's marshaling can write the value differently, and the diff shows the same value written two ways.

Tell Argo CD which fields of the CRD use built-in types:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations.knownTypeFields.argoproj.io_Rollout: |
    - field: spec.template.spec
      type: core/v1/PodSpec

Another variant: your manifest contains a field the schema doesn't know, often a typo or a field at the wrong indentation level. The API server drops it, and Argo CD keeps reporting it as missing. That one is a real bug in Git. Fix the manifest instead of ignoring the field, because the setting you think you have doesn't exist in the cluster.

Helm charts that use randAlphaNum or similar functions render a different value on every helm template run, so the resource never matches. The best fix is to generate the value once, store it in a Secret you create outside the chart, and have the chart reference that Secret by name.

Step 7: try Server-Side Diff

The default diff strategy is a three-way diff between the live state, the desired state and the last-applied-configuration annotation. It runs inside Argo CD and has to guess what the API server would do with defaults and field ownership. Server-Side Diff asks the API server instead, by sending a Server-Side Apply in dry-run mode and comparing the result with the live object.

Many false diffs disappear with it, because defaults that the API server fills in are part of the predicted state. Validating webhooks also run at diff time, so an invalid manifest shows up before the sync instead of during it. Applications that use the ServerSideApply=true sync option already use it. For others, test it on one application first:

argocd app diff my-app --server-side-diff

If the diff gets shorter, enable it on that application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true

Changes from mutating webhooks are left out of a Server-Side Diff by default, which is usually what you want for injected sidecars. Add IncludeMutationWebhook=true to the same annotation, separated by a comma, if you want to see them. To turn it on for every application, set controller.diff.server.side: "true" in argocd-cmd-params-cm and restart the application controller.

Why auto-sync doesn't fix it for you

People often expect automated sync to keep retrying until the application is green. It doesn't. Automated sync makes one attempt per combination of commit SHA and application parameters. If the last sync already ran against that commit, Argo CD will not sync again, even though the application is still OutOfSync. It also won't retry a failed sync against the same commit.

With selfHeal: true, Argo CD does sync again when the live state drifts, after a short timeout (five seconds by default). That is what you want for manual edits, and it turns every unresolved diff from the steps above into an endless loop. So when an application flaps between Synced and OutOfSync with self-heal on, don't disable self-heal. Find the field from step 2 and fix it at the source, or ignore exactly that field and nothing more.

Test yourself

Ready to test your Argo CD knowledge? Try our Argo CD quiz!