Skip to main content
Version 3.0

Rollback

This guide describes the procedures for rolling back updates to TDP Kubernetes components, both via Helm and via ArgoCD. Rollback allows you to quickly restore a previous version in case of crashes or unexpected behavior after an update.

Attention

Before performing any rollback, back up the persistent data. Depending on the component, a rollback can cause data loss if schema migrations are backwards incompatible.

Data Backup before Rollback​

Before starting rollback, protect persistent data by creating snapshots of PVCs.

Identify Component PVCs​

Terminal input
kubectl get pvc -n <NAMESPACE> -l app.kubernetes.io/instance=<RELEASE_NAME>

Create Volume Snapshots​

Terminal input
kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: <RELEASE_NAME>-pre-rollback-snapshot
  namespace: <NAMESPACE>
spec:
  volumeSnapshotClassName: <snapshot-class>
  source:
    persistentVolumeClaimName: <pvc-name>
EOF

Check Snapshot Status​

Terminal input
kubectl get volumesnapshot -n <NAMESPACE>

Make sure the snapshot has a status of readyToUse: true before proceeding.

Rollback via Helm​

Consult Revision History​

Use the helm history command to view all revisions of a release:

Terminal input
helm history <RELEASE_NAME> -n <NAMESPACE>

The output will display a table with the columns: REVISION, UPDATED, STATUS, CHART, APP VERSION and DESCRIPTION. Identify the revision you want to revert to.

Example output:

REVISION    UPDATED                     STATUS        CHART              APP VERSION    DESCRIPTION
1           2025-01-15 10:30:00         deployed      tdp-kafka-1.0.0    3.5.1          Install complete
2           2025-02-10 14:00:00         superseded    tdp-kafka-1.0.0    3.5.1          Upgrade complete
3           2025-03-01 09:15:00         deployed      tdp-kafka-2.0.0    3.6.0          Upgrade complete

Rollback for a Specific Review​

To revert to a specific revision:

Terminal input
helm rollback <RELEASE_NAME> <revisao> -n <NAMESPACE>

For example, to roll back tdp-kafka to revision 2:

Terminal input
helm rollback tdp-kafka 2 -n <NAMESPACE>

Rollback to Previous Revision​

To revert to the immediately previous revision (without specifying the number):

Terminal input
helm rollback <RELEASE_NAME> -n <NAMESPACE>

Rollback with Pod Recreation​

In some cases, you may need to force all pods to be recreated during rollback:

Terminal input
helm rollback <RELEASE_NAME> <revisao> -n <NAMESPACE> --recreate-pods

Check Rollback Result​

After rollback, confirm that the release was successfully rolled back:

Terminal input
helm status <RELEASE_NAME> -n <NAMESPACE>
helm history <RELEASE_NAME> -n <NAMESPACE>

The last entry in the history must indicate STATUS: deployed with the description Rollback to <revision>.

Rollback via ArgoCD​

Rollback via ArgoCD consists of reverting the state of the Git repository to the previous version of the manifests.

Reverter o Manifesto no Git​

The recommended approach is to revert the commit that introduced the update:

Terminal input
git log --oneline -10  # Identify the commit for the update.
git revert <commit-hash>
git push origin main

ArgoCD will detect the change in the repository and synchronize the cluster automatically (if automatic synchronization is enabled).

Force Sync after Rollback​

If automatic sync is disabled, force sync manually:

Terminal input
argocd app sync tdp-kafka

Rollback pelo ArgoCD CLI​

ArgoCD also allows you to reverse the last sync operation:

Terminal input
argocd app rollback tdp-kafka
Note

Rollback via the ArgoCD CLI only rolls back the cluster state. The Git repository is not changed, which may cause an automatic resync restoring the unwanted version. Always roll back in Git as well to ensure consistency.

Check Status after Rollback​

Terminal input
argocd app get tdp-kafka

The status should indicate Synced and Healthy after the rollback is complete.

Common Rollback Scenarios​

Scenario 1: Pods failed to initialize​

Symptom: Pods are in CrashLoopBackOff or Error state after update.

Diagnosis:

Terminal input
kubectl describe pod -n <NAMESPACE> -l app.kubernetes.io/instance=<RELEASE_NAME>
kubectl logs -n <NAMESPACE> -l app.kubernetes.io/instance=<RELEASE_NAME> --previous

Solution: rollback to previous version:

Terminal input
helm rollback <RELEASE_NAME> -n <NAMESPACE>

Scenario 2: Database Schema Mismatch​

Symptom: Schema migration errors in component logs.

Diagnosis:

Terminal input
kubectl logs -n <NAMESPACE> -l app.kubernetes.io/instance=<RELEASE_NAME> | grep -i "migration\|schema\|database"

Solution:

  1. Restore the database backup (PVC snapshot)

  2. Rollback the release:

    Terminal input
    helm rollback <RELEASE_NAME> -n <NAMESPACE>

  3. Verify that the component initializes correctly with the restored database

Scenario 3: Connectivity Issues between Components​

Symptom: Components are unable to communicate after partial update.

Diagnosis:

Terminal input
kubectl get endpoints -n <NAMESPACE>
kubectl get svc -n <NAMESPACE>

Solution: Verify that all dependent components have been updated to supported versions. If necessary, roll back all affected components.

Scenario 4: Performance Degradation​

Symptom: High latency or excessive resource consumption after update.

Diagnosis:

Terminal input
kubectl top pods -n <NAMESPACE> -l app.kubernetes.io/instance=<RELEASE_NAME>
kubectl describe pod -n <NAMESPACE> -l app.kubernetes.io/instance=<RELEASE_NAME>

Solution: check configuration differences between versions and adjust resources in values.yaml. If the problem persists, perform a rollback.

Partial Rollback (Individual Component)​

In scenarios where only one component has problems after a general update, it is possible to roll back only that component.

Identify the Problematic Component​

Terminal input
kubectl get pods -n <NAMESPACE> --field-selector=status.phase!=Running
helm list -n <NAMESPACE> --failed

Rollback Individual via Helm​

Terminal input
helm rollback <release-problematico> -n <NAMESPACE>

Check Compatibility​

After partial rollback, check if the rolled back component is compatible with the other updated versions:

Terminal input
kubectl logs -n <NAMESPACE> -l app.kubernetes.io/instance=<release-problematico> --tail=50
kubectl get pods -n <NAMESPACE>
Attention

Partial rollback can cause incompatibilities between components if there are version dependencies. Please consult the compatibility matrix before proceeding.

Good Rollback Practices​

  1. Always back up persistent data before initiating rollback
  2. Document the cause of the rollback for future reference
  3. Test in a staging environment before updating production, reducing the need for rollbacks
  4. Monitor logs after rollback to confirm stability
  5. Keep Git consistent with the state of the cluster -- if using ArgoCD, always roll back to Git
  6. Do not skip revisions -- in case of multiple rollbacks, roll back one revision at a time