Table of Contents
Managing multiple Kubernetes clusters can be a daunting task, but Rancher simplifies the process by centralizing access, administration, and provisioning. However, when it comes to migrating clusters provisioned through Rancher, some limitations can arise that you need to be aware of.
Why Migrate Your K8S Rancher Cluster?
Rancher’s provisioning capabilities allow you to deploy clusters using various Kubernetes distributions and cloud providers through a single Terraform provider. However, one significant limitation is that you cannot unregister clusters provisioned through Rancher without destroying them. In contrast, imported clusters have a straightforward migration process.
Note: This article shares personal experiences and should not be considered an official procedure. Always back up your data before making any changes. For backups, I highly recommend using Velero, which we’ll discuss in a future article.
Current Scenario
In our case, we had an older Rancher instance running on an RKE cluster that we aimed to de-provision. Over time, we managed to detach several clusters as they were decommissioned. However, we still had two Azure AKS clusters, labeled “Rancher-Owned,” that were not scheduled for decommissioning anytime soon. Given that these are sensitive and longstanding production environments, we wanted to avoid extensive platform reconstruction.
Procedure Overview
The clusters in question are Azure AKS, but this procedure should work for other cluster types as long as you can generate a KUBECONFIG file independent of Rancher.
Step 1: Generate a KUBECONFIG
The first step is to create a local KUBECONFIG file, enabling you to manage the cluster directly via the API without going through Rancher. For Azure AKS, this is done using the Azure CLI:
export KUBECONFIG=~/.kube/tmp-aks.yaml
az aks get-credentials --resource-group MyRG --name MyCluster
After running the above commands, you can verify the nodes in your cluster:
kubectl get nodes
Step 2: Clean Up Rancher
Next, you need to remove the Rancher agent and any traces of Rancher from the cluster. Use the Rancher Cleanup tool to do this:
git clone https://github.com/rancher/rancher-cleanup.git
kubectl create -f deploy/rancher-cleanup.yaml
You can monitor the progress of this operation with:
kubectl -n kube-system logs -l job-name=cleanup-job -f
Once the job is completed, don’t forget to delete it to prevent accidental execution later:
kubectl delete -f deploy/rancher-cleanup.yaml
Now, your cluster should be restored to a standard Azure AKS environment without any Rancher traces.
Step 3: Remove the Cluster from Rancher
At this point, your old Rancher instance’s UI will likely display several error messages indicating communication issues with the cluster. This is normal, as the Rancher agent and RBAC account have been removed. However, you cannot delete the cluster through the UI just yet; Rancher will attempt to destroy it.
To resolve this, you’ll need to edit the cluster configuration in Rancher. This configuration is stored as a Kubernetes object, which you can access with kubectl
from your supporting cluster.
First, retrieve the cluster ID from the Rancher dashboard URL, which looks like this: https://rancher.mydomain.com/dashboard/c/c-12345/explorer#cluster-events
.
Then, you’ll want to remove the finalizers that handle complete deprovisioning when the cluster is deleted:
kubectl get clusters.management.cattle.io
kubectl patch clusters.management.cattle.io c-12345 -p '{"metadata":{"finalizers":null}}' --type=merge
After running this command, you can return to the Rancher UI, navigate to Cluster Management, and proceed to delete the cluster.
Step 4: Reintegrate into the Destination Rancher Instance
Finally, you can follow the standard procedure to import your AKS cluster into the new Rancher instance. If you’re using Terraform, take your existing code and add the imported = true
switch. Just be cautious about potential drifts:
resource "rancher2_cluster" "my-aks-to-import" {
name = "my-aks-to-import"
description = "Terraform AKS Cluster"
aks_config_v2 {
cloud_credential_id = rancher2_cloud_credential.aks.id
name = var.aws_aks_name
region = var.aws_region
imported = true
}
}
After a few minutes, your cluster should be available in the new Rancher instance, ready for further management.
Conclusion
Migrating a K8S Rancher cluster doesn’t have to be complicated. By following the steps outlined in this guide, you can effectively manage your Kubernetes clusters without facing the common pitfalls associated with Rancher’s provisioning model. Remember to back up your data and proceed cautiously to ensure a smooth migration.