Guides - Rotate Kubernetes Secrets
A managed Kubernetes service that enables you to easily control and scale your application’s infrastructure.
A Kubernetes secret is an object that stores sensitive authentication information like passwords, tokens, or keys. A secret is either auto-generated by the control plane and populated as a service-account-token or created manually by a user. Anyone with root-level access to a cluster has access to the “secrets” for that cluster. It is a best practice to rotate secrets on a regular basis or otherwise as-needed (i.e. an employee leaves a company or role, general security practices, etc.). This guide reviews the steps required to rotate the auto-generated service-account-token secrets in kube-system using kubectl, as well as how to rotate your lke-admin-token via the Linode CLI.
Note that the instructions in this guide only apply to service-account-token secrets in kube-system; depending on your application configuration, there may be additional user-generated secrets in other namespaces you may want to delete or rotate manually.
You will need to have the kubectl command line tool installed and your local system configured to communicate with your cluster. To do this, you must download your cluster’s kubeconfig file and save its file path to your
$KUBECONFIG environment variable. Instructions can be found here: Access and Download your kubeconfig
Once your local
$KUBECONFIG environment variable is configured with your cluster’s kubeconfig file, follow these steps to rotate the auto-generated service-account-token secrets within your cluster.
List the secrets in your cluster’s kube-system namespace and filter by service-account-token to see secrets auto-generated by the control plane:
kubectl -n kube-system get secrets --field-selector="type=kubernetes.io/service-account-token"
You should see output similar to the following:
NAME TYPE DATA AGE ccm-user-token-5d5lk kubernetes.io/service-account-token 3 59d cluster-autoscaler-user-token-sc6qr kubernetes.io/service-account-token 3 59d kubernetes-dashboard-lke-user-token-59vpx kubernetes.io/service-account-token 3 59d lke-admin-token-b9vdj kubernetes.io/service-account-token 3 59dIn Kubernetes version 1.26, there are additional auto-generated service-account-token secrets that exist beyond those included in the output above.
Delete the secrets.
kubectl -n kube-system delete secrets --field-selector="type=kubernetes.io/service-account-token"
You should receive a confirmation the secrets have been deleted:
secret "ccm-user-token-5d5lk" deleted secret "cluster-autoscaler-user-token-sc6qr" deleted secret "kubernetes-dashboard-lke-user-token-59vpx" deleted secret "lke-admin-token-b9vdj" deleted
Once your service-account-token secrets have been deleted, you will no longer have access to your cluster using your existing kubeconfig. To regain access, you will need to regenerate your kubeconfig and re-export your local environment variable using the steps below in Regenerate Your Kubeconfig.
In order to expedite secret recreation after deletion and regain access to your cluster, you can regenerate your kubeconfig. This process will also regenerate the lke-admin-token secret in kube-system. Note that these steps use the Linode CLI and follow the Kubernetes Cluster Regenerate instructions from our API documentation. See our API documentation for correlating API commands.
View your list of Kubernetes clusters to get the ID number (clusterId) of the cluster you wish to target:
linode-cli lke clusters-list
The clusterId can be found in the
┌────────┬─────────────────┬────────┬─────────────┬─────────────────────────────────┐ │ id │ label │ region │ k8s_version │ control_plane.high_availability │ ├────────┼─────────────────┼────────┼─────────────┼─────────────────────────────────┤ │ 12345 │ example-cluster │ us-iad │ 1.27 │ True │ └────────┴─────────────────┴────────┴─────────────┴─────────────────────────────────┘
Regenerate your cluster’s kubeconfig file using the clusterId obtained in step 1. Replace 12345 with your cluster’s ID number:
linode-cli lke regenerate 12345 --kubeconfig=true --servicetoken=true
Download your new kubeconfig file from Cloud Manager by navigating to the Kubernetes section, clicking on your cluster’s more options ellipsis, and selecting Download Kubeconfig. It may take a few minutes for the new kubeconfig file to become available.
Alternatively, you can use the Linode CLI to view your new kubeconfig and overwrite your existing kubeconfig.yaml file, replacing 12345 with your cluster ID and ~/Downloads/kubeconfig.yaml with the file path to your current kubeconfig:
linode-cli --json lke kubeconfig-view 12345 | jq -r '..kubeconfig' | base64 -d > ~/Downloads/kubeconfig.yaml
If you choose to download your new kubeconfig via the Cloud Manager, you will need to reconfigure your kubeconfig by saving the file path to your
$KUBECONFIGenvironment variable. Replace ~/Downloads/kubeconfig.yaml with the file path for your new kubeconfig file:
Verify that you have access to your cluster by viewing your nodes:
kubectl get nodes
Delete your old kubeconfig file once you have verified access to your cluster.
Now that a new kubeconfig has been generated and configured, your service-account-token secrets and lke-admin-token secret have been rotated.
This guide reviews steps for rotating secrets in the kube-system namespace used by the lke-managed control plane. As a best practice, it is recommended to use separate namespaces for any application-specific service-account-tokens and secrets.
Depending on your specific use case and configuration, there may be additional security steps you wish to take for your application-specific service-account-tokens once your secrets have been rotated, including restarting pods or recycling nodes.
View your pods:
kubectl get pods
Restart your pods:
kubectl rollout restart
When recycling a worker node, the corresponding Compute Instance is deleted and replaced with a new one. Nodes can be recycled via Cloud Manager using the instructions in our Manage Nodes and Node Pools guide here: Recycle Nodes
This page was originally published on