Guides - Drain a Node Pool
A managed Kubernetes service that enables you to easily control and scale your application’s infrastructure.
You can use
kubectl drain to safely evict all of the pods from a node before you perform maintenance on the node such as kernel upgrade, hardware maintenance, and others. Safe evictions allow the containers of the pods to gracefully terminate and respect the PodDisruptionBudgets that you specified. For more information see, Disruptions.
Kubernetes workloads move around the cluster, which enables use cases like highly available distributed systems. Linode recommends you to move any data storage on the filesystem of the Compute Instances in an LKE cluster to Persistent Volumes with network attached storage. Avoid using local storage on LKE nodes whenever possible. If you are using Persistent Volume Claim for the application on an LKE cluster, skip the entire Copy the application data to a Persistent Volume section and proceed directly to Add a new node pool to the cluster and drain the nodes.
This guide provides instructions to:
- Copy the application data to a Persistent Volume if you are using a local storage to store application data.
- Add a new node pool to the cluster and then drain the nodes.
This guide assumes you have a working Linode Kubernetes Engine (LKE) cluster running on Linode and you are familiar with PodDisruptionBudget concept and Configured PodDisruptionBudgets for applications that need them.
Install the Kubernetes CLI (
kubectl) on the local computer.
Follow the instructions in Deploying and Managing a Cluster with Linode Kubernetes Engine Tutorial to connect to an LKE cluster.Ensure that the
KUBECONFIGcontext is persistent
Ensure that Kubernetes CLI is using the right cluster context. Run the
get-contextssubcommand to check:
kubectl config get-contexts
The instructions in this section creates a Block Storage volume billable resource on your Linode account. A single volume can range from 10 GB to 10,000 GB in size and costs $0.10/GB per month or $0.00015/GB per hour. If you do not want to keep using the Block Storage volume that you create, be sure to delete it when you have finished the guide.
If you remove the resources afterward, you are only billed for the hour(s) that the resources were present on your account. Consult the Billing and Payments guide for detailed information about how hourly billing works and for a table of plan pricing.
Create a Persistent Volume Claim (PVC) that consumes a Block Storage Volume. To create a PVC, create a manifest file with the following YAML:
To retain the Block Storage Volume and its data, even after the associated PVC is deleted, use the
- File: pvc.yaml
1 2 3 4 5 6 7 8 9 10 11
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-test spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: linode-block-storage-retain
linode-block-storage-retainStorageClass. If, instead, you prefer to have the Block Storage Volume and its data deleted along with its PVC, use the
linode-block-storageStorageClass. For more information, see the Delete a Persistent Volume Claim.
The PVC represents a Block Storage Volume. Because Block Storage Volumes have a minimum size of 10 gigabytes, the storage has been set to
10Gi. If you choose a size smaller than 10 gigabytes, the PVC defaults to 10 gigabytes. Currently the only mode supported by the Linode Block Storage CSI driver is
ReadWriteOnce, meaning that it can only be connected to one Kubernetes node at a time.
Create the PVC in Kubernetes, and pass in the
kubectl create -f pvc.yaml
After a few moments the Block Storage Volume is provisioned and the Persistent Volume Claim is ready to use.
Check the status of the PVC by typing the following command:
kubectl get pvc
An output similar to the following appears:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-test Bound pvc-0e95b811652111e9 10Gi RWO linode-block-storage-retain 2m
You can now attach the PVC to a Pod.
Create a manifest file for the new Pod using the following YAML, where
applicationis using local storage at
pvc-testis a Persistent Volume Claim at
- File: new-pod.yaml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
apiVersion: v1 kind: Pod metadata: name: new-pod labels: app: application volumes: - name: application hostPath: path: $HOSTPATH - name: pvc-test persistentVolumeClaim: claimName: pvc-test ........ volumeMounts: - name: application mountPath: $MOUNTPATH - name: pvc-test mountPath: $CSIVolumePath
Create a new Pod named
kubectl create -f new-pod.yaml
After a few moments the Pod should be up and running. To check the status of the Pod, type the following command:
kubectl get pods
An output similar to following appears:
NAME READY STATUS RESTARTS AGE new-pod 1/1 Running 0 2m
Connect to a shell in the new Pod, type the following command:
kubectl exec -it new-pod -- /bin/bash
From the shell, copy the files from local storage to the PVC. In the following command
$MOUNTPATHis the location of the local storage and
$CSIVolumePathis the location on the PVC:
cp -P $MOUNTPATH $CSIVolumePath.
Delete the new Pod that you created, and then re-create it:
kubectl delete pod new-pod kubectl create -f new-pod.yaml
You should now see that all the data is stored in the CSI Volume.
Add an additional Node Pool to the LKE cluster, of a plan type and size which can accommodate the existing workloads.
After the new Compute Instances have joined the cluster, drain any instances that are scheduled for maintenance. This causes the workloads to be rescheduled to other Compute Instances in the cluster. Linode recommends draining one instance at a time to ensure that the workloads have been rescheduled to other instances. An example Node drain command:
kubectl drain lke9297-11573-5f3e357cb447
You can delete the old Node Pool or choose to keep it for after the maintenance is complete. Note, if you keep the Node Pool, you will be charged for it.
When the maintenance has been completed and if you kept your previous Compute Instances, after they have booted you can mark them as scheduled again by using the following command:
kubectl uncordon lke9297-11573-5f3e357cb447
You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.
This page was originally published on