How do I troubleshoot PVC mounting issues in LKE
Linode
Linode Staff
I keep running Volume mounting errors when deploying PVC's in LKE. Specifically I keep seeing the errors:
- FailedAttachVolume
- FailedMount
Are there any troubleshooting tips you can provide that will help me fix this?
1 Reply
Babbott417
Linode Staff
You may see the following errors that can happen in different stages of the PVC lifecycle, for example:
- FailedAttachVolume
- FailedMount
FailedAttachVolume:
You will typically see FailedAttachVolume present for a few reasons:
- API-server call failure
- There was a storage failure on a previous node
Troubleshooting:
- Kubectl describe pod -n [namespace] [pod]
This command will provide a more verbose explanation of the failures in the “Events” section near the end of the output.
FailedMount:
A FailedMount issue can happen for a number of reasons like:
- Incorrect access mode was defined on the new node
- The new node has too many disks attached
- A new node does not have enough mount points
- Network partitioning error
Resolving an issue:
The best first step to fixing the issue would be to fix the mount configuration if there is an issue with a wrong network path or network partition that is preventing the PV from mounting.
If there is still an issue after ruling out the network configuration you would want to try recycling the node or forcing the pod to run on another node:
Recycling nodes:
From your cluster’s page in Cloud Manager you can click the “Recycle” button next to the problematic node.
Generally speaking this tends to resolve this issue but you may want to try forcing the PV to mount on another node manually.
Manually scheduling PV
There are a few different approaches to do this:
- Mark the node as unable to be scheculed with the
kubectl cordon -n [node_name]command.
Don’t forget to reverse this after you are finished with kubectl uncordon -n [node_name]
- Delete the pod with
kubectl delete pod -n [pod_name]
If you have no available nodes or have tried the previous steps you can try and resolve the issue on a node level by:
- Reducing the number of disk partitions or mount points (a single LKE node can support up to 7 Block Storage Volumes)
- Checking the access mode on the new node (ReadWriteOnce) is the only Linode supported access mode).