Monitor PVC usage in Prometheus
I am using Prometheus to track cluster metrics. In all the documentation I can find, along with pre-packaged alerts, I see a metric, kubelet_volume_stats_available_bytes, that is being used to alert when a PVC is full or on track to become full. However, this metric is not being scraped and sent to Prometheus. In the Kubernetes documentation (https://kubernetes.io/docs/reference/instrumentation/metrics/), this metric is labeled as ALPHA stability. In my Grafana instance, I do see other kubelet metrics from Prometheus, many of them ALPHA, so I do not believe I have a configuration issue.
For those interested, I am simply using the Prometheus Community kube-prometheus-stack helm chart.
In my research, I did run across this issue regarding the AWS CSI driver.
https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/223
This got me wondering if the Linode CSI driver does not support these metrics. It would be great to receive confirmation one way or the other. And if the metric is unavailable, are other solutions being used for volume usage alerts? Or is there a timeline on when this metric might be added to the Linode CSI driver?
1 Reply
jhartman
Linode Staff
✓ Best Answer
I wanted to confirm that the LKE CSI-Driver currently does not expose that specific metric which admittedly complicates PVC space monitoring.
Although I haven't tested this myself, it may be possible to programmatically SSH into your LKE Nodes or exec into their respective Pods to run df -h to gather remaining space. Alerts could then be designed around those read outs to warn when space is either depleted or reaches set thresholds:
How can I use ssh to access my k8s cluster (22408) | Linode Questions
kubernetes - How to identify the storage space left in a persistent volume claim? - Stack Overflow
Otherwise, exposing that metric would be a helpful feature so I have submitted this for consideration within our internal request tracker.