How Do I Scrape Kubelet Prometheus Metrics in LKE?
I've read and followed the guide here for installing prometheus and grafana on LKE: https://www.linode.com/docs/guides/deploy-prometheus-operator-with-grafana-on-linode-kubernetes-engine/
I think it is incomplete though. As I open up prometheus to check that metrics are being scraped, there are several targets that are failing to scrape when you check the Status -> Targets page in the prometheus UI http://localhost:9090/targets.
The failures are kubelet, kube-proxy, and kube-prometheus-node-exporter metrics which are attempting to access host ports on the nodes. It appears the nodes have firewalls restricting this (which is a good thing!).
What is the recommended solution to fix this so that we can scrape kubelet metrics with prometheus? Obviously opening these ports on the network outside of the cluster is a bad idea, I would like to keep the traffic from prometheus pods to the host secure.
Thank you!
1 Reply
I was able to fix the prometheus-node-exporter by changing it to ClusterIP. Still am unable to access kubelet metrics.
I did notice in the kubelet-config-1.18 configmap, there is a value that could be problematic:
healthzBindAddress: 127.0.0.1
This would probably need to be set to 0.0.0.0 to be readable from pods.
I don't know how I'd change this at the moment, since the nodes are all managed and kubelet config is part of LKE. Any ideas on how I can get at these metrics?