LKE Kubernetes Node Image Management
I have several questions about how OS images are managed in LKE. It appears my 1.18 nodes are running Debian 9.13 and kernel version
Linux 5.8.0-1-cloud-amd64 #1 SMP Debian 5.8.7-1 (2020-09-05) GNU/Linux and Kubelet version
It appears that nodes are fully managed by LKE and we are not able to run custom images when using LKE. Do you plan on allowing custom images to be used in the future in node pools?
How often are Kubernetes and OS security patches applied to base LKE images?
Is the recommended method of updating nodes to latest OS and patches to recycle nodes?
How do we know when a new image version is available for recycling? The Kubernetes versions endpoint does not list current patch version.
It appears kubelet and API versions are out of sync. API version is 1.18.12 and kubelets are at 1.18.8. Why?
I've found it useful in AWS EKS to be able to customize the image to install packages at the node level. For example, anything that requires kernel modules or other system-level packages.
If this node-level customization is needed, I'm not sure how I'd accomplish it with LKE. I'm also looking to solidify a strategy for updating nodes with security patches.
Greetings @kekoav! It's clear that you put a lot of time and thought into these questions, and I reached out for some input from our LKE developers to make sure we were getting you the right answers. Each of these are straight from our LKE team:
At the moment, we don't have any plans for custom images for LKE nodes. In the future we may offer other base images to select from, but unless we start getting similar requests, custom images aren't likely to be supported.
There is no definitive answer for this right now, and there's no regular cadence by which patches are released. We're looking to improve the process, though, and it's something that's been discussed very recently.
Yes - recycling is the recommended method for upgrading.
The recommended process right now is to use the API to check the list of currently available images. You can do this by grepping for "kube" and looking for relevant debian-kube images. This requires you to know which image you're currently using and then check periodically for an updated one. In the future, we'd like to proactively tell customers when upgrades are available.
We roll out upgrades to control plane components on every release; however, we don't currently build new images on every release. As I mentioned above, we're aiming to improve this process and get a regular cadence going. That said, Kubernetes allows version skew up to two minor versions, so being on different patch versions is completely fine.
If you need a more customizable option, you can always try manually setting up a cluster and managing your master node via Rancher. I know it bypasses LKE as a product, and I've only used it in a limited capacity myself, but the Rancher interface is pretty user friendly. If you're interested in trying it out, we have a guide here: How to Deploy Kubernetes on Linode with Rancher 2.3
I hope this helps clear some things up and that you're able to get things working for your use case. We also see that you've got a couple other Community posts about LKE awaiting answers, and we'll be answering those soon. :)
@jdutton Thanks for the responses, I am having fun digging into LKE. I'm just doing some due diligence here, sorry if I'm generating too many questions :)
I could do Rancher or kubespray myself but since I've been using k8s for a while I know enough to know that I don't trust myself alone to keep everything up and running.