Is Kubernetes a glass that is half empty or half full? Well, it depends on how you look at it! The mechanics of its orchestration capabilities are rather complex (in my opinion), and it’s not uncommon, even for the more experienced practitioners, to bang our heads against the wall because of it. At the same time, I’ve often recommended it for novice users and even for business cases in which the capacity for technical debt is more limited―for the purpose of working smarter, not harder. I’m working a second part-time job as a freelancer, maintaining a system for a small non-profit. So when I don’t have 8 hours a day to babysit, the more heavy lifting I can offload to K8s, the better life is for all of us. Even better, if I’m working for a large enterprise where I am paid 8 hours a day to babysit, I still would rather leverage the power of K8s to do most of that for me. This is the glass half-full perspective―considering K8s an ally who is there to make your life easier. Declare what you want and let it do what it does best.
As this approach may seem overly simplistic, I will also point out our tendency to make things overly complex. For instance, when we over-engineer a solution for some “future-proofing” that isn’t even on the horizon yet, or when we re-invent the wheel rather than using battle-tested tooling that is already purpose-built for the task. The dive into platform engineering is no exception, but we’ll get more into that in a subsequent article.
K8s is a powerful and complex organism, but remember that many of our industry peers have poured hours into making it something that “just works,” and like learning to drive a car, the best way to learn is hands-on.
I was first introduced to Kubernetes back in 2019―on the frontlines of Linode’s customer support team―when we released the Linode Kubernetes Engine (LKE). It was a sink or swim moment that we couldn’t avoid because the user demand was just too enormous. We had to support this product, meaning we had to learn it, and better yet…troubleshoot it! Albeit frustrating at times, these were some of the most valuable experiences of my career.
In this article, we’ll explore some strategies for learning and managing Kubernetes, based on real-world experiences like mine.
Learning by Doing
The best way to truly understand the platform is by building and breaking things. Combining documentation and tutorials is a great way to get your hands on a working cluster. Throw your manifests in a Git repo and you have a template you reference forever. Next, find an interesting way to break it. This could be letting a friend or co-worker take a stab at it so you have no idea the source of the problem until you start diagnosing. Or another method I’ve used for this is to do something more complex than what’s demonstrated in the guide. For example, say you’re taking the CKAD course provided by the Linux Foundation. They may instruct you to build a simplified cluster with just a single control plane and one worker node. Instead, try to successfully bootstrap a cluster with three control plane nodes for high availability and three worker nodes. If that wasn’t challenging enough, try implementing a VPC and customizing the cluster network addressing to avoid collisions in the IP space. Even if you get frustrated and throw in the towel after some time, you will have learned a lot more than you would have otherwise. These are just two examples from my own experience. The sky is the limit.
In addition, tools like Minikube or Kind can be super helpful for localized experimentation before touching anything in the cloud. Regardless of how you go about breaking, troubleshooting, and fixing your cluster, make sure you leave some trail of documentation. The act of writing it out serves your memory well, as it requires you to articulate the steps and solutions. If you have the time to make this public, you not only get to show off your stellar troubleshooting methodology, but you help your peers too.
Troubleshooting as a Learning Path
On the topic of troubleshooting methodology, there is no “best practice” per se, there is just “your practice” meaning you should find the itch that works best for you―whatever that is. What’s important is that you put the intention into building this muscle for yourself, and it gets easier over time. The best troubleshooters I’ve met became the best through years of doing exactly this and as a result, I can throw anything in front of them. Subsequently, they are also the spongiest learners.
Here are some techniques that work best for me:
- Divide and conquer: Cut the problem in half and systematically eliminate potential causes. While for some it’s easier to systematically climb the stack, I typically find success in jumping right to whatever I can rule out first. Does that defy the sound advice of others? Possibly! But I care more about what works for me.
- Monitoring and observability are your friends, and so are errors: Metrics, logs, and traces can paint a holistic view of the system, and better yet, a holistic view of the problem! Prometheus and Grafana are the de facto for monitoring and bode well with your choice of log aggregation and tracing tools. In the real world, however, you’re not always blessed with a full observability stack―sometimes the best you can get is Prometheus and Grafana with some remote targets to scrape. Fortunately, this still goes a very long way much of the time, and with or without, treat error messages as your friend! Sure, some are more helpful than others, but nonetheless, this is feedback from the system with information that something went awry.
- Recreate the problem: As much as possible, recreate the problem in a separate environment. Doing this can provide a wealth of information about the cause, which opens the doors to finding the solution. It also means you have a test environment that you can safely experiment with. One of many reasons it is good practice to leverage the power of infrastructure as code (IaC), is the ability to quickly recreate or destroy this environment as needed. Even if this means writing it yourself, in situations where the environment was not previously codified, spending a little more time on this upfront can save a ton more time after.
- Keep a troubleshooting log: You may be familiar with the concept of an architecture decision record/log. What’s to stop you from applying a similar concept to troubleshooting? Nothing! And it doesn’t need any special formatting or convention. Simply keep a record of your troubleshooting steps as you take them. This can be useful if you need to backtrack or reassert the things you’ve already ruled out. Best yet, you can review it later to document the solution and explain how you arrived there. Being able to articulate the steps and logic behind them can lay more permanent tracks in your memory. It will also prove useful to anyone who reads your documentation. A good candidate for this documentation is on an internal knowledge-base platform, or even a public-facing blog to teach others.
When facing particularly difficult problems, embracing troubleshooting as a learning opportunity can lead to long-term improvements in problem-solving skills and help improve the way you manage your infrastructure. Teams that refine and iterate their debugging processes will find themselves better equipped to enjoy the power of Kubernetes.
Using Simple, Reliable Tech
The cloud native ecosystem (and cloud landscape overall) offers a plethora of tools and frameworks to stitch together. The possibilities of what you can build are near endless! However, this notion can often lead to some major pitfalls: too many tools, too much complexity, and too much technical debt. Fortunately, this is an easy trap to avoid with just a little discipline and one mindset: less is more. The best tech is the most maintainable and the most stable tech, and you’ll find that your users tend to agree. If you needed to climb a ladder to get onto the roof of a house, you wouldn’t want it so over-engineered to the point that you have doubts about whether you’re using it correctly. That would be frightening!
There is no pride in maintaining a code base so complicated that no one can read it. There is no gold medal or trophy for building a system so complex that almost no one can use it. What we want are things that work, that are stable, and that we can reliably use correctly without piling on more toil. Simplicity is your friend here. Also, remember that a cloud native design pattern is one that embraces rapid change. Your application architecture will naturally gain complexity over time as it evolves, so there is no need to force that hand. With a minimalistic approach in addition to a modular cloud-native design, teams can have leaner and safer systems, and streamline deployments, maintenance, and troubleshooting.
Furthermore, keeping it simple helps to reduce the risk of misconfigurations that bottleneck performance and/or increase the attack surface for security flaws.
Kubernetes itself is an abstraction. We couldn’t call it “cloud native” if it wasn’t, but keeping the amount of unnecessary abstractions at bay means that engineers spend less time managing the underlying complexity, and more time focusing on delivering value. I stand by my own opinion here, against the grain of the “all you can eat” approach because the most advanced innovation I’ve seen throughout my career comes from companies that embody the “strict diet” approach; those who focused on mastering the basics, so that they can safely and reliably build some very cutting edge products that solve some very complex problems.
Continuing to Master Kubernetes
A combination of hands-on experience, a strong troubleshooting methodology, and a thoughtful approach to tooling can make Kubernetes much easier to master. Learning by doing (rather than just reading) helps engrain problem-solving skills that are critical in real-world operations. Troubleshooting isn’t just about fixing what’s broken; it’s an opportunity to refine your understanding, improve documentation, and develop a systematic approach that makes future challenges easier to solve. And simplifying your Kubernetes stack by minimizing unnecessary abstractions reduces cognitive overhead, making it easier to manage over time.
As many experienced practitioners have learned, a lean and well-structured cluster is easier to maintain and scale than one overloaded with tools that add complexity without delivering clear benefits. Ultimately, success with Kubernetes isn’t about using every new tool on the market, it’s about knowing which ones add real value in the long term.
If you’d like to learn more about Kubernetes, I talk about this in more detail on the KubeFM podcast, which you can watch on YouTube or below.
Comments