Kubernetes and the Importance of Friendship
In which I discuss a CNCF survey, share everything I know about Kube and reveal my secret to achieving mature FinOps processes in a container environment. Then music with brass sections.
No man is an Iland, entire of itself; every man is a piece of the Continent, a part of the maine; if a Clod be washed away by the Sea, Europe is the less, as well as if a Promontorie were, as well as if a Mannor of thy friends or of thine owne were; any mans death diminishes me, because I am involved in Mankinde; And therefore never send to know for whom the bell tolls; it tolls for thee.
― John Donne
The CNCF published a survey in June that highlighted the growing adoption of Kubernetes and the far slower growth of FinOps practices in Kubernetes environments. It’s an interesting survey that shows that we’re still in the early phases of this FinOps movement. While 68% of the survey participants saw their Kube costs grow significantly, 24% of participants do absolutely no cost monitoring and 44% do some form of estimating of the cost impact of their environment. This is: not great.
Allow me to humble-brag and state that I’ve worked as part of a team that successfully implemented an accurate chargeback of their Kubernetes environment. Ooh-la-la! How did we do it? Well, let's start at the beginning with a quick review of Kubernetes.
Kubernetes is a portable, extensible, open-source platform that orchestrates containerized workloads and services, that facilitates both declarative configu . . .
Oh-my-gawd! Look at this squirrel eating the seeds I left out for it!
So cute! I even left a little water for it. Fun little guy, just chilling and having a snack
Where was I? Kubernetes . . . gets you some super-efficient compute resources. What else do you need to know? Look, this stuff is complicated - thus the lack of mature FinOps practices around it. There are concepts like pods and namespaces and labels and . . . there’s more but my brain shuts off after a certain point, and the squirrel is STILL sitting there, chomping away. Have you ever noticed how squirrels look different in different countries? Some have furry ears. Some are tiny. Some are huge. In Australia, they stand on their hindlegs, hop around and have a pouch on their tummy.
Kubernetes is not my strength. The likelihood of me presenting at KubeCon is low (although I have a presentation slide title: Kubernetes and the Squirrels of North America.) Yet despite my brain's inability to hold the thread of Kubernetes, I run a team that has a fairly mature Kube FinOps process. How did someone so not smart at Kube get a K8 environment with mature FinOps processes? Here is the moral of today’s story: it isn’t as much about knowing everything yourself as it is about partnering with smart people.
I’m a generalist. Some things I have deep knowledge on and some things I have squirrel knowledge on. It’s important to know who to go to for help with your squirrel knowledge. The reason we have a mature Kube chargeback process is that I work with some intensely smart Kube subject matter experts (SMEs) and I can go to them for help.
Is the theory that simple? Find the smartest engineers and go to them for help with any question that arises. LOLZ! That will work exactly twice before they get pissed off.
The Kube SMEs and I cannot hold a conversation for too long. They get frustrated with my inability to hold knowledge in my head and I get frustrated with their inability to dumb down their answers to a third-grade level. (Yes, I’ve seen The Illustrated Children’s Guide to Kubernetes. Don’t patronize me. I said 3rd Grade, not 3 years old. It moves far too slowly for me to get through.) The bottom line is they ‘like me’ but they don’t ‘like-me like me.’ This begs the question, is it harder to learn Kubernetes or to become friends with someone who understands how Kubernetes actually works?
Both are difficult, but I still think it’s easier to try to be friends with nerds than actually be one. I cheated a bit. I have a nerd sherpa. His name is Matt. Matt has one foot in the world with fresh air and squirrels and his other foot in the world of Kubernetes and living in parents' basements. It’s important that you search people out like him. Matt can hear my needs, engage his team in the appropriate way and they can come up with a solution for how we use labels to identify chargeback, allocate CPU and memory and make it all visual for people like me to slowly gain knowledge. Matt makes it so I feel comfortable not being a subject matter expert and helps me not drain the mental energy of his subject matter experts.
In a nutshell (#squirrel): find the nerds, engage them appropriately, be extremely nice and appreciative of them, have them show you what is smart for your environment, and then thank them profusely.
A note on tooling: tools do tend to make things easier, don’t they? There are a variety of tools out there, as shown in the CNCF survey:
An important lesson on tools - generalists shouldn’t expect to force a tool on a group of SMEs and expect them to be happy about it. It’s like going up to DaVinci and saying ‘you are an amazing artist, but I really insist on you using this finger paint . . .’ Perhaps you have a tool that is part of a suite that you already paid for? That is understandable. Ask the SMEs what the strengths and weaknesses of the tool are, is it a workable solution and if yes, how best to implement it? Make sure they are part of the decision.
You don’t gain FinOps maturity by yourself. It’s a team sport. You need nerds, or rather, you need deep subject matter experts. You need generalists. You absolutely need to know where to go when you need help. Then you just need to start.
Recommended Reading:
Another subject matter expert I can have a conversation with is Bhups Hirani. Bhups wrote a post about running cost-efficient EKS clusters. I particularly appreciate his discussion on RI vs Spot as well as his tips on what to watch out for. Give it a read: https://medium.com/ynap-tech/running-cost-efficient-eks-clusters-40043fb6dd44
Brass Monkey
No, not a Beastie Boys playlist - although that's a great idea (My favorite of theirs is Make Some Noise from their not listened to enough last album, Hot Sauce Committee Part Two). I like brass sections in my music. Sure guitars are cool, but they don’t have spit valves.
You and Me - Meute (German Techno Marching Band. Watch their YouTube videos. You’re welcome.)
Colors - Black Pumas and The Hypnotic Brass Ensemble (I picked the original Colors on another list, but it's even better with a brass section. This is my preferred version of the song.)
100 Days, 100 Nights - Sharon Jones & The Dap-Kings (second time I’ve picked Miss Sharon Jones, and I should probably have picked her for every single list because she was THE BEST.)
Baker’s Dozen - Galactic (The first time I ever heard this band, I nearly fell off a treadmill.)
I Need Never Get Old - Nathaniel Rateliff & The Night Sweats (Have become my favorite band over the last few years - consistently great songs that make my feet move.)
Do you enjoy my long-form ramblings on FinOps? Sure you do, subscribe and make sure you don’t miss any of my thoughts on what we face and of course - the music.