Rendered at 15:53:33 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
zrail 5 days ago [-]
Hey, author here. This is a piece about moving away from kubernetes and toward something that I can actually maintain as a solo person who has a life outside of k9s. It's not really intended to be "anti-kubernetes", more like "kubernetes really is too hard for my purposes".
IMO the best change that I've made has been to give deterministic IPv6 addresses to every container and then using those for ingress.
I'm curious to hear where y'all think the line is between docker compose with Ruby glue and "Dear friend, you have built a Kubernetes".
sierra1011 7 hours ago [-]
For transparency, I work with K8s every day and run it with FluxCD as my homelab and have in various formats for a few years.
Before that though, I had a single computer (a NUC if memory serves) with systemd running docker containers. Dead simple.
lloydatkinson 7 hours ago [-]
Did you ever consider NixOS for your homelab?
MisterKent 9 hours ago [-]
Lot of kubernetes hate here, which is surprising. I run a little 3 node cluster and besides the hardware issues I had (long story), it has been rock solid and dead easy to setup.
Talos + longhorn + fluxcd (optional), is super nice. And everything beyond that is additive and just works within the ecosystem.
If anything, it helped keep my stuff alive during all the hardware issues a lot longer.
I think like 5-6 years ago, kubernetes on baremetal was pretty painful. People should really give it another try, an LLM can probably set it up for you and fire off the docker compose to manifests in one shot. Or just follow the docs yourself, maybe a dozen commands to get a cluster running?
All the enterprisey stuff makes it feel a lot more complex than it really is.
dspillett 4 hours ago [-]
> an LLM can probably
That would not match their reason for preferring to move off k8:
> I built a system that I didn't actually know how to maintain without the time or energy necessary to dig myself out of trouble.
If you want a system that you can understand all at once in your own head then making it with your own head is a better way to go IMO, for a small system at least.
3 hours ago [-]
wannabe44 3 hours ago [-]
> All the enterprisey stuff makes it feel a lot more complex than it really is.
Kubernetes is not PaaS. It is a low level container orchestrator. But people start by thinking of it as a ready-made PaaS and then keep on trying to make it closer to that vision by continually slapping random plugins on it.
In corporate environments, incompetence is the norm. Every kubernetes deployment in such places ends up relying on rando docker images, helm charts and "operators" written by script kiddies and webshits which break every night and are hardly better than coding the same functionality yourself.
You need a strong instinct to say "No" and taste to keep the system complexity under reasonable limits. Sometimes this involves actually architecting your "platform" code around the barebones orchestration provided by kubernetes instead of slapping on latest webshit shown on CNCF dot org. That sort of judgement and taste is impossible to find in corporate environments in the age of resume driven development. That is how most people get Kubernetes scars.
dewey 8 hours ago [-]
When you are only used to the Kubernetes at work I can understand how people dislike it. If you set it up yourself and start with minimal feature and not lots of annotations the config files become very simple and not more complicated than a docker compose file.
It quickly realized that after just using the managed Kubernetes from Digital Ocean and deploying a side project there.
PunchyHamster 3 hours ago [-]
Oh running stuff on k8s is great, but running k8s itself is just adding a lot of code to your stack and might be not so trivial to debug when something goes wrong.
Then again tools to deploy it went a long way
dewey 3 hours ago [-]
Agreed, personally I'd only do it through a hosted provider or maybe consider https://k3s.io for a bit simpler setup. I'd also only do it if Kubernetes is something I'm already familiar with.
vachina 5 hours ago [-]
> Talos + longhorn + fluxcd
Are you running a startup at home.
lachiflippi 4 hours ago [-]
That's, in my opinion, a pretty vanilla starter stack. Could probably argue that you don't need replicated storage, and that ArgoCD is nicer to look at, but Talos in particular simplifies a lot of the deployment because there's not much of an OS to configure or keep updated.
felooboolooomba 4 hours ago [-]
> Lot of kubernetes hate here, which is surprising
It's easier when you think of it using cats. A person wants a cat. They want a big one to deal with their rodent problem. They've heard that tigers are really big cats and extremely efficient in dealing with unwanted biomass around them.
Said person gets a tiger. They get an impromptu expedition to hell and barely make it back. Person hates cats, buys a poodle and advocates for mouse traps.
maxrecursion 3 hours ago [-]
That's a very accurate and elegant analogy.
Kubernetes was designed for large organizations with massive applications that are usually micro services. It allowed the applications to be load balanced and easily updated with no downtime. Also, these organizations paid top dollar to highly technical and competent employees.
Now, all organizations feel like they need kubernetes because it's the next logical step in the never ending IT progression. So, they force all their apps to be containerized and run on an multi-tenant cluster.
However, these apps worked just fine on a server, and now the devs have to refactor them and get them to work on kubernetes. All the System Admins and Engineers also need to learn kubernetes. But most of these people don't want to constantly learn and use kubernetes, so no one ever really knows how it all works.
This inevitably led to companies whose sole purpose is to help places get off of kubernetes because it is too hard to maintain. I've seen it play out at several organizations.
pjmlp 8 hours ago [-]
I got introduced to UNIX with Xenix, have used plenty of flavours including containers in HP-UX and Solaris before they became a thing in Linux, I have zero need to use Kubernetes at home.
In fact, I also have zero needs for it at work directly, as I have become an advocate for serverless and managed runtimes, unless there is really a business need to control the whole infrastructure, including the Kubernetes cluster directly.
gbalduzzi 6 hours ago [-]
Did you already know kubernetes, or did you have to learn it for this setup?
I believe this is the key difference, because it doesn't make sense to learn such a complicated tool for a simple use case.
If you already know it sure, it may make sense. But you are removing a huge initial effort required otherwise.
PunchyHamster 3 hours ago [-]
It's all nice till something breaks and you need to debug something, and when it does the less layers you have the easier it becomes
> All the enterprisey stuff makes it feel a lot more complex than it really is.
It is insanely complex under the hood. We just got to level when for most cases the tools available work just fine.
eqvinox 10 hours ago [-]
> Kubernetes is Too Hard. I built a system that I didn't actually know how to maintain without the time or energy necessary to dig myself out of trouble.
Couldn't agree more. Unless your homelab's point is to learn Kubernetes, just keep it simple. Proxmox sounds good, or just QEMU, libvirt, lxc, Docker, podman, whatever. Install packages, not containers where possible. Shell scripts are fine where needed. If it works for you, that's it, end of discussion, don't spend time on "pretty" if it's not the thing you want to get into / enjoy / learn.
(My "thing" is networking, I can assure you my homenet is beautiful. Couldn't give a rat's ass how & where my paperless is running tho. It runs. Done.)
preisschild 5 hours ago [-]
I enjoy using Kubernetes on bare metal nodes at home and took it over Proxmox, quite happy with it (not so happy with costs of hardware to expand the cluster though)
eqvinox 6 minutes ago [-]
If that works for you, great!
drusklo 2 hours ago [-]
The problem with using k8s on homelabs, is that a lot of the applications you would usually deploy, are not designed for it; having to manage a bunch of persistent volumes because most of your applications use sqlite is not very practical, and if the backend is sqlite, then you are probably running only one pod, so no real HA (if the pod goes down k8s will start a new one though), if you have to go through hoops to deploy an application that's not designed for it decreases its value.
Having said that, I keep a k3s node running for learning purposes, and all my homemade apps live in k3s; it is nice to have the option to escalate my app from 1 to 100 running instances, in case I want to test something, with the press of a button.
Unless you’re running multiple home servers, I can not reason any reason to use k8s. I’ll push any business to k8s but I would never bother with 1-3 home labs. Surprised this is even a hacker news topic of interest.
SMFloris 10 hours ago [-]
IMO, kubernetes is overkill for a small non-homogeneous home cluster.
What I use and really recommend is using systemd +/- docker. It just becomes so darn simple. Do not go the compose route (that route is filled with sadness of the incomplete stacks because db container failed silently kind) - instead aim to decompose the compose files and write a separate systemd service file for each of them, you can then assign limits separately.
I don't want to set anyone on the path ... but I use NixOs and this is so easy to do there.
maccard 5 hours ago [-]
I disagree - I think k8s is absolutely the right choice at that level. We use ECS instead of k8s for small projects because the control plane is more than the actual service, but when you want container management, some sort of basic deployment management, service discovery, storage management and secret management, k8s is super simple to run and works pretty much out of the box. For a homelab, I'd expect you to want this. Building it all yourself may be fun (and that's a good reason to do it), but it's going to result in all the complexity with a lot of issues that k8s just solves for you.
jbarberu 9 hours ago [-]
As someone who one week ago switched from a Debian to NixOS setup, using docker compose, I'd be very interested in hearing more if you have any resources or tips to share.
I was hoping to move over to running rootless containers, but so far my HA setup has proven to be a pita to get working.
xienze 6 hours ago [-]
> What I use and really recommend is using systemd +/- docker
Even better, systemd+podman (=quadlets).
Quadlets are great, you basically can declare a compose file as a systemd unit with all the good and bad that comes with that.
astralkraft 3 hours ago [-]
NixOS, IncusOS, FCOS Atomic. Completely agree. Immutable host. This is the way
rtpg 4 hours ago [-]
I've recently reached for pyinfra of all things and found it straightforward enough... just an epsilon above a pile of bash scripts.
ocd 10 hours ago [-]
I still don't know really what Kubernetes is for or why so many people outside specific environments are using it, but it's cool that you're using Ruby.
s_ting765 6 hours ago [-]
Kubernetes is good for two things. Zero downtime deployments and self-healing (where the looping state mechanism comes in). There are people who want k8s to handle every single operation that can run on a server, do not listen to those kinds of people they will lead you astray.
catdog 5 hours ago [-]
It's the industry standard for running and shipping containerized applications, at least for the ones which are more complex and/or dynamic than an application container and maybe 1-2 databases. It provides a well defined and stable orchestration API which is also quite capable. It's even sometimes worthwhile to spin up a single node cluster for that reason. Does not mean it is perfect, is it sometimes a bit complex? yes, is it verbose? yes, could the established ecosystem be better (thinking about e.g. helm)? yes. But at some point it's still easier with k8s than without it.
It is of course not the best choice for everything, there are enough things where you better just spin up a VM and deploy directly or with only a simple container runtime if you use containers.
anymouse123456 4 hours ago [-]
Kubernetes is an attempt to solve the problems that come from deploying, Microservices™ across a gigantic enterprise with hundreds (or thousands) of contributors.
If you are one person and just need to expose a handful of macro-services, Kubernetes is almost entirely made of waste in terms of complexity, abstraction, indirection and resource consumption.
LaurensBER 10 hours ago [-]
Kubernetes makes complex things (e.g blue/green deployment, auto-scaling, failover) possible irrespective of the underlying cloud/hardware with a good and standardized API.
It's absolutely overkill for small teams and homelabs (I run a cluster myself) but an absolute godsend if you do need the advanced functionality.
pjmlp 8 hours ago [-]
Somehow we were doing that with deployment scripts and VM management tooling before Kubernetes became a thing, and without having to deal with YAML spaghetti.
chuckadams 8 hours ago [-]
Yes, and Kubernetes came around as another player in that ecosystem and became popular for a reason, largely so we didn't have to manage clusters with imperative non-idempotent scripts with no runtime introspection or self-healing. I've done light devops (lab scale, not enterprise) off and on since cfengine was a thing, and while I'm no fan of the explosion of YAML (there's a special place in hell for helm in particular for using text/template to generate yaml), I'll take the controller loop design any day over most of the alternatives. Just having a sane API alone is a godsend: you ever try scripting vSphere?
pjmlp 8 hours ago [-]
What sane API, for what Kubernetes runtime, in what version?
Because that is the thing, not only it is a YAML spaghetti, everything changes depending on the puzzle pieces.
I had to follow CNCF related podcasts only to be aware of what cool projects were changing Kubernetes all the time.
Thankfully nowadays I only care about managed containers, regardless how the hyperscalers do it, it is no longer my headache.
Never needed to deal with vSphere directly.
chuckadams 7 hours ago [-]
Kubernetes is built around a JSON API: those yaml files are just an encoding of the objects that it manipulates with fairly vanilla REST commands. You could actually use JSON in your manifests instead, it's a proper subset and all, though it's not going to help the verbosity. For that you need a better abstraction like Pulumi, cdk8s, Yoke, or even just good old Terraform/OpenTofu. Or just write your own, there's an openapi spec and clients in every language out there.
I'm hardly trying to pretend like I'm a hyperscaler: I usually run k3s on a single node, and most of the time I admin it "clickops" style with k9s, something I find much easier than most other management tools.
catdog 4 hours ago [-]
> there's a special place in hell for helm in particular for using text/template to generate yaml
Indeed. Using text templating for structured documents is already quite bad, doing it with a significant whitespace language it's absolute hell.
LaurensBER 7 hours ago [-]
Yes but Kubernetes takes these battle-tested scripts, and allow everyone to use them with a few lines of YAML ;)
I understand the dislike of YAML but a Kubernetes deployment is ~50 lines, if I had to build my own scripts with a similar feature set I don't think I would be able to get it down much more than that.
catdog 4 hours ago [-]
Also your script often might not be that battle tested, more likely it will have bugs and miss a few important edge cases.
randompartytime 8 hours ago [-]
>Kubernetes makes complex things (e.g blue/green deployment, auto-scaling, failover) possible
How?
9 hours ago [-]
Havoc 8 hours ago [-]
Meanwhile I’m busy moving the other direction. More K8S.
Main motivation is that I’ve got a lot of compute and memory but it’s spread across many smaller devices. Meaningfully leveraging that requires a way to coordinate…
I do also have a classic Proxmox setup too though so can decide whether something should live in VM/LXC or k8s
pelasaco 6 hours ago [-]
I did the same move as you away from k8s to plain proxmox containers and VMs. Professionally i do work with k8s, and see the benefits of it (not always, but i see the use cases), but in my homelab it was consuming a lot of energy. Just dropping the whole k8s, made me save 1 kw energy when idle... I guess mainly because of the active API, shifting workload from different workloads and the whole machinery that happens behind the scenes..
astralkraft 5 hours ago [-]
Proxmox is merely Debian with some fancy curtains. I'm sort of fascinated by this social belief it is something more than that. Although I agree with your sentiment, I'd rather be on XCP-ng or IncusOS as the foundation for containers. I don't believe sacrificing immutability in the host is an improvement over k8s for stability and maintenance.
catdog 4 hours ago [-]
> Proxmox is merely Debian with some fancy curtains.
It's part of the appeal that it is mostly just Debian under the hood. If I want to run containers I would also not see it as the best choice but it really shines in managing qemu VMs.
astralkraft 4 hours ago [-]
Proxmox upgrades make K8s appealing, and that's not easily done.
I've been on Incus with my homelab for a few months now, and couldn't be happier. Maybe I should make a post about it, but I don't have much to report. My biggest maintenance outage was forgetting to set WakeOnPower in the BIOS.
Edit: And yes, all I want to do is run containers. Nothing fancy, Nginx Proxy Manager suffices for networking, and Linstor/DRBD handles disk. I wanted to remove hyperscaling complexity, but keep immutability, and so far, life is good.
catdog 5 hours ago [-]
On low power, mostly idle systems that's definitely a downside. K8s is always busy, probably largely because it's whole architecture is centered around reconciliation loops which contentiously compare the actual state to the desired state. It's not much but enough to prevent the CPU from entering its lowest power states.
trallnag 5 hours ago [-]
I would have assumed that observability components (Prometheus, etc) dwarf the control plain when it comes to activity. At least with not extremely dynamic workloads. And you mention that it was due to shifting workloads. If the demands are static, nothing should shift in my experience?
wsdn 7 hours ago [-]
[dead]
pjmlp 8 hours ago [-]
Most of the stuff hyperscalers use have no place in homelabs, unless you're training for a job application.
IMO the best change that I've made has been to give deterministic IPv6 addresses to every container and then using those for ingress.
I'm curious to hear where y'all think the line is between docker compose with Ruby glue and "Dear friend, you have built a Kubernetes".
Before that though, I had a single computer (a NUC if memory serves) with systemd running docker containers. Dead simple.
Talos + longhorn + fluxcd (optional), is super nice. And everything beyond that is additive and just works within the ecosystem.
If anything, it helped keep my stuff alive during all the hardware issues a lot longer.
I think like 5-6 years ago, kubernetes on baremetal was pretty painful. People should really give it another try, an LLM can probably set it up for you and fire off the docker compose to manifests in one shot. Or just follow the docs yourself, maybe a dozen commands to get a cluster running?
All the enterprisey stuff makes it feel a lot more complex than it really is.
That would not match their reason for preferring to move off k8:
> I built a system that I didn't actually know how to maintain without the time or energy necessary to dig myself out of trouble.
If you want a system that you can understand all at once in your own head then making it with your own head is a better way to go IMO, for a small system at least.
Kubernetes is not PaaS. It is a low level container orchestrator. But people start by thinking of it as a ready-made PaaS and then keep on trying to make it closer to that vision by continually slapping random plugins on it.
In corporate environments, incompetence is the norm. Every kubernetes deployment in such places ends up relying on rando docker images, helm charts and "operators" written by script kiddies and webshits which break every night and are hardly better than coding the same functionality yourself.
You need a strong instinct to say "No" and taste to keep the system complexity under reasonable limits. Sometimes this involves actually architecting your "platform" code around the barebones orchestration provided by kubernetes instead of slapping on latest webshit shown on CNCF dot org. That sort of judgement and taste is impossible to find in corporate environments in the age of resume driven development. That is how most people get Kubernetes scars.
It quickly realized that after just using the managed Kubernetes from Digital Ocean and deploying a side project there.
Then again tools to deploy it went a long way
Are you running a startup at home.
It's easier when you think of it using cats. A person wants a cat. They want a big one to deal with their rodent problem. They've heard that tigers are really big cats and extremely efficient in dealing with unwanted biomass around them.
Said person gets a tiger. They get an impromptu expedition to hell and barely make it back. Person hates cats, buys a poodle and advocates for mouse traps.
Kubernetes was designed for large organizations with massive applications that are usually micro services. It allowed the applications to be load balanced and easily updated with no downtime. Also, these organizations paid top dollar to highly technical and competent employees.
Now, all organizations feel like they need kubernetes because it's the next logical step in the never ending IT progression. So, they force all their apps to be containerized and run on an multi-tenant cluster.
However, these apps worked just fine on a server, and now the devs have to refactor them and get them to work on kubernetes. All the System Admins and Engineers also need to learn kubernetes. But most of these people don't want to constantly learn and use kubernetes, so no one ever really knows how it all works.
This inevitably led to companies whose sole purpose is to help places get off of kubernetes because it is too hard to maintain. I've seen it play out at several organizations.
In fact, I also have zero needs for it at work directly, as I have become an advocate for serverless and managed runtimes, unless there is really a business need to control the whole infrastructure, including the Kubernetes cluster directly.
I believe this is the key difference, because it doesn't make sense to learn such a complicated tool for a simple use case.
If you already know it sure, it may make sense. But you are removing a huge initial effort required otherwise.
> All the enterprisey stuff makes it feel a lot more complex than it really is.
It is insanely complex under the hood. We just got to level when for most cases the tools available work just fine.
Couldn't agree more. Unless your homelab's point is to learn Kubernetes, just keep it simple. Proxmox sounds good, or just QEMU, libvirt, lxc, Docker, podman, whatever. Install packages, not containers where possible. Shell scripts are fine where needed. If it works for you, that's it, end of discussion, don't spend time on "pretty" if it's not the thing you want to get into / enjoy / learn.
(My "thing" is networking, I can assure you my homenet is beautiful. Couldn't give a rat's ass how & where my paperless is running tho. It runs. Done.)
Having said that, I keep a k3s node running for learning purposes, and all my homemade apps live in k3s; it is nice to have the option to escalate my app from 1 to 100 running instances, in case I want to test something, with the press of a button.
What I use and really recommend is using systemd +/- docker. It just becomes so darn simple. Do not go the compose route (that route is filled with sadness of the incomplete stacks because db container failed silently kind) - instead aim to decompose the compose files and write a separate systemd service file for each of them, you can then assign limits separately.
I don't want to set anyone on the path ... but I use NixOs and this is so easy to do there.
I was hoping to move over to running rootless containers, but so far my HA setup has proven to be a pita to get working.
Even better, systemd+podman (=quadlets).
Quadlets are great, you basically can declare a compose file as a systemd unit with all the good and bad that comes with that.
It is of course not the best choice for everything, there are enough things where you better just spin up a VM and deploy directly or with only a simple container runtime if you use containers.
If you are one person and just need to expose a handful of macro-services, Kubernetes is almost entirely made of waste in terms of complexity, abstraction, indirection and resource consumption.
It's absolutely overkill for small teams and homelabs (I run a cluster myself) but an absolute godsend if you do need the advanced functionality.
Because that is the thing, not only it is a YAML spaghetti, everything changes depending on the puzzle pieces.
I had to follow CNCF related podcasts only to be aware of what cool projects were changing Kubernetes all the time.
Thankfully nowadays I only care about managed containers, regardless how the hyperscalers do it, it is no longer my headache.
Never needed to deal with vSphere directly.
I'm hardly trying to pretend like I'm a hyperscaler: I usually run k3s on a single node, and most of the time I admin it "clickops" style with k9s, something I find much easier than most other management tools.
Indeed. Using text templating for structured documents is already quite bad, doing it with a significant whitespace language it's absolute hell.
I understand the dislike of YAML but a Kubernetes deployment is ~50 lines, if I had to build my own scripts with a similar feature set I don't think I would be able to get it down much more than that.
How?
Main motivation is that I’ve got a lot of compute and memory but it’s spread across many smaller devices. Meaningfully leveraging that requires a way to coordinate…
I do also have a classic Proxmox setup too though so can decide whether something should live in VM/LXC or k8s
It's part of the appeal that it is mostly just Debian under the hood. If I want to run containers I would also not see it as the best choice but it really shines in managing qemu VMs.
I've been on Incus with my homelab for a few months now, and couldn't be happier. Maybe I should make a post about it, but I don't have much to report. My biggest maintenance outage was forgetting to set WakeOnPower in the BIOS.
Edit: And yes, all I want to do is run containers. Nothing fancy, Nginx Proxy Manager suffices for networking, and Linstor/DRBD handles disk. I wanted to remove hyperscaling complexity, but keep immutability, and so far, life is good.
So duh.