Kubernetes at Home
I recently converted my Docker Swarm-based homelab to one powered by Kubernetes.
My original goals with setting up Swarm were to fully leverage all the compute and memory available on my two-node setup. Ultimately though, because of shared storage and other complications with hardware devices, I ended up manually scheduling containers onto specific nodes, which totally defeated the purpose.
Plus Kubernetes is cooler 😎.
The process took about a week of a few hours at a time to get set up. I wanted to write about it here because otherwise I’ll forget everything.
Start with the end in mind
At the end of this adventure, I ended up with the following setup:
- 2-node k3s cluster
- Fast high availability replicated storage using Longhorn
- NFS for large file storage
- Optional OAuth for cluster ingresses (put web UIs behind Google Login)
- Optional automatic TLS for ingresses using Letsencrypt (via
If any of that is of interest, read on…
There are 2 nodes:
🐳 whale and
🦆 duck (don’t ask). They are not
- Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz
- 16GB DDR4
- 120GB SSD
- 8TB ZFS array (more on that later)
- 1Gbe Networking
- Intel(R) Core(TM) i5-7600 CPU @ 3.50GHz
- 32GB DDR4
- 120GB SSD
- 1Gbe Networking
Both are running Ubuntu 20.04, and both have fixed IP addresses.
I used Ansible for this part. I tried to keep it simple, and the basics of the
configuration I used were to install
open-iscsi (required for Longhorn) and
nfs-common (required for NFS mounts) and then:
- name: Enable IPv4 forwarding sysctl: name: net.ipv4.ip_forward value: "1" state: present reload: yes - name: Enable IPv6 forwarding sysctl: name: net.ipv6.conf.all.forwarding value: "1" state: present reload: yes # adapted from https://germaniumhq.com/2019/02/14/2019-02-14-Disabling-Swap-for-Kubernetes-in-an-Ansible-Playbook/ - name: Disable SWAP in fstab since kubernetes can't work with swap enabled (1/2) replace: path: /etc/fstab regexp: '^([^#].*?\sswap\s+sw\s+.*)$' replace: '# \1' register: swap_was_disabled - name: Disable SWAP since kubernetes can't work with swap enabled (2/2) shell: | swapoff -a when: swap_was_disabled.changed
I decided to use k3s in this setup, because it’s simpler for this type of environment (small, 2 node cluster) and it just works.
Installing the k3s server (master)
Our master node is going to be
🐳 whale, so we have to run the following on
sudo curl -sfL https://get.k3s.io | sh -
This will set up and enable the
containerd daemon and install the rest of the
k3s control plane. Since docker also uses
containerd, my existing running
containers are not affected at all!
Once it’s done, you can test it with:
$ sudo k3s kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 5m30s
If that works, we can proceed to adding a worker node. To do that we need the
node-token for the master node. The
k3s server helpfully places the token in
a file on the server:
Copy the contents of that file somewhere (I just
catted it and then
Installing the k3s agent (worker)
Now, to make a real cluster, we need to get the worker node (
To install the k3s agent, run this on the worker node (replace
the token from the previous step, and
<MASTER IP> with the IP of the master
sudo curl -sfL https://get.k3s.io \ | K3S_TOKEN=<TOKEN> K3S_URL=https://<MASTER IP>:6443 sh -
Once that’s finished, it’s time to make sure we can run workloads on both nodes.
Testing the setup
Deployment that will run pods on both nodes:
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-test labels: app: nginx-test spec: replicas: 4 selector: matchLabels: app: nginx-test template: metadata: labels: app: nginx-test spec: containers: - name: nginx-test image: nginx
Put that in a file called
nginx.yaml on your server node, and run
sudo k3s kubectl apply -f nginx.yaml.
Then check the pods:
$ sudo k3s kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-test-7795b97f48-t48g5 1/1 Running 0 5s 10.42.1.106 duck <none> <none> nginx-test-7795b97f48-rjkwb 1/1 Running 0 6s 10.42.1.107 whale <none> <none> nginx-test-7795b97f48-ln5kp 1/1 Running 0 6s 10.42.1.108 whale <none> <none> nginx-test-7795b97f48-6x56q 1/1 Running 0 6s 10.42.1.109 duck <none> <none>
Note: If you’re lucky you’ll end up with pods spread over your nodes, but
they might all land on one. You can keep messing with the
replicas: 4 value
apply-ing the manifest, or look into
Pod Topology Spread Constraints.
Don’t forget to clean up:
sudo k3s kubectl delete -f nginx.yaml or
sudo k3s kubectl delete deployment nginx-test.
Using kubectl from elsewhere
You don’t want to log into your servers to manage the cluster, so on the master
/etc/rancher/k3s/k3s.yaml onto your local machine as
~/.kube/config. Then replace
127.0.0.1 with the IP or name of your k3s
Obviously you need to install
kubectl for this to work.
Also PRO TIP: put this at the bottom of your
Then getting pods can be a little characters as
k get po. Additionally most
resources types have a short form;
nodes. If it saves a few keyboard strokes, it’s worth doing.
I mentioned I was running Docker Swarm on these nodes before. Luckily, since
containerd-shim) under the hood, it can coexist
on the same machine as k3s. Which is good for my cutover period, while I slowly
migrate each service from docker-compose YAMLs to Kubernetes YAMLs.
The downside however is that my Swarm-hosted Traefik 2.x setup was conflicting with the Traefik 1.7 setup that comes bundled with k3s.
I rashly decided to disable Traefik in k3s the wrong way (I just did
kubectl delete -n kube-system deploy traefik).
This works, Traefik stops doing things in the kube cluster, but then putting it
back is a huge pain. You can convince k3s to re-run the
helm chart that runs
on startup by using this eldritch nightmare (requires
# Save the job config as an edited JSON file kubectl get -n kube-system job "helm-install-traefik" -o json \ | jq 'del(.spec.selector)' \ | jq 'del(.spec.template.metadata.labels)' \ > helm-install-traefik.json # Force the job back into k8s again :'( cat helm-install-traefik.json | kubectl replace --force -f -
Gross. But it works 🤷♂️
Again, this wouldn’t be an issue if you were starting from scratch, just if you already have something listening on 80/443 that you don’t want to shut down right away.