Setting up your own K3S home cluster

How I started

I’ve selected 3 HP ProDesk 600 G2 computers as the nodes. One of the nodes is the control plane node, the other two are worker nodes.

I’ve selected mini computers instead of Raspberry PI’s because of the processor architecture (x86 instead of ARM) and the possibility to expand resources. I was inspired by the Tiny Mini Micro project.

Initially, I installed Ubuntu 20.04 LTS as operating system and Kubernetes 1.21 on it (using Kubeadm). The purpose was to prepare for my CKA exam (which I’ve passed).

After passing the exam, I wanted a different setup, existing of the following components:

• K3S as Kubernetes distribution
• MetalLB as load balancer
• Rancher as cluster management
• Traefik as ingress controller
• Cert-Manager as certificate manager

During the process, I found a lot of articles, but a lot of them weren’t very clear (especially for K3S). I hope to provide a clear guide for other engineers with the same challenge.

K3S installation

To install k3s I chose to use shell scripting, because I don’t need a lot of options to install (only exclude Traefik installation).
As mentioned earlier, I have a 3 node clusters with one control plane and 2 workers. During the following steps, the node token needs to be retrieved from the control plane node, in order to join the worker nodes.

The used scripts are shown below.

Install k3s server

SSH into the control plane server and execute the script beneath.

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC=”server --disable traefik 
--disable servicelb” sh -

Retrieve token

Retrieve the node token from the control plane, which is needed further in the process to join the worker nodes.

cat /var/lib/rancher/k3s/server/node-token

Retrieve certificate

Retrieve the certificate in order to communicate with the cluster.
Save this on your local machine in ~/.kube/<your config file>.

cat /etc/rancher/k3s/k3s.yaml

Add worker node

SSH into your worker node(s) and apply the script below. This will add the node as worker node to the cluster.

curl -sfL https://get.k3s.io | K3S_URL=https://<Contol Plane IP>:6443 
K3S_TOKEN=<YourToken> sh -

Load certificate

Once the steps before are applied, you can execute the following script to load the Kubernetes config.

export KUBECONFIG=/.kube/<your config file>

Now you should be able to use the cluster.

Check this by executing the following command.

kubectl get nodes

The result is a list of the available nodes.

MetalLB installation

Because I wanted to access all applications, running on the cluster, via one endpoint, I’ve added MetalLB to the cluster. To add it I’ve used the default manifests for version 0.11. To do this, I’ve used the default install scripts. To configure, I’ve used the layer 2 configuration scripts. Select a range of IP-addresses which isn’t already in use.

Rancher installation

The rancher installation was straight forward. I’ve used Helm templates to install Rancher. If you would like to use the options in the helm values file, please have a look at this website.

Helm install

Add repository

helm repo add rancher-latest https://releases.rancher.com/
server- charts/latest

Install helm chart

helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=<Your Domain Name>
  --set bootstrapPassword=<Your password>

When done, you should see that all deployments in the cattle-system namespace (which is the Rancher default namespace) are running. After that you can access the dashboard, with the earlier provided hostname. If you didn’t the provide a hostname, you can access it by setting up a port forward using the following command.

kubectl port-forward svc/rancher 8443:443 -n cattle-system

Traefik installation

The Traefik installation was also pretty straight forward. I’ve used the helm template with an additional values file. These values are necessary to access the dashboard.

Helm install

Add repository

helm repo add traefik https://helm.traefik.io/traefik

Create the values file with the following content.

dashboard:
 enabled: true
 domain: traefik.<Your Domain Name>
rbac:
 enabled: true

Install the helm chart, in the traefik namespace, with the configuration from the values file. If the namespace doesn’t exist, create it.

helm install traefik traefik/traefik -n traefik -f values.yaml

When done, you should see that all deployments in the traefik namespace (which I’ve created as the Traefik namespace) are running. After that you can access the dashboard, with the earlier provided hostname. If you didn’t the provide a hostname, you can access it by setting up a port forward using the following command.

kubectl port-forward deploy/traefik 9000 -n traefik

Cert-Manager

Because I want automatic certificate renewal I needed a certificate manager.Cert-Manager is the default certificate manager for Kubernetes. I’ve installed Cert-Manager using helm templates. To install it I’ve used the commands you’ll find below.

Install Cert-Manager

Add helm repository

helm repo add jetstack https://charts.jetstack.io

Install cert manager with the following script.

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.5.1

When Cert-Manager is installed, a ClusterIssuer or Issuer is needed to help with the provisioning of certificates. To do this with self signed certificates, a simple yaml file needs to be applied (which is shown below).

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}

If you wish to use CA signed certificates, the ClusterIssuer or Issuer should be configured differently.

apiVersion: v1
kind: Namespace
metadata:
  name: test

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: test
spec:
  replicas: 1
  selector:
    matchLabels:
      # manage pods with the label app: nginx
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  namespace: test
spec:
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: https
  selector:
    app: nginx

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: test
  annotations:
    kubernetes.io/ingress.class: traefik
    cert-manager.io/cluster-issuer: selfsigned
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  rules:
  - host: test.<Your Domain Name>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-svc
            port:
              number: 80
  tls:
  - hosts:
    - test.<Your Domain Name>
    secretName: test.<Your Domain Name>

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test.<Your Domain Name>
  namespace: test
spec:
  commonName: test.<Your Domain Name>
  secretName: test.<Your Domain Name>
  dnsNames:
    - test.<Your Domain Name>
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer

10.0.0.1 test.<Your Domain Name>