Trying K3s on ARM, Part 1

Why k3s and not stick with k8s?

I wanted to experiment with k3s. They package everything you need in a single binary, don’t package in deprecated parts of k8s, and it works on Intel, ARMv7 and ARM64. It seemed like it’d be a less painful way to runn Kubernetes on my ARM cluster.

Prerequisites

You must have set up DNS entries for the nodes you want to cluster, or update /etc/hosts on all the nodes so they can find each other.

Installing k3s

I chose to install k3s without the built-in traefik install so I could install that with a custom configuration. I also chose to use docker instead of the baked-in containerd so that I could also run containers outside k3s on my worker nodes without wasting RAM.

Installing the master

curl -sfL https://get.k3s.io > install-k3s.sh && chmod 755 ./install-k3s.sh
sudo ./install-k3s.sh --no-deploy traefik --docker
sudo chgrp docker /etc/rancher/k3s/k3s.yaml
sudo chmod g+r /etc/rancher/k3s/k3s.yaml

I also updated /etc/systemd/system/k3s.service to add

After=network-online.target cluster-mfsmount.service docker.service

because I don’t want k3s to attempt to start until after the docker service has started and the cluster’s moosefs distributed filesystem is mounted.

Once all that is done, copy the node token from /var/lib/rancher/k3s/server/node-token to each of the worker nodes.

Installing the workers

Copy /var/lib/rancher/k3s/server/node-token from the server to your worker.

Run

./install-k3s.sh --agent --server https://master-server:6443 --kubelet-arg="address=0.0.0.0" --token "$(cat node-token)" --docker

Remove the --docker if you want to use the containerd bundled into k3s - I wanted to be able to also run apps in docker on my nodes and didn’t want it using extra RAM for another containerd.

If you’re using a distributed filesystem like I am, add

After=network-online.target cluster-mfsmount.service docker.service

to /etc/systemd/system/k3s-agent.service, and

After=network-online.target cluster-mfsmount.service

to /lib/systemd/system/docker.service to keep docker from starting until after the distributed filesystem is mounted.

Set up Networking

MetallB

I wanted to be able to use LoadBalancerIP entries in my cluster services to make using Traefik easier.

Installing MetallB

On my master node, I ran

kubectl apply -f https://raw.githubusercontent.com/danderson/metallb/master/manifests/metallb.yaml

Configuring MetallB

I used the following configuration for metallb (in metallb-conf.yaml)

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.0.1.16/28

And applied it with kubectl apply -f metallb-conf.yaml.

This allows me to use 10.0.1.17 through 10.0.1.30 as LoadBalancerIP entries in my k8s service configurations. 14 entries should be more than enough for my immediate needs.

You will want to change the addresses entry to conform to your own network.

Traefik

Installed traefik with my own configuration, which I have posted on github:

Here are the configuration files I used - you’ll need to tweak them for your own network.

traefik-rbac.yaml

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system

traefik-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-conf
  namespace: kube-system
data:
  traefik.toml: |
    defaultEntryPoints = ["http","https"]
    debug = false
    logLevel = "INFO"
    
    # Do not verify backend certificates (use https backends)
    InsecureSkipVerify = true

    [entryPoints]
      [entryPoints.http]
      address = ":80"
      compress = true
      [entryPoints.https]
      address = ":443"
        [entryPoints.https.tls]

    #Config to redirect http to https
    #[entryPoints]
    #  [entryPoints.http]
    #  address = ":80"
    #  compress = true
    #    [entryPoints.http.redirect]
    #    entryPoint = "https"
    #  [entryPoints.https]
    #  address = ":443"
    #    [entryPoints.https.tls]

    [web]
      address = ":8080"

    [kubernetes]

    [metrics]
      [metrics.prometheus]
      buckets=[0.1,0.3,1.2,5.0]
      entryPoint = "traefik"
    [ping]
    entryPoint = "http"

traefik-deployment.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 2
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik:1.7.9
        name: traefik-ingress-lb
        volumeMounts:
        - mountPath: /config
          name: config
        ports:
         - name: http
           containerPort: 80
         - name: https
           containerPort: 443
         - name: admin
           containerPort: 8080
        args:
        - --api
        - --kubernetes
        - --configfile=/config/traefik.toml
        livenessProbe:
          httpGet:
            path: /ping
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 3
          timeoutSeconds: 1
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: role
                  operator: In
                  values:
                  - data
              topologyKey: kubernetes.io/hostname
      volumes:
      - name: config
        configMap:
          name: traefik-conf

traefik-service.yaml

---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  selector:
    k8s-app: traefik-ingress-lb
  externalTrafficPolicy: Local
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 443
      name: https
    - protocol: TCP
      port: 8080
      name: admin
  type: LoadBalancer
  loadBalancerIP: 10.0.1.20
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-ingress-lb
  namespace: kube-system
spec:
  rules:
  - host: traefik.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-ingress-service
          servicePort: admin

You’ll want to change the loadBalancerIP entry and the host entry in the spec section to match your network and DNS configurations.

for traefik_yaml in traefik-rbac.yaml traefik-configmap.yaml traefik-deployment.yaml traefik-service.yaml
do
  kubectl apply -f $traefik_yaml
done

Updates

Updated URL for metallb install yaml file to use latest instead of pinning a specific version. k3s is actively being updated and the old version no longer worked.

Why k3s and not stick with k8s?#

Prerequisites#

Installing k3s#

Installing the master#

Installing the workers#

Set up Networking#

MetallB#

Installing MetallB#

Configuring MetallB#

Traefik#

traefik-rbac.yaml#

traefik-configmap.yaml#

traefik-deployment.yaml#

traefik-service.yaml#

Updates#