microk8s on LXD

I tried to follow the documentation to run MicroK8s on LXD. However, it didn’t work because the microk8s nodes (really node since I was just trying on one system) wouldn’t start. I’m not sure of the root cause of that but it was vexing.

After a little bit of searching, I found an article on the Ubuntu blog that described running Apache Spark on MicroK8s on Ubuntu Core in the cloud. The use case is different, but the initial part of the configuration was to get MicroK8s running on LXD.

I adapted the instructions to map to my situation and got it working on my Ubuntu 20.04 desktop. First thing is to create a new profile for microk8s. I’m using ZFS on my system which is also what is used in the article, so that made it straightforward.

lxc profile create microk8s
cat > microk8s.profile <<EOF
config:
  boot.autostart: "true"
  linux.kernel_modules: ip_vs,ip_vs_rr,ip_vs_wrr,ip_vs_sh,ip_tables,ip6_tables,netlink_diag,nf_nat,overlay,br_netfilter
  raw.lxc: |
    lxc.apparmor.profile=unconfined
    lxc.mount.auto=proc:rw sys:rw cgroup:rw
    lxc.cgroup.devices.allow=a
    lxc.cap.drop=
  security.nesting: "true"
  security.privileged: "true"
  security.syscalls.intercept.bpf: "true"
  security.syscalls.intercept.bpf.devices: "true"
  security.syscalls.intercept.mknod: "true"
  security.syscalls.intercept.setxattr: "true"
description: ""
devices:
  aadisable:
    path: /sys/module/nf_conntrack/parameters/hashsize
    source: /sys/module/nf_conntrack/parameters/hashsize
    type: disk
  aadisable1:
    path: /sys/module/apparmor/parameters/enabled
    source: /dev/null
    type: disk
  aadisable2:
    path: /dev/zfs
    source: /dev/zfs
    type: disk
  aadisable3:
    path: /dev/kmsg
    source: /dev/kmsg
    type: disk
  aadisable4:
    path: /sys/fs/bpf
    source: /sys/fs/bpf
    type: disk
name: microk8s
used_by: []
EOF
cat microk8s.profile | lxc profile edit microk8s
rm microk8s.profile

Using that profile, I created a new container and installed microk8s:

lxc launch -p default -p microk8s ubuntu:20.04 microk8s
lxc exec microk8s -- sudo snap install microk8s --classic

There are a few additional configuration steps that need to take place on the container. Basically, this fixes a problem with apparmor and sets a couple of configuration variables.

lxc shell microk8s

cat > /etc/rc.local <<EOF
#!/bin/bash

apparmor_parser --replace /var/lib/snapd/apparmor/profiles/snap.microk8s.*
exit 0
EOF

chmod +x /etc/rc.local
systemctl restart rc-local

echo 'L /dev/kmsg - - - - /dev/null' > /etc/tmpfiles.d/kmsg.conf
echo '--conntrack-max-per-core=0' >> /var/snap/microk8s/current/args/kube-proxy
exit

After that, stop and restart the container. You can try restart, but I had it timeout.

lxc stop microk8s
lxc start microk8s

The last config was to turn the swap off in the container. I’m not sure if this is critical, but it makes sense to not swap in this circumstance.

lxc exec microk8s -- sudo swapoff -a

Now we are ready to test by deploying a small app to microk8s.

lxc exec microk8s -- sudo microk8s.kubectl create deployment microbot --image=dontrebootme/microbot:v1

To see the app outside the container, we expose the service.

lxc exec microk8s -- sudo microk8s.kubectl expose deployment microbot --type=NodePort --name=microbot-service --port=80

Finally, we can test it using curl.

MICROBOT_PORT=$(lxc exec microk8s -- sudo microk8s.kubectl get all | grep service/microbot | awk '{ print $5 }' | awk -F':' '{ print $2 }' | awk -F'/' '{ print $1 }')
MK8S_IP=$(lxc list microk8s | grep microk8s | awk -F'|' '{ print $4 }' | awk -F' ' '{ print $1 }')
curl http://$MK8S_IP:$MICROBOT_PORT/

If you get some results out of that, it all worked.