A Raspberry Pi cluster with Docker, Kubernetes, and Jenkins

A functional personal development pipeline and deployment infrastructure

This winter, I have decided to try building a Pi’s cluster to test a small but meaningful installation of Kubernetes and Jenkins pipeline. There are many tutorials online that helped me along the way. However, as any developer will attest, none of the tutorials are perfect fits for your setup. It took me a couple of days but at the end, I am quite happy with my setup. I hope this article will help you achieve your Pi cluster setup as well.

My setup consist of two Raspberry Pi 4 (4G version). One of them is the server node and the other is the worker node. The server node will serve as the Kubernetes master node. It will also run a private Docker Registry to publish my own images. These images will be built using Jenkins on the same server. The worker node will only run the Kubernetes worker node.

Common Setup (Server node and Worker node)

Base Installation

First of all, you need to download Raspbian Lite and write to your SD card. Before you boot up your Pi, follow the Setting up a Raspberry Pi headless instructions to setup WiFi and SSH. NOTE: Do NOT use a generated PSK. It didn’t work for me.

With WiFi and SSH setup, you can boot up your Pi with the prepared SD card. Once it’s up, you can ssh pi@raspberrypi.localwith password raspberry .

This works because avahi-daemon broadcasts the hostname using mDNS. If avahi-daemon isn’t installed, install it with apt. If you’re host machine is Windows, you may need to install Bonjour Print Service first.

If your Pi have multiple network interfaces (e.g. WiFi and Ethernet), it is better to enforce avahi-daemon to use one specific interface. This can be done by changing the allow-interfaces setting in /etc/avahi/avahi-daemon.conf.

Once you’re in, run raspi-config and setup the followings:

1. Expand file system under Advanced Options.
2. Change hostname under Network Options. I call my server node server and worker node agent . Their hostname in mDNS are server.local and agent.local respectively.
3. Set the GPU memory split to 16mb under Advanced Options.

Since we are going to use the Pi for Kubernetes, let’s enable container features in the kernel by editing /boot/cmdline.txt and adding the following to the end of the line:

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

With Kubernetes, it is important to have a static IP for all of your nodes. You can either do it inside your Pi or reserve the IP on your DHCP server (usually your router). I use the router approach.

The basic idea is to use the command ip address to discover the MAC Address of the Pi and use it to reserve IPs on my router. In my case, the server node is 192.168.0.12 and the worker node is 192.168.0.13 .

We are almost done with the base setup. But before we proceed, let’s upgrade our installed packages.

$ sudo apt update
$ sudo apt upgrade

Now, your initial Pi setup is done. Let’s reboot it!

Passwordless Login

Save yourself some trouble in the future and enable passwordless login. On your host machine, run ssh-copy-id pi@<server node>.local and enter the password raspberry (if you haven’t changed it yet). Do the same for your worker node.

Now you can ssh into your Pi from this host without entering a password. While you’re in your Pi, change your password too.

Docker

Both nodes need to have Docker installed. Use the following command to install Docker and add user pi into the docker group.

$ curl -sSL https://get.docker.com | sh
$ sudo usermod -aG docker pi

The docker script will install a service which alters the firewall rules in your system. Specifically, it will enable ip_forward and change your default FORWARD policy from ACCEPT to DROP . Please refer to the Docker and iptables article for more information.

The FORWARD policy change will cause issues with the coredns pod in the future. So, let’s modify it. Debian Buster uses nftables (instead of iptables) by default but the Buster Lite distribution does not come with it. So we need to install it with apt . While we are at it, let’s install dnsutils as well so that we can test our changes later.

$ sudo apt install nftables dnsutils

Next, we create a new file /lib/systemd/system/nftables-forward-accept.service with the following content:

[Unit]
Description=Change FORWARD policty to ACCEPT
After=docker.service
[Service]
Type=simple
ExecStart=/usr/sbin/nft insert rule ip filter DOCKER-USER counter accept
[Install]
WantedBy=multi-user.target

This service will be invoked after the docker.servie and it will add a new firewall rule. Once the file is in place, we need to enable it during bootup.

$ sudo systemctl enable nftables-forward-accept.service

Reboot and verify that the DOCKER-USER has the following rules by running sudo nft list table filter :

chain DOCKER-USER {
  counter packets 224 bytes 94572 accept
  counter packets 0 bytes 0 return
}

Once this is verified, you can proceed.

Docker & Kubernetes (Server Node)

The following instructions apply to the server node only. Worker node specific instructions will follow this section.

Docker Registry

We will be using Jenkins to build and publish images to our private Docker registry. Since we are in closed door, we don’t need TLS or authentication. Create/modify the file /etc/docker/daemon.json and add the following inside:

{
  "insecure-registries" : ["server.local:5000"]
}

Now, we are ready to spin up the registry:

$ docker run -d -p 5000:5000 --restart always --name registry registry:2

Kubernetes (k3s)

K3S is a lightweight Kubernetes installation. It is especially suitable for Raspberry Pi. Install it with:

$ export K3S_KUBECONFIG_MODE="644"
$ curl -sfL https://get.k3s.io | sh -

Verify DNS

Remember we changed some firewall rules settings before? Now it is the time to test that the coredns pod can resolve external IP addresses. Execute the following command

$ nslookup quay.io 10.43.0.10

(NOTE: 10.43.0.10 is the IP address of the coredns pod. You can use kubectl get svc -n kube-system to verify it.)

If the above command completes successfully, you are good to go.

Now, we are ready to install the worker node. Before we leave the server node, obtain its installation token first. We will need it when we install the worker node.

$ sudo cat /var/lib/rancher/k3s/server/node-token

Docker & Kubernetes (Worker Node)

Kubernetes (k3s)

We will need to install k3s on the worker node as well. The two special environment variables below will change the installation mode to worker mode.

$ export K3S_KUBECONFIG_MODE="644"
$ export K3S_URL="https://192.168.0.12:6443"
$ export K3S_TOKEN="XXXX” # the token you saved before.
$ curl -sfL https://get.k3s.io | sh -

Remember that my server node’s IP is 192.168.0.12

Docker Registry

Our docker registry is installed on the server node. However, we need to tell the agent where it is so that it can pull the image properly. Create a /etc/rancher/k3s/registries.yaml file with the following content:

mirrors:
  "server.local:5000":
    endpoint:
      - "http://192.168.0.12:5000"

What it means is that for the registry server.local:5000 , resolve it to http://192.168.0.12:5000 . We have to use the IP address here because K3S resolves name using the nameserver defined in /etc/resolv.conf instead of mDNS.

Now, you have the basic Docker and Kubernetes setup with your two Pi nodes. You should be able to push an image to your private Docker and create Kubernetes deployment using that image. The prefix of your image should be server.local:5000 (or whatever you decide to be in your registries.yaml)

Jenkins Installation (Server Node)

Once you have Docker and Kubernetes setup, you probably want to setup a Jenkins pipeline to build something meaningful. To install Jenkins, you first need a JDK. The following will install JDK 11 as of today:

$ sudo apt install default-jdk

Then, you can add Jenkins’s key to your package keys:

$ wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -

Once the key is installed, modify/create the file /etc/apt/sources.list.d/jenkins.list and add the following line:

deb https://pkg.jenkins.io/debian binary/

Finally, we can install Jenkins:

$ sudo apt update
$ sudo apt install jenkins

Once this is done, open http://server.local:8080 on your browser window and follow the onscreen instructions to continue with the installation.

It is also a good idea to change the Jenkins port to something less frequently use. You can do that by editing /etc/default/jenkins and modify the HTTP_PORT variable to the new port.

Finally, since Jenkins will be used to run docker, we need to add the jenkins user to the docker group.

$ sudo usermod -aG docker jenkins

Restart Jenkins to activate these changes:

$ sudo service jenkins restart

Summary

These are the steps you’d need to get a cluster and a functional development pipeline going at home. Obviously, some of these steps can be automated through Ansible. However, by showing the steps in details, I hope you will find it useful when you setup your own cluster and development pipeline.