Constructing a Kubernetes Cluster spanning a Public Cloud VM and a Local VM: Part II

Kubernetes Cluster (Source: Ashish Patel, Medium)

This is the second installment of a three part series on how to set up a Kubernetes cluster spanning a cloud VM and a local VM:

• Part I : Describes how to set up a VM on a cloud provider (I used Azure) and a local VM in VirtualBox , how to configure them for Kubernetes, and how to connect them together using a Wireguard VPN.
• Part II (here): Describes how to set up the Kubernetes cluster spanning the cloud VM and local VM withkubeadm, and how to ensure that the control and data plane run over the encrypted VPN connection.
• Part III: Describes how to set up an Nginx reverse proxy and dnsmasq on the cloud VM, and configure them so that a web application in the Kubernetes cluster can be served to the Internet through the cloud VM’s Internet-accessable DNS host name.

In this installment, we’ll use kubeadm to install a Kubernetes node on central-node as the cluster control node and gateway-node as a worker node. Instructions are included for removing taints on central-node to allow workload placement, so we can deploy applications there. Flannel is used for the intra-cluster, inter-pod network, though we need to make some changes in the default yaml manifest to enable the data plane to run over the Wireguard VPN.

The instructions below are a condensation of the kubeadminstallation
instructions here and cluster configuration instructions here.

Installing the containerd container runtime package and configuring a system service

Kubernetes no longer uses Docker as the default container runtime and, as a practical matter, installation and configuration of Docker for Kubernetes is now difficult. Instead, the default container runtime is containerdwhich is another CNCF project. Docker is still a great environment for container development however.

Instructions for installing the latest version of containerd and configuring can be found here, but they are slightly different for Ubuntu 20.04, so we use the easier path of installing the apt package:

sudo apt install containerd 

This will install the package, configure a systemd service with properly formatted unit file, and start the service, which installs a container runtime socket in the right place. This has the added advantage of installing the default systemd cgroup driver.

Establishing Kubernetes apt repo access on the nodes

Check whether you already have access to the Kubernetes repository with:

sudo apt policy | grep kubernetes 

If you see something like:

500 https://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
 release o=kubernetes-xenial,a=kubernetes-xenial,n=kubernetes-xenial,l=kubernetes-xenial,c=main,b=amd64
 origin apt.kubernetes.io 

Skip forward to the next section. If not, do the following.

Update the apt package index and install packages needed to use the Kubernetes apt repository:

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl 

Download the Google Cloud public signing key:

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg 

Add the Kubernetes apt repository:

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list 

Installing the Kubernetes system utility packages and kubeadm

To install the Kubernetes utilities and kubeadm type the following commands:

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl 

This installs kubeadm, kubectl and kubelet, and starts kubelet as a systemd service. It also marks the Kubernetes utilites so that they are not automatically updated. The kubelet on the gateway-node will need some additional configuration but it needs to be done after the gateway-node has joined the cluster.

Note that if the installation suggests you install any firewall packages, don’t install them since local firewalls will considerably complicate the cluster connectivity as discussed in Part I. If your host is connected directly to the Internet and you followed the directions in Part I on firewalls, you should already have a firewall installed, running, and configured.

Installing central-node as a control node

We first need to run kubeadm on central-nodewith arguments specific to configuring central-node as a control node. The following arguments need to be set:

• Since we are using Flannel, we need to reserve the pod CIDR using --pod-network-cidr=10.244.0.0/16. This reserves IP address space for the intra-cluster, inter-pod network.
• We need to advertise the Kubernetes API server on the wg0 interface so all traffic between the central node and the gateway nodes is encrypted. Use --apiserver-advertise-address=10.8.0.1.
• The containerd socket should be specified using the argument --cri-socket=unix:///var/run/containerd/containerd.sock in case there are any other container runtime sockets lying about.

Note that if you think you may later want to turn the cluster into an HA cluster with multiple control plane nodes and a cloud provider load balancer, you should also include the --control-plane-endpoint parameter as described here since it is not possible to configure a cluster HA unless the control plane IP address is specified when the cluster is created.

Run kubeadm init on the central-node:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.8.0.1 --cri-socket=unix:///var/run/containerd/containerd.sock 

When kubeadm is finished, it will print out:

Your Kubernetes control-plane has initialized successfully!
You should now deploy a Pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node as root:
kubeadm join <control-plane-host>:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

Note that <control-plane-host> will be the IP address of the primary interface on your cloud VM, an address in a private IP address space on the virtual private cloud network which the cloud provider has assigned to your VM. It will not be the wg0 interface IP address. Even though the cloud VM’s IP network and your site local network connected to the local VM are in completely separate routing domains, they have a route between them through the wg0 interface.

Since we will usekubeadm join shortly to join the gateway-node to the central-node control plane, create a shell script file on gateway-node called join.sh with the first line #! /bin/bash and copy the kubeadm join command into it. Add --cri-socket=unix:///var/run/containerd/containerd.sock at the end of the second line, save and exit the editor. Make the file executable with chmod a+x join.sh so it can run as a shell script.

To start using your cluster as a nonroot user, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config 

This creates a subdirectory in your home directory called .kube and copies the cluster configuration file into it.

Removing the taints on the control node prohibiting application workload deployment.

As installed out of the box, kubeadm places taints on the control node disallowing deployment of application workloads. If you want to deploy applications to central-node, you need to remove the taints:

kubectl taint node central-node node-role.kubernetes.io/master- 

which should print out:

node/central-node untainted 

and also run the command:

kubectl taint node central-node node-role.kubernetes.io/control-plane- 

You can check if the node is untainted with:

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints --no-headers 

which should show <none> on central-node.

Installing the Flannel CNI plugin on the central node

Prior to installing Flannel, we need to modify the yaml manifest to utilize the Wireguard VPN interface for data plane traffic instead of the default interface on central-node.By default, the Flannel controller assumes it should use the IP address from the first nonloopback interface as the public address of the node which is typically eth0. Other CNI network plugins may have different ways of changing the data plane network interface.

Download the Flannel manifest onto central-node:

curl -k https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml > kube-flannel-wireguard.yml 

Search for where the container specification for the kube-flannel container is located after the init-containersection and look for theargs:key. This key has a list of arguments to the Flannel container when it is started. Edit the list so it looks like this:

args:
— --ipmasq
— --kube-subnet-mgr
— --iface
— wg0

Note that the wg0is on a separate line after the --ifaceentry. It is the next element in the yaml array. On several web pages, there are instructions to put it directly after the iface with an equals, this is incorrect.

To install Flannel:

kubectl apply -f kube-flannel-wireguard.yml

Note that Flannel now deploys into its own namespace, kube-flannel, rather than into the kube-system namespace.

You can check if Flannel is running with:

kubectl get -n kube-flannel all 

Flannel uses an annotation on the Kubernetes Node object to determine the public IP address of the node. Check for the annotation with:

kubectl describe node central-node | grep flannel.alpha.coreos.com/public-ip 

It should be 10.8.0.1 on central-node. When gateway-node is up, check that it is 10.8.0.2.

Also, you need to turn the checksum off on the flannel.1 interface to improve performance. The one shot command is :

sudo ethtool -K flannel.1 tx off 

For reboot, set up a systemd service as follows.

• As superuser, create a file called 100-flannel-trigger-rules in/etc/udev/rules.d:

#This sets triggers for kube-flannel to remove checksum calculation
#on the flannel.1 interface.
SUBSYSTEM=="net",ACTION=="add",KERNEL=="flannel.*",TAG+="systemd",ENV{SYSTEMD_WANTS}="flannel-created@%k.service"

• As superuser, create a file called flannel-created@.service in/etc/systemd/system:

[Unit]
Description=Disable TX checksum offload on flannel interface
[Service]
Type=oneshot
ExecStart=/sbin/ethtool -K %I tx off

Then reload via:

sudo systemctl daemon-reload && sudo systemctl restart systemd-udevd.service

Installing gateway-node as a worker node

To join gateway-node to the cluster as a worker node, use the join.sh shell script on gateway-nodeyou created above:

sudo ./join.sh 

Note that the token is only good for a day, if you need to install more gateway nodes, use this command on central-node to create a new one:

kubeadm token create 

If you don’t have the value of --discovery-token-ca-cert-hash you can find by running this command on central-node:

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
openssl dgst -sha256 -hex | sed 's/^.* //' 

Finally, to use kubectl on the gateway-nodeto deploy and monitor pods, you need to copy the config file from your ~/.kube directory on central-node to gateway-node. Also copy the file into /etc/kubernetes/admin.conf as root on the gateway-node so other users have access to it.

Reinitializing the kubelet on gateway-node with the wg0 IP address

Like Flannel, the kubeadm join command uses the first nonloopback interface on the node to to determine the Node object's public IP address. This address is then used to restart the kubelet which acts as the local node manager. Since wg0 is not the first interface, the node's kubelet will not be accessible from central-node. In addition, if the Kubernetes control plane doesn't run over the Wireguard VPN, it will not be encrypted. We need to reinitialize the kubelet so that it gets the correct address, 10.8.0.2. If this step is not taken, central-node will get a routing error when it tries to contact the kubelet on gateway-node, for example, when trying to exec into a pod using kubectl.

First, create as superuser a file calledkubelet in /etc/defaultwith one line in it:

KUBELET_EXTRA_ARGS="--node-ip=10.8.0.2"

This argument is added to the startup command for the kubelet by the systemd unit file and will cause the kubelet start with the IP address on the wg0 interface.

Next, edit the file /var/lib/kubelet/config.yaml as superuser and add a line after the apiVersion: line with the address: key and the wg0 IP address as the value, for example:

address: 10.8.0.2 

This will cause the kubelet to listen on that address rather than the default (0.0.0.0) .

Finally, restart the kubelet using systemctl:

sudo systemctl stop kubelet
sudo systemctl disable kubelet
sudo systemctl enable kubelet
sudo systemctl start kubelet 

You can test it by checking what interface the kubelet is running on:

sudo ss -ltp | grep kubelet | grep 10.8.0.2 

If you are wondering whether you should change thekubelet address on central-node,it isn’t necessary since traffic between the kubelet and the Kubernetes API server on the central-node stays on the the VM and does not go out over the Internet.

Checking the cluster

Check if everything is running OK by running kubectl on gateway-node:

kubectl get -n kube-system pods --output wide

And for the Flannel network:

kubectl get -n kube-flannel pods --output wide

These commands will write out all the Pods, Services, Deployments, etc. in their respective system namespaces.

Summary

Your cluster should now be ready for application deployment. You can deploy either on central-nodeor on gateway node. Part III leads your through deploying a demo microservice application onto the cluster and exposing the web service for Internet access.