Constructing a Kubernetes Cluster spanning a Public Cloud VM and a Local VM: Part I

Kubernetes Cluster (Source: Ashish Patel, Medium)

Recently, I’ve been working on a Kubernetes deployment of the open source Volttron(tm) energy management platform from Pacific Northwest Labs used for building energy management services, converting it from a monolithic application into a collection of containerized microservices deployed on Kubernetes. The architecture of an energy management service requires a centralized node, typically running in a public cloud, where the web site and database live, connected over a VPN with multiple local sites running a gateway to IoT devices on the site that measure and control energy generating and consuming devices.

Kubernetes seems like the ideal technology for implementing such an architecture. In a Kubernetes energy management cluster, the central node runs in a cloud VM serving the web site over the Internet while the worker nodes run at the local sites with the cluster spanning both. Management, configuration, upgrade, etc. can all be handled from the central node with Kubernetes, and Kubernetes has a excellent authentication and authorization support including role-based access control. However, if you don’t want to use a cloud provider’s IoT managed service, you are not interested in dealing with the complexity of ClusterAPI’s BYOH, and you don’t want a heavyweight solution like Kilo but still want to manage the cluster yourself, there is actually no complete description of how to do it that I could find on the Internet.

This series is about how to set up such a cluster. There are three parts:

• Part I (here): Describes how to set up a VM on a cloud provider (I used Azure) and a local VM in VirtualBox for Kubernetes, and how to connect them together using a Wireguard VPN.
• Part II: Describes how to set up the Kubernetes cluster spanning the cloud VM and local VM with kubeadm, and how to ensure that the control and data plane run over the encrypted VPN connection.
• Part III: Describes how to set up an Nginx reverse proxy and dnsmasqon the cloud VM, and configure them so that a web application in the Kubernetes cluster can be served to the Internet through the cloud VM’s Internet-accessable DNS host name.

Part III isn’t, strictly speaking, about setting up a Kubernetes cluster, but if you don’t need high availability or want to provide it yourself rather than using a cloud provider’s load balancer service, it explains a low cost way to make a web site in the cluster Internet-accessible without having to use a cloud provider’s load balancer. In addition, the instructions are oriented to a cluster with one VM running in a public cloud and another node, typically a VM, running locally, but they could as well be applied to one VM in one public cloud provider and another VM in another cloud provider.

So let’s get started!

Creating the VMs

My local node consisted of a VirtualBox 6.1 VM running Ubuntu 20.04 on top of a Ubuntu 20.04 host with 2 vCPUs, 4 GB of memory and 20 GB of virtual disk. My remote node consisted of an Ubuntu 20.04 Standard B2ms Azure VM with 2 vCPUs, 8 GB memory and 30 GB disk. The instructions in this series will likely work for any Ubuntu 20.04 host, virtual or bare metal, for the local node. I named my node central-nodefor the cloud VM and gateway-node for the local VM.

No special configuration is required for the VirtualBox VM on your local host, except that the MAC address should be unique and the network adapter type should be set to Bridged Adapter. If the VM will be running behind a corporate firewall, you may need to contact your IT staff about opening a port for Wireguard (51820). You should not have to open any ports for Kubernetes since all the control plane and data plane traffic will run over the Wireguard VPN. If your local VM is connected directly up to the Internet, you should install and configure a local firewall on the VM and open a port in it for Wireguard, locked to the IP address of your cloud VM. The instructions below assume the local VM is running behind an ISP’s firewall, in which case, the firewall will open a port for you upon the first outbound connection.

You will need the public addresses of the cloud VM and local VM to configure Wireguard and to configure the cloud provider’s firewall. The public IP address of the cloud VM central-nodeshould be on the dashboard for the VM. For example, in Azure, it is located in a field with title Public IP address. Finding the public address of the gateway-node depends on whether you are running in a corporate LAN or accessing the Internet through an ISP network. If you are running in a corporate LAN, ask your IP support people what you should use for your public IP address. If you are connecting through an ISP, you can find your address by browsing to whatsmyip.org with a browser running on the local VM. Write down both addresses somewhere.

Once the VM is running, bring up the Azure dashboard display for your VM by clicking on All resources-><VM name>. Click on the Public IP address on the right side of the VM dashboard. When the IP address dashboard comes up, click on the Static radio button under the IP address assignment label. This ensures that the IP address of the VM will remain the same if you bring down and restart the VM. Move down to the DNS name label (optional) field and
type in “central-node”. This renames the VM to central-node.Click on Save on the top bar menu to save the configuration, and the X in the upper left corner to return to the VM dashboard.

In the left side menu, click on Networking. This will bring up a table of the open ports in the Azure firewall. The table should have a row for port 22, open for ssh . Double click on the row for ssh. The port tab will come up on the right. If the source is Any, meaning any IP address, you should set the source to the gateway-node VM public IP address which you recorded above since you will likely want to work from an sshwindow there.

You will need to open new ports for Wireguard and the Nginx reverse proxy. Click on the blue Add inbound port rule button and fill out the form with the following for the Wireguard port:

• Source: IP Address.
• Source IP addresses/CIDR ranges: Public IP address of the gateway-node host.
• Source port ranges: Leave at default * (any).
• Destination: Leave at default of Any.
• Service: Leave at Custom.
• Destination Port Ranges: 51820, the Wireguard port
• Protocol: Check the UDP box.
• Action: Leave the Allow box checked.
• Priority: Leave at the default.
• Name: Wireguard.
• Description: Port for Wireguard VPN.

After you’ve configured a port for Wireguard, configure one for Nginx with the following:

• Source: IP Address.
• Source IP addresses/CIDR ranges: Public IP address of the gateway-node host.
• Source port ranges: Leave at default of * (any).
• Destination: leave at default of Any.
• Service: HTTP.
• Destination Port Ranges: It will be set to 80 and greyed out.
• Protocol: Leave the TCP box checked.
• Action: Leave the Allow box checked.
• Priority: leave at the default.
• Name: Nginx.
• Description: Port for Nginx reverse proxy.

Disable any local firewalls

Unless your local VM is directly connected to the Internet without an ISP or corporate firewall wall, you should disable any firewall running directly on both VMs since a local firewall will require extra configuration. Your local VM outside of the cluster will be protected by the ISP or corporate firewall and inside the cluster by the Kubernetes firewall and the cloud VM will be protected by the cloud provider’s firewall. However, if the host running the local VM is directly connected to the Internet, then by all means, install and start a firewall and configure the Wireguard ports to be opened the host!

Setting the host names and installing tools

Prior to installing Wireguard, be sure to set the hostname on both nodes:

hostnamectl set-hostname <new-hostname>

and confirm the change with:

hostnamectl

If you used the names central-node and gateway-node to create the VMs, the host name should be set, but if you cloned the VMs, you may need to change it. You should not have to reboot the node to have the host name change take effect.

You should also install your favorite editor if it isn’t there, and packages containing network debugging tools including net-tools and inetutils-traceroute (for ifconfig and traceroute) just in case you need them.

Ensuring the VMs have unique MAC addresses

Kubenetes uses the hardware MAC addresses and machine id to identify pods. Use the following to ensure that the two nodes have unique MAC addresses and machine ids:

• Get the MAC address of the network interfaces using the command ip link or ifconfig -a,
• Check the product_uuid by using the command sudo cat /sys/class/dmi/id/product_uuid.

Turning off swap

Turn off swap on the VMs:

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Turning on routing

We need to enable routing on both nodes by sudo editing /etc/sysctl.conf and deleting the # character at the beginning of the lines with net.ipv4.ip_forward=1 and net.ipv6.conf.all.forwarding=1.This enables routing on the VMs after reboot. Then use the command:

sudo sysctl <routing variable>=1

where <routing variable> is net.ipv4.ip_forward and net.ipv6.conf.all.forwarding to enable routing in the running VMs.

You can test whether your configuration has worked by running:

sudo sysctl -a | grep <routing variable>

Enabling the bridge filter kernel module

Enable the bridge net filter driver br_netfilter:

sudo modprobe br_netfilter

Edit the file /etc/modules as superuser and add a line with br_netfilter so the module will be reloaded when the VMs reboot.

Installing and configuring the Wireguard VPN

Wireguard creates a virtual interface on a node across which a UDP VPN connects to other peers. The interface has a public and private key associated with it that are used for decrypting and encrypting the packets, respectively. The interface is locked to the public IP address of the other node, and the IP address of the interface must be within a particular CIDR subnet. The link between one peer and another is point to point. We will be using the 10.8.0.0/24 CIDR subnet over an interface namedwg0 with the central-node having address 10.8.0.1 and gateway-node having address 10.8.0.2.

Most of the pages with instructions for installing and configuring Wireguard on Ubuntu 20.04 assume you want to deploy the cloud node as a VPN server and route through it to other services, in order to hide your local machine’s or mobile phone’s IP address. The result is that the instructions include configuring iptables to route packets out of the VM, which is completely unnecessary for our use case. We don’t need this, since we will be running the Kubernetes control and data plane traffic between nodes in the cluster and routes to other services on other nodes will be handled by Kubernetes or the host operating system. The best guide I've found is at this link. The author notes where you can skip configuration instructions for deploying Wireguard as a VPN server, and includes instructions for configuring with IPv6 which are nice if you have IPv6 available but only increase the complexity. Below, I've summarized the instructions for installing and configuring Wireguard for this use case using IPv4.

Installing Wireguard and related packages

Update on both the gateway node and central node with:

sudo apt update 

After the update is complete, install Wireguard:

sudo apt install wireguard 

apt will suggest you install one of openresolv or resolvconf, the following instructions are based on installing resolvconf:

sudo apt install resolvconf 

Wireguard commands

Wireguard comes with two utilities:

• wg: the full command line utility for setting up a Wireguard interface.
• wg-quick: a simpler command line utility that summarizes frequently used collections of command arguments under a single command.

The configuration file and public and private key files for the Wireguard are kept in the directory /etc/wireguard.

Generating public and private keys on both nodes

Starting on central-node, generate a private and public key using the Wireguard wg command line utility and change the permissions on the file to only allow access to root:

wg genkey | sudo tee /etc/wireguard/private.key
sudo chmod go= /etc/wireguard/private.key 

The first command generates a base64 encoded key and echoes it to /etc/wireguard/private.key and to the terminal, the second changes the permissions so nobody except root can look at it.

Next, generate a public key from the private key as follows:

sudo cat /etc/wireguard/private.key | wg pubkey | sudo tee /etc/wireguard/public.key 

This command first writes the private key from the file to stdout, generates the public key with wg pubkey, then writes the public key to /etc/wireguard/public.key and to the terminal.

Now, switch to gateway-node and run the same commands.

Creating the wg0 interface configuration file on the gateway-node

The next step is to create a configuration file for gateway-node. Using your favorite editor, open a new file /etc/wireguard/wg0.conf (running as sudo). Edit the file to insert the following configuration:

[Interface]
PrivateKey = <insert gateway-node private key here>
Address = 10.8.0.2/24
ListenPort = 51820 
[Peer]
PublicKey = <insert central-node public key here>
AllowedIPs = 10.8.0.0/24
Endpoint = <insert public IP address of your central node here>:51820
PersistentKeepalive = 21 

Add the private key of the gateway-node, public key of central-node, and public IP address of central-node where indicated. Save the file and exit the editor.

Creating the wg0 interface configuration file on the central node

Edit the file /etc/wireguard/wg0.conf as superuser:

[Interface]
PrivateKey = <insert private key of central-node here>
Address = 10.8.0.1/24
ListenPort = 51820
[Peer]
PublicKey = <insert gateway-node public key here>
AllowedIPs = 10.8.0.0/24
Endpoint = <insert public IP address of gateway-node here>:51820
PersistentKeepalive = 21 

Add the private key of the central-node, public key of gateway-node, and public IP address of gateway-node where indicated. Save the file and exit the editor.

Installing a system service for the wg0 interface

Starting on gateway-node, we'll use systemctl to install a system service that creates and configures the Wireguard wg0 interface when the node boots. To enable the system service, use:

sudo systemctl enable wg-quick@wg0.service 

start the service with:

sudo systemctl start wg-quick@wg0.service 

and check on it with:

sudo systemctl status wg-quick@wg0.service 

to see the service status. This should show something like:

wg-quick@wg0.service - WireGuard via wg-quick(8) for wg0
 Loaded: loaded (/lib/systemd/system/wg-quick@.service; enabled; vendor preset: enabled)
 Active: active (exited) since Fri 2022-05-20 15:17:27 PDT; 12s ago
   Docs: man:wg-quick(8)
         man:wg(8)
         https://www.wireguard.com/
         https://www.wireguard.com/quickstart/
         https://git.zx2c4.com/wireguard-tools/about/src/man/wg-quick.8
         https://git.zx2c4.com/wireguard-tools/about/src/man/wg.8
Process: 2904 ExecStart=/usr/bin/wg-quick up wg0 (code=exited, status=0/SUCCESS)
Main PID: 2904 (code=exited, status=0/SUCCESS)
May 20 15:17:27 central-node systemd[1]: Starting WireGuard via wg-quick(8) for wg0...
May 20 15:17:27 central-node wg-quick[2904]: [#] ip link add wg0 type wireguard
May 20 15:17:27 central-node wg-quick[2904]: [#] wg setconf wg0 /dev/fd/63
May 20 15:17:27 central-node wg-quick[2904]: [#] ip -4 address add 10.8.0.1/24 dev wg0
May 20 15:17:27 central-node wg-quick[2904]: [#] ip link set mtu 1420 up dev wg0
May 20 15:17:27 central-node systemd[1]: Finished WireGuard via wg-quick(8) for wg0. 

Notice that the status prints out the ip commands that were used to create the interface. You can also see the wg0 interface using:

ip address 

After you’ve started the system service on gateway-node, follow the same instructions as above for installing a system service to bring up the wg0 interface on central-node.

Checking wg0 status and bidirectional connectivity

You can check the status of the wg0interface by running:

sudo wg 

on both nodes. It should print out something like:

public key: dScu7fYSEKrZFdNYqycAyeqJdz9utYXGcgYP9YPc2Sg=
private key: (hidden)
listening port: 51820
peer: V7EWFuM1qARFs1ldwCx2P6HOMcTMU5yY51QSw4t5gCI=
  endpoint: <public address of cloud or local VM>
  allowed ips: 10.8.0.0/24
  latest handshake: 1 minute, 42 seconds ago
  transfer: 1.29 KiB received, 4.72 KiB sent
  persistent keepalive: every 21 seconds 

where the important information is that the keep-alive transfer is not showing zero.

Check for bidirectional connectivity by pinging first on the central-node:

ping 10.8.0.2 

then on the gateway-node:

ping 10.8.0.1 

Summary

You now have two nodes, one in your local network and one in a cloud VM, connected through a Wireguard VPN that you can use to create a Kubernetes cluster. Please continue to Part II at this link for instructions on how to set up the cluster.