Constructing a Kubernetes Cluster spanning a Public Cloud VM and a Local VM: Part III

Kubernetes Cluster (Source: Ashish Patel, Medium)

This is the third and final installment of a three part series on how to set up a Kubernetes cluster spanning a cloud VM and a local VM:

• Part I : Describes how to set up a VM on a cloud provider (I used Azure) and a local VM in VirtualBox , how to configure them for Kubernetes, and how to connect them together using a Wireguard VPN.
• Part II : Describes how to set up the Kubernetes cluster spanning the cloud VM and local VM withkubeadm, and how to ensure that the control and data plane run over the encrypted VPN connection.
• Part III (here): Describes how to set up an Nginx reverse proxy and dnsmasq on the cloud VM, and configure them so that a web application in the Kubernetes cluster can be served to the Internet through the cloud VM’s Internet-accessable DNS host name.

In this, the third installment, we’ll install dnsmasqand Nginx, configure dnsmasqand the CoreDNS Kubernetes DNS service to resolve service names in the cluster from outside, configure the Nginx reverse proxy to proxy the service’s web application to the Internet through the central-nodeVM’s DNS name, deploy a test service to the cluster, and test the service from an Internet connected browser.

Options for exporting a Service running in a Kubernetes Cluster to the Internet

While the instructions in this article aren’t, strictly speaking, integral to bringing up a Kubernetes cluster spanning a cloud VM and a local VM, I found in my work with Volttron that the documentation for exporting a service’s web application through a cloud provider Internet accessible DNS name is somewhat difficult to piece together. Your choices are the following:

• Use the externalIPs: key in a NodePort service definition to allocate an external IP address from the local network’s range. Except this won’t work on a cloud VM, since the cloud provider’s SDN API won’t recognize the address and configure the routes for you.
• Define the service as a LoadBalancer service and use a cloud provider load balancer. Deploying a cloud provider load balancer is complicated with many steps that can go wrong, and can be expensive. If you don’t want the service to be HA or you want to manage the HA yourself, an alternative is needed.
• Define the service as a LoadBalancer service and use MetalLB, an open source load balancer designed for on-prem deployments. Except MetalLB won’t work with a cloud deployment because nobody has fixed it yet to use the cloud provider SDN APIs.
• Define the service as a NodePort service, which theoretically gives you a public IP address on each node and a port in the 30000–32768 which is accessible from any node. But this also doesn’t give you an external IP address unless you use externalIPs.

Strictly speaking, you don’t have to configure dnsmasqto resolve the service name. You could use kubectl port-forward to forward a port from the service to localhost, and configure Nginx to reverse proxy to the port on localhost. But DNS service discovery is a nice feature of Kubernetes that makes services more understandable for people who are administering and using the cluster. In addition, port forwarding requires the kubectl process to run as long as the service is accessable, so it must either be run as a daemon or you need to keep a shell window up on it.

There is also another option:

• Use a Kubernetes Ingress controller to control traffic to the cluster-resident service.

The problem with Ingress controllers is that they are very restricted in the traffic they handle. They don’t forward to cluster-resident HTTPS endpoints for example, and will only reverse proxy traffic coming in through ports 80 and 443, the ports for HTTP and HTTPS. The Ingress controller needs to terminate the TLS connection for HTTPS, and if you want encrypted traffic between the Ingress controller and the cluster-resident service, you have to use a service mesh. For the application we’ll be deploying below as a test, this isn’t a problem because the cluster-resident service accepts HTTP traffic and accessing it through port 80 from the Internet is fine.

For Volttron, it doesn’t work. Volttron as a legacy application uses HTTPS and accepts traffic on port 8443. It would have required considerable modification to the Volttron source code to convert it to HTTP, since the Volttron services use HTTPS to communicate amongst themselves. In the next round of Volttron work, I expect to fix this. But for now, using Nginx as a reverse proxy outside the cluster solves the problem nicely. But it does come with a cost, namely the external Nginx systemd service doesn’t co-ordinate during boot with the service it will reverse proxy to in the cluster, so we need to define a systemd service to check if the cluster-resident service is up at boot time and restart Nginx when it is.

Fixing DNS on central-node so that it can resolve service/host names in the cluster

Kubernetes runs the CoreDNS servers as an optional service, but when you install with kubeadm, the servers are installed and configured by default. CoreDNS allows services in the cluster to find other services using the svc.cluster.localdomain name. However, outside the cluster, the CoreDNS server is unavailable without some configuration. Since the Nginx reverse proxy is performing service name lookup from outside the cluster, we need to configure the DNS service on the host to forward names containing the cluster.localdomain name to the CoreDNS servers inside the cluster.

The only reference I could find about how to configure DNS lookup is here, and it assumes that your host is running NetworkManager. The instructions for editing the CoreDNS ConfigMap are also out of date (proxy has been replaced by forwardfor forwarding to outside resolvers). Unfortunately, my Azure cloud VM was not running NetworkManager, it was running systemd-resolved. When I tried installing NetworkManager I ran into problems, so I had to piece together these instructions from various sources. The following instructions should be run on the central-node VM.

First step is to edit your /etc/hosts file to ensure commands don't hang trying to look up central-node's name if you lose access to DNS. Find your ip address on central-node with:

ip address

and select the address associated with the eth0 interface or the interface with the lowest number as the last character. Then as superuser, edit /etc/hosts and insert one line at the top with the IP address and name of central-node:

<central-node IP address> central-node

This ensures that when you turn off DNS, should you run into problems, commands won’t hang for lengthy periods.

You should also find your current DNS server addresses by running:

systemd-resolve --status

The addresses will appear near the end of the output, under the link names for your primary network interface (typically eth0).

Next step is to install dnsmasq with systemd-resolvd running (so you have DNS service):

sudo apt update
sudo apt install dnsmasq

Check status with:

systemctl status dnsmasq.service

The status may indicate failure because systemd-resolved.service is still running, but don't worry, we'll restart it after we configure dnsmasq.

Now sudo edit the dnsmasq configuration file /etc/dnsmasq.conf and uncomment the following lines:

domain-needed
bogus-priv 

These ensure that names without a dot are not forwarded upstream and that addresses from non-routed address spaces are not returned downstream.

Search for the line beginning #server=. Delete the line and add the following lines:

server=/cluster.local/10.96.0.10
server=<IP address of first name server>
server=<IP address of second name server>
...

where the IP addresses of the names servers printed out by systemd-resolve --status are at the end of the lines indicated. The first line is the default address of the CoreDNS service in the cluster. If you suspect the service is deployed at a different address, you can check with kubectl get -n kube-system svc kube-dns.

The first line will cause queries for names with the Kubernetes cluster domain name to be sent to the CoreDNS server, the others will ensure you have default DNS service.

Save the file and exit.

Disable systemd-resolved:

sudo systemctl stop systemd-resolved.service
sudo systemctl disable systemd-resolved.service 

Restart dnsmasq:

sudo systemctl restart dnsmasq.service

Check for upstream connectivity:

ping google.com

Editing CoreDNS ConfigMap to forward to upstream DNS

Next, we need to edit the CoreDNS ConfigMap so that it forwards to the upstream DNS rather than using /etc/resolv.conf through 127.0.0.1, the dnsmasq address, which can slow name resolution due to looping between CoreDNS and dnsmasq.

Use kubectl to edit the CoreDNS ConfigMap:

kubectl edit -n kube-system configmap coredns

The ConfigMap should come up in your favorite editor (the value of the EDITOR shell environment key). Search for the line with forward in it and replace /etc/resolv.conf with the IP address of the DNS server on your primary interface from above:

forward . <IP address of primary interface DNS server> {

Save the file and exit.

Now restart the CoreDNS service by finding the pod names (there should be two):

kubectl get -n kube-system pods | grep coredns

Delete the pods:

kubectl delete -n kube-system pod <coredns pod name> <coredns pod name>

The CoreDNS Deployment should restart the pods automatically.

When the pods are running, test resolution into the Kubernetes cluster with:

dig kube-dns.kube-system.svc.cluster.local

It should return something like:

...
;; QUESTION SECTION:
;kube-dns.kube-system.svc.cluster.local.	IN A

;; ANSWER SECTION:
kube-dns.kube-system.svc.cluster.local.	30 IN A	10.96.0.10
...

Deploying the emojivoto web application to the cluster

Emojivoto is a microservice application that allows you vote for your favorite emoji. It has an autovoter feature that automatically generates votes, and some additional advanced features like the ability to connect into Prometheus for observability. Emojivoto is heavily used in tutorials on service mesh and other advanced Kubernetes topics, but we’ll just deploy it as a functional microservice app that is accessible from a web page. You can find out more about emojivoto at this link (but ignore the deployment instructions as they are for minikube).

On central-node, deploy emojivoto directly from its website:

curl -sL https://run.linkerd.io/emojivoto.yml | kubectl apply -f -

You should see a list of Kubernetes objects created in the emojivoto namespace:

namespace/emojivoto created
serviceaccount/emoji created
serviceaccount/voting created
serviceaccount/web created
service/emoji-svc created
service/voting-svc created
service/web-svc created
deployment.apps/emoji created
deployment.apps/vote-bot created
deployment.apps/voting created
deployment.apps/web created

You should be able to get to the emojivoto web application from the command line in an ssh window on central-nodeby typing http://web-svc.emojivoto.svc.cluster.localNote that dnsmasq does not export the name to any nodes outside the local VM itself, which is why we need to use Nginx to reverse proxy it. Note also that the pods are actually deployed on gateway-node which you can determine by running kubectl get -n emojivoto pods -o wide

Configuring Nginx to proxy the emojivoto web-svc web page through a cloud VM’s DNS name

You now need to configure Nginx to proxy the emojivoto web-svc web page through the cloud VM’s DNS name. Edit /etc/nginx/nginx.conf as superuser and comment out the line for sites enabled:

# include /etc/nginx/sites-enabled/*;

This ensures that only path names from /etc/nginx/conf.d/are used by Nginx. Save the file and exit.

Create the following file named kube.conf in /etc/nginx/conf.d/:

server {
    listen 80 default server;
    listen [::]:80 default server;
    location / {
        proxy_pass http://web-svc.emojivoto.svc.cluster.local;
    }
}

Warning: do not put anything additional into the pathname, or the Javascript app in the web-svc container won’t display.

Reload Nginx with:

sudo nginx -s reload

It should not print anything out if your edits were correct.

Creating a system service to restart Nginx after the emojivoto service is running

You may want to shut down your central-nodeperiodically to save cost or for other reasons then boot it up again when you need it. Because systemd does not synchronize starting dnsmasq.service and nginx.service with the Kubernetes cluster, if the CoreDNS pods are not running when dnsmasq boots and emojivotohasn’t started, dnsmasq won't be able to resolve the web-svcname to an address and the Nginx service will fail to start when it checks for the web-svcsite to proxy.

To fix this, we define a systemd service, restart-nginx-dnsmasq.service which starts the shell script in restart-nginx-dnsmasq.sh. This shell script does the following:

• Checks if Nginx has been configured to proxy requests to web-svc,
• Waits until the web-svcservice is running,
• Restarts nginx.service and exits.

As superuser, create the file restart-nginx-dnsmasq.service in /etc/systemd/system by typing in the following:

[Unit]                             
Description=Restart nginx when web application in the cluster has booted.                             
After=network.service kubelet.service dnsmasq.service 
[Service]                             
ExecStart=/usr/local/bin/restart-nginx-dnsmasq.sh
                                                           
[Install]                             
WantedBy=multi-user.target

Save the file and exit.

As superuser, create the file /usr/local/bin/restart-nginx-dnsmasq.sh by typing in the following:

#! /bin/bash                                                           # Restart nginx.service when the application is                             # running.
                                                           
# Check if Nginx is configured for web-svc
if [[ ! -e /etc/nginx/conf.d/kube.conf ]] ; then
    exit                             
fi  
                                                         
grep web-svc /etc/nginx/conf.d/kube.conf &>/dev/null                             RETURN=$?                                                          
if [[ $RETURN -ne 0 ]]; then                                 
    exit                             
fi                                                           
# Wait until the API server comes up.                                                           
curl -k http://web-svc.emojivoto.svc.cluster.local &>/dev/null                 
RETURN=$?                                                           
while [[ $RETURN -ne 0 ]]; do  
    curl -k http://web-svc.emojivoto.svc.cluster.local &>/dev/null
    RETURN=$?                                                           done 
                                                          
# Restart nginx                                                           
echo "Restarting nginx.service."                                                           systemctl restart nginx.service                                                           echo "Done."

Enable and start the service:

sudo systemctl enable restart-nginx-dnsmasq.service
sudo systemctl start restart-nginx-dnsmasq.service

Check if the service started OK with:

systemctl status restart-nginx-dnsmasq.service

Bringing up the emojivoto web application in a browser

To get to the emojivoto app from the Internet, you need to first find the cloud VM’s public DNS name. On Azure, the DNS name for your VM is on the VM Overview (dashboard) page. You can then use a browser from the gateway-node host or a laptop to access the emojivoto app over the Internet by typing http://<DNS name for VM> into the browser address bar. Note that using the VM's global DNS name may not work from the command line on the cloud VM itself, but you can always use the service name directly if you want to access it from outside the cluster on central-node.

You should see the following Web page displayed:

Emojivoto Splash Screen

You can click around to try out the app and vote for your favorite emoji.

Summary

Congratulations! You have just deployed a microsevice web app onto a multi-site Kubernetes cluster and accessed it from the Internet!